DSE Search node start

Troubleshooting tips to handle startup issues on DSE Search nodes.

Slow startup on nodes with large encrypted indexes

Slow startup on nodes with large encrypted indexes is resolved in 6.0.0 and later, 5.1.6 and later, and 5.0.12 and later. However, action is required to realize the performance gains.

First upgrade the nodes and then do a full reindex of all encrypted search indexes on each node in your cluster. Plan sufficient time after the upgrade is complete to reindex with deleteAll=true in a rolling fashion. For example:

dsetool reload_core keyspace_name.table_name distributed=false reindex=true deleteAll=true

DSE Search nodes fail to start

DSE 5.1.11 and later and DSE 6.0.3 and later refuse to start when required Tomcat files are not present.

When the tomcat/conf directory is missing, a WARN message like this appears in the system.log:

WARN  [localhost-startStop-2] 2018-07-26 11:55:41,104  DirectJDKLog.java:182 - Failed to scan [file:/usr/local/dse/dse-5.1.9/resources/cassandra/lib/apache-cassandra-thrift-3.11.1.2261.jar] from classloader hierarchy
java.io.FileNotFoundException: /usr/local/dse/dse-5.1.9/resources/cassandra/lib/apache-cassandra-thrift-3.11.1.2261.jar (No such file or directory)
        at java.util.zip.ZipFile.open(Native Method) ~[na:1.8.0_181]
        at java.util.zip.ZipFile.<init>(ZipFile.java:225) ~[na:1.8.0_181]
        at java.util.zip.ZipFile.<init>(ZipFile.java:155) ~[na:1.8.0_181]
        at java.util.jar.JarFile.<init>(JarFile.java:166) ~[na:1.8.0_181]
        at java.util.jar.JarFile.<init>(JarFile.java:130) ~[na:1.8.0_181]
        at org.apache.tomcat.util.scan.JarFileUrlJar.<init>(JarFileUrlJar.java:60) ~[tomcat-util-scan-8.0.47.jar:8.0.47]
        at org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49) ~[tomcat-util-scan-8.0.47.jar:8.0.47]
        at org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:334) ~[tomcat-util-scan-8.0.47.jar:8.0.47]
        at org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:284) ~[tomcat-util-scan-8.0.47.jar:8.0.47]
        at org.apache.catalina.startup.ContextConfig.processJarsForWebFragments(ContextConfig.java:1898) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1131) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:783) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:307) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:95) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5213) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:753) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:131) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:153) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:143) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_181]
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:727) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1129) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1871) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]

Followed by reference to the missing locale Tomcat JAR files:

WARN  [localhost-startStop-2] 2018-07-26 11:55:41,106  DirectJDKLog.java:182 - Failed to scan [file:/usr/local/dse/dse-5.1.9/resources/spark/lib/derbyLocale_cs.jar] from classloader hierarchy
java.io.FileNotFoundException: /usr/local/dse/dse-5.1.9/resources/spark/lib/derbyLocale_cs.jar (No such file or directory)
        at java.util.zip.ZipFile.open(Native Method) ~[na:1.8.0_181]
        at java.util.zip.ZipFile.<init>(ZipFile.java:225) ~[na:1.8.0_181]
        at java.util.zip.ZipFile.<init>(ZipFile.java:155) ~[na:1.8.0_181]
        at java.util.jar.JarFile.<init>(JarFile.java:166) ~[na:1.8.0_181]
        at java.util.jar.JarFile.<init>(JarFile.java:130) ~[na:1.8.0_181]
        at org.apache.tomcat.util.scan.JarFileUrlJar.<init>(JarFileUrlJar.java:60) ~[tomcat-util-scan-8.0.47.jar:8.0.47]
        at org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49) ~[tomcat-util-scan-8.0.47.jar:8.0.47]
        at org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:334) ~[tomcat-util-scan-8.0.47.jar:8.0.47]
        at org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:284) ~[tomcat-util-scan-8.0.47.jar:8.0.47]
        at org.apache.catalina.startup.ContextConfig.processJarsForWebFragments(ContextConfig.java:1898) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1131) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:783) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:307) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:95) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5213) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:753) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:131) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:153) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:143) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_181]
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:727) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1129) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1871) [tomcat-embed-core-8.0.47.jar:8.0.47]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]

To resolve:

Ensure that the catalina.properties and context.xml files are present in the Tomcat conf dir. DSE will not start after upgrade if these files are missing.
The default location of the Tomcat conf directory depends on the type of installation:
- Package installations: /etc/dse/tomcat/conf
- Tarball installations: installation_location/resources/tomcat/conf