Start and stop DataStax Enterprise (DSE)
Use these commands to start and stop DataStax Enterprise (DSE) nodes in a cluster.
|
Many DSE configuration changes require a rolling restart. This means that you stop and start each node one at a time, rather than stopping the entire cluster simultaneously. Rolling restarts allow the cluster to continue servicing requests while you apply configuration changes. |
Considerations for starting a cluster
Be aware of the following when starting a DSE cluster:
- Nodes must be segregated by datacenters
-
Transactional-only, DSE Search, DSE Analytics, and SearchAnalytics nodes must be deployed to dedicated datacenters based on node type. For example, in a cluster with both DSE Search and transactional-only nodes, deploy the DSE Search nodes to their own datacenters, and deploy the transactional-only nodes to different datacenters.
Segregating nodes by datacenter is required for the cluster to form properly. This segregation doesn’t restrict access to the advanced workload functionality; after deploying an advanced workload datacenter, all nodes in the cluster can access the advanced workload functionality.
DataStax Graph (DSE Graph) can be enabled on any node in any datacenter.
- Always start seed nodes first, following the node type hierarchy
-
When starting a multi-node cluster, start seed nodes first.
If a cluster has any nodes that run in DSE Analytics, Search, Graph, or combined advanced workload modes, you must start the nodes in the following order:
-
Analytic seed nodes.
-
Transactional and DSE Graph-only seed nodes.
-
DSE Search seed nodes.
-
SearchAnalytics seed nodes and non-seed nodes.
-
All other non-seed nodes one at a time.
The cluster might not initialize properly if nodes are started out of order.
For more information, see Initializing multiple datacenters per workload type.
-
- Replication factor for DSE Analytics nodes
-
Before starting DSE Analytics nodes, ensure that the replication factor is configured correctly for the analytics keyspaces. Every time you add a new datacenter, you must manually increase the replication factor of the
dse_leaseskeyspace for the new DSE Analytics datacenter.
Node types (workloads)
A node’s type refers to the DSE workloads it supports. Node types are also referred to as workloads, workload types, and modes.
All nodes are transactional (cassandra) nodes that run the database.
Advanced workload types extend the base transactional workload with additional capabilities for DSE Analytics (Apache Spark™), Search, or Graph.
You can start a nodes with zero, one, or many advanced workloads enabled.
| Type | Description |
|---|---|
Transactional ( |
Database node with no advanced workloads enabled. This is the default node type when you start DSE with no advanced workload options. All nodes are inherently transactional nodes. Advanced workloads are enabled in addition to the base transactional functionality. There is no explicit setting to start a transactional-only node. Instead, you omit or disable all advanced workload options when starting the node. |
DataStax Graph (DSE Graph) |
Enables Graph workloads. |
DSE Search |
Enables a DSE-integrated Apache Solr™ cluster |
DSE Analytics with Apache Spark |
Enables a DSE-integrated Spark cluster, and starts the Spark master service. |
Bring Your Own Spark (BYOS) |
Leverages an external Spark cluster from a vendor other than DataStax. To prevent the node from attempting to join a DSE integrated Spark cluster, a BYOS node must not run in DSE Analytics mode.
For package installations, set For combined BYOS and DSE Graph nodes, enable only the Graph type. |
Allow analytics jobs to be performed using CQL queries on DSE Search indexes. Can be resource intensive. Test before enabling in production. |
|
Combined Analytics, Graph, and Search |
Enables all advanced workloads. Can be resource intensive. Test before enabling in production. |
Start and stop nodes with Mission Control
|
Nodes managed by Mission Control must be started and stopped using Mission Control. Don’t start or stop them manually. The operator attempts to reconcile the actual state of the nodes with the desired state defined in Mission Control; manually stopped or started nodes will be reverted to the desired state by the operator. |
For clusters managed by Mission Control, use Mission Control to start and stop nodes.
Start and stop nodes with DSE OpsCenter
For clusters managed by DSE OpsCenter:
Start DSE as a standalone process (tarball)
When installed from a binary tarball, DSE runs as a standalone process.
-
Run
bin/dse cassandrawith the desired node type flags.Run this command from your DSE installation directory.
For example, to start a default database (transactional/
cassandra) node:INSTALL_DIRECTORY/bin/dse cassandraThe following table describes the command line options for each node type. For multi-type nodes, use spaces to separate each type flag. For more information, see Node types (workloads).
Node/datacenter Command Transactional (
cassandra) onlybin/dse cassandraDataStax Graph
bin/dse cassandra -gDSE Search
bin/dse cassandra -sDSE Analytics
bin/dse cassandra -kSearchAnalytics
bin/dse cassandra -k -sAnalytics, Graph, and Search
bin/dse cassandra -k -g -sBring Your Own Spark (BYOS)
bin/dse cassandraGraph and BYOS
bin/dse cassandra -g -
To start multiple nodes, repeat the previous command on each node.
-
Check node and cluster state to verify that the node is running.
Start DSE as a service (package)
When installed from an RHEL or Debian package, DSE runs as a service.
Package installations include start and stop scripts for the DSE service.
Binary tarballs don’t include these scripts.
-
Set the node type in the DSE start-up scripts in
/etc/default/dseor/etc/init.d.The default configuration starts a transactional-only node with no advanced workloads enabled.
To enable advanced workloads, set the relevant environment variables to
1. Unspecified workloads are disabled. If you prefer to explicitly disable a workload, set it to0.The following table describes the environment variables for different node types. For more information, see Node types (workloads).
Node type Environment variables Example Transactional (
cassandra) onlySet all types to
0or omit them.Explicitly disable advanced workloads:
SPARK_ENABLED=0 SOLR_ENABLED=0 GRAPH_ENABLED=0Or use the implied form:
# No node type entriesGRAPH_ENABLED=1SPARK_ENABLED=0 SOLR_ENABLED=0 GRAPH_ENABLED=1DSE Search
SOLR_ENABLED=1SPARK_ENABLED=0 SOLR_ENABLED=1 GRAPH_ENABLED=0DSE Analytics
SPARK_ENABLED=1SPARK_ENABLED=1 SOLR_ENABLED=0 GRAPH_ENABLED=0SearchAnalytics
SPARK_ENABLED=1andSOLR_ENABLED=1SPARK_ENABLED=1 GRAPH_ENABLED=0 SOLR_ENABLED=1Analytics, Search, and Graph
Set all types to
1.SPARK_ENABLED=1 GRAPH_ENABLED=1 SOLR_ENABLED=1Bring Your Own Spark (BYOS)
DSE Analytics must be disabled with
SPARK_ENABLED=0(or omitted).BYOS only:
SPARK_ENABLED=0 SOLR_ENABLED=0 GRAPH_ENABLED=0BYOS with DataStax Graph:
SPARK_ENABLED=0 SOLR_ENABLED=0 GRAPH_ENABLED=1 -
Start DSE on a node:
sudo service dse start -
To start multiple nodes, repeat the previous command on each node.
-
Check node and cluster state to verify that the node is running.
Stop a node
The commands to stop DSE on a node depend on your installation method:
- Package installations
-
-
Run
nodetool drainto flush commit logs to disk before stopping the node:nodetool drainIf you disabled durable writes (not recommended), you must run drain the node to prevent data loss.
When durable writes are enabled (the default), draining the node is beneficial when restarting nodes. Because the commit logs are not replayed, the startup process is faster.
-
Stop the DSE service:
sudo service dse stopFor RHEL package installations with systemd, run
systemctl stopwith the full node ID, including thedse-prefix. The default node ID isdse.systemctl stop FULL_NODE_ID
-
- Tarball installations
-
Stop the DSE process with
cassandra-stop. You don’t need to runnodetool drainbecausecassandra-stopautomatically drains the node before stopping it. If necessary, runcassandra-stopwithsudo.INSTALL_DIRECTORY/bin/dse cassandra-stop
Check node and cluster state
After you start or stop nodes, run nodetool status to check the state of the cluster.
You can also use dsetool status.
- Package installations
-
nodetool status - Tarball installations
-
INSTALL_DIRECTORY/bin/nodetool status
The output lists all nodes in the cluster, separated by datacenter. The first column reports the state and status of the node. The contents of the other columns varies by workload type, use of virtual nodes (vnodes), and other cluster configuration options. For example:
- Transactional node with vnodes
-
With vnodes, the
Tokenscolumn indicates the number of virtual nodes assigned to each physical node:Datacenter: DC1 =============== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.0.0.1 97.87 KiB 16 100.0% e4b7eb22-31d0-43ed-95e0-97915efcf58d RAC1The
Ownscolumn reports the percentage of data in the cluster that the node is responsible for. A?indicates that the ownership information is unavailable or undetermined, such as when bootstrapping a new node or rebalancing the cluster. - DSE Analytics node without vnodes
-
Without vnodes, the
Tokencolumn lists the specific token value assigned to each physical node:Datacenter: Analytics ===================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Owns Host ID Token Rack UN 172.16.222.136 103.24 KB ? 3c1d0657-0990-4f78-a3c0-3e0c37fc3a06 1647352612226902707 rack1
When starting or stopping a node, the following status and state combinations are expected from healthy nodes.
If a node reports an unexpected status or fails to appear in the nodetool status output, see Troubleshoot issues with starting and stopping nodes.
| Operation | Status/State | Description |
|---|---|---|
Start a node |
|
The node is running and joined the cluster. |
Start a node |
|
When bootstrapping a node, the node temporarily reports as Run |
Stop a node |
|
The node is stopped and remains in the cluster. Stopped nodes don’t leave the cluster unless other operations occur, such as decommissioning the node. This status is uniquely reported when you run |
Stop a node |
|
The node is stopped and remains in the cluster. Stopped nodes don’t leave the cluster unless other operations occur, such as decommissioning the node. This status is reported for stopped nodes when you run |
Troubleshoot issues with starting and stopping nodes
The following errors might occur when starting or stopping nodes:
- Timed out while waiting for DSE to start
-
This error can indicate a genuine timeout, but it can also occur when the user running the service doesn’t have permission to read the required files. For more information, see DSE cannot start with YAML parsing error.
- Node missing from
nodetool statusoutput, ornodetool statusreports only the current node in a multi-node cluster -
This issue indicates that the node failed to join the cluster (ring). Common causes of this issue include:
Cause Description Resolution Seed node started after non-seed nodes
When starting a new cluster or performing a full cluster restart, you must start the seed nodes first. The seed nodes describe the cluster topology and help new nodes discover and join the cluster.
Stop all nodes, and then start the seed nodes before starting the non-seed nodes.
Advanced workload nodes not started in the correct order
Mixed-workload clusters must start nodes according to the node type hierarchy described in Considerations for starting a cluster.
Stop all nodes, and then start the nodes in the required order.
Mismatched versions or software
Unless you are in the process of upgrading, all nodes in a cluster must have the same version of DSE installed in the same way (tarball, Debian package, or RHEL package). Nodes might fail to join the ring if they are running different DSE versions, different DSE distributions, or other Cassandra-based database distributions.
Compare the node’s DSE version to the other nodes in the cluster, and then take action accordingly to align the versions.
Misconfiguration
Various configuration issues can prevent a node from joining the cluster, such as:
-
Incorrect or incomplete seed node topology
-
Mismatched cluster names
-
Non-seed node cannot reach seed node
-
Incorrect or mismatched hostnames, IP addresses, or port numbers
-
Intentional node isolation is enabled, such as
write_surveymode
Investigate the cluster and node logs, configuration files, and startup options (JVM properties).
Network connectivity
Outages, timeouts, or firewall rules can block communication between nodes.
Investigate network configurations and logs.
-
cassandra-stopfails due to missing Java process ID (PID)-
This error is exclusive to tarball installations;
cassandra-stopisn’t available to package installations. To resolve this error:-
Get the DSE PID:
ps auwx | grep dse -
Pass the PID to
cassandra-stop:bin/dse cassandra-stop -p PID
-