Replacing a dead node or dead seed node
Steps to replace a node that has died for some reason, such as hardware failure.
The procedure for replacing a dead node is the same for vnodes and single-token nodes. Extra steps are required for replacing dead seed nodes.
Warning: Only add new nodes to the
cluster. A new node is a system in which the DataStax Distribution of Apache Cassandra™ (DDAC) has never started. The
node must have absolutely NO PREVIOUS DATA in the data
directory, saved_caches, commitlog, and hints. Adding nodes
previously used for testing or that have been removed from
another cluster, merges the older data into the cluster and
may cause data loss or corruption.
cassandra.yaml
- The cassandra.yaml file is located in the installation_location/conf directory.
cassandra-rackdc.properties
- The cassandra-rackdc.properties file is located in the installation_location/conf directory.
cassandra-topology.properties
- The cassandra-topology.properties file is located in the installation_location/conf directory.
cassandra-env.sh
- The cassandra-env.sh file is located in the installation_location/conf directory.
Procedure
-
Run nodetool status to verify that the node
is dead (
DN
). - Record the datacenter, address, and rack settings of the dead node; you will use these later.
- Add the replacement node to the network and record its IP address.
-
If the dead node was a seed node, change the cluster's seed node configuration
on each node:
-
On an existing node, gather setting information for the new node from the
cassandra.yaml file:
cluster_name
endpoint_snitch
- Other non-default settings: Use the diff tool to compare current settings with default settings.
-
Gather rack and datacenter information:
- If the cluster uses the PropertyFileSnitch, record the rack and data assignments listed in the cassandra-topology.properties file, or copy the file to the new node.
- If the cluster uses the GossipingPropertyFileSnitch, Ec2Snitch, Ec2MultiRegionSnitch, or GoogleCloudSnitch, record the rack and datacenter assignments in the dead node's cassandra-rackdc.properties file.
-
Make sure that the new node meets all prerequisites and then install Cassandra on the new
node, but do not start Cassandra.
Note: Be sure to install the same version of Cassandra as is installed on the other nodes in the cluster. If not using the latest version, see Installing DataStax Distribution of Apache Cassandra 3.11 patch releases.
- If Cassandra automatically started on the node, stop and clear the data that was added automatically on startup.
-
Add values to the following properties in
cassandra.yaml file from the information you
gathered earlier:
- auto_bootstrap: If this setting exists and is set to
false
, set it totrue
. (This setting is not included in the default cassandra.yaml configuration file.) - cluster_name
- seed listWarning: If the new node is a seed node, make sure it is not listed in its own
- seeds
list.
- auto_bootstrap: If this setting exists and is set to
-
Add the rack and datacenter configuration:
- If the cluster uses the GossipingPropertyFileSnitch, Ec2Snitch, and Ec2MultiRegionSnitch or GoogleCloudSnitch:
- Add the dead node's rack and datacenter assignments to the
cassandra-rackdc.properties
file on the replacement node.Note: Do not remove the entry for the dead node's IP address yet.
- Delete the cassandra-topology.properties file.
- Add the dead node's rack and datacenter assignments to the
cassandra-rackdc.properties
file on the replacement node.
- If the cluster uses the PropertyFileSnitch:
- Copy the cassandra-topology.properties file from an existing node, or add the settings to the local copy.
- Edit the file to add an entry with the new node's IP address and the dead node's rack and datacenter assignments.
- If the cluster uses the GossipingPropertyFileSnitch, Ec2Snitch, and Ec2MultiRegionSnitch or GoogleCloudSnitch:
-
Start the new node with the
replace_address
option, passing in the IP address of the dead node.
-
Package and Installer-Services installations:
- Add the following option to
cassandra-env.sh
file:
JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=address_of_dead_node
- Start the node.
- After the node bootstraps, remove the
replace-address
parameter from cassandra-env.sh. - Restart the node.
- Add the following option to
cassandra-env.sh
file:
Start DataStax Enterprise from the installation_location with this option:
sudo installation_location/bin/dse cassandra -Dcassandra.replace_address=address_of_dead_node
-
-
Run nodetool status to verify that the new node has
bootstrapped successfully.
Path:
installation_location/bin
-
In environments that use the PropertyFileSnitch, wait at least 72 hours and
then remove the old node's IP address from the
cassandra-topology.properties file.
CAUTION: This ensures that old node's information is removed from gossip. If removed from the property file too soon, problems may result. Use nodetool gossipinfo to check the gossip status. The node is still in gossip until LEFT status disappears.Note: The cassandra-rackdc.properties file does not contain IP information; therefore this step is not required when using other snitches, such as GossipingPropertyFileSnitch.