Replacing a dead node or dead seed node

Steps to replace a node that has died for some reason, such as hardware failure.

1. Confirm that the node is dead using nodetool status:

   The nodetool command shows a down status for the dead node (DN):

   

2. Note the Address of the dead node; it is used in step 5.
3. Install Cassandra on the new node, but do not start Cassandra.

   If you used the Debian/Ubuntu install, Cassandra starts automatically and you must and stop the node and clear the data.

4. Set the following properties in the cassandra.yaml and, depending on the snitch, the cassandra-topology.properties or cassandra-rackdc.properties configuration files:
   • auto_bootstrap - This property is not listed in the default cassandra.yaml configuration file, but it might have been added and set to false by other operations. If it is not defined in cassandra.yaml, Cassandra uses true as a default value. For this operation, search for this property in the cassandra.yaml file. If it is present, set it to true or delete it..
   • cluster_name - The name of the cluster the new node is joining.
   • listen_address/broadcast_address - Can usually be left blank. Otherwise, use IP address or host name that other Cassandra nodes use to connect to the new node.
   • endpoint_snitch - The snitch Cassandra uses for locating nodes and routing requests.
   • num_tokens - The number of vnodes to assign to the node. If the hardware capabilities vary among the nodes in your cluster, you can assign a proportional number of vnodes to the larger machines.
   • seeds - Determines which nodes the new node contacts to learn about the cluster and establish the gossip process. Make sure that the -seeds list includes the address of at least one node in the existing cluster.
     Note: This new node will not bootstrap if it is listed as a seed node. Make sure the new node's address is not listed in the -seeds list. For more information about seed nodes, see Internode communications (gossip).

     To add the new node as a seed node, complete these steps, then go on to Promoting a new node to a seed node.

   • Check the cassandra.yaml file and cassandra-topology.properties or cassandra-rackdc.properties files in other nodes in the cluster for any non-default settings, and make sure to replicate these settings on the new node.
     Note: Use the diff command to find and merge (by head) any differences between existing and new nodes.
5. Start the replacement node with the replace_address option:
   • Cassandra Package installations: Add the following option to /usr/share/cassandra/cassandra-env.sh file:

     JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=address_of_dead_node

   • Cassandra Tarball installations: Start Cassandra with this option:

     sudo bin/cassandra -Dcassandra.replace_address=address_of_dead_node

6. If using a packaged install, after the new node finishes bootstrapping, remove the option you added in step 5.
7. If using the cassandra-topology.properties wait at least 72 hours and then remove the old node's IP address from the file.
   This ensures that old node information is removed from gossip. If removed from the property file too soon, problems may result. Use nodetool gossipinfo to check the gossip status. The node is still in gossip until LEFT status disappears.
   The location of the cassandra-topology.properties file depends on the type of installation:

   Package installations	/etc/cassandra/cassandra-topology.properties
   Tarball installations	install_location/conf/cassandra-topology.properties

   The location of the cassandra-rackdc.properties file depends on the type of installation:

   Package installations	/etc/cassandra/cassandra-rackdc.properties
   Tarball installations	install_location/conf/cassandra-rackdc.properties

   The location of the cassandra.yaml file depends on the type of installation:

   Package installations	/etc/cassandra/cassandra.yaml
   Tarball installations	install_location/resources/cassandra/conf/cassandra.yaml