Replacing a dead node or dead seed node

Steps to replace a node that has died for some reason, such as hardware failure.

1. Confirm that the node is dead using nodetool status:

   The nodetool command shows a down status for the dead node (DN):

   

2. Note the Address of the dead node; it is used in step 5.
3. Install Cassandra on the new node, but do not start Cassandra.

   If using the Debian/Ubuntu install, Cassandra starts automatically and you must stop the node and clear the data.

4. Set the following properties in the cassandra.yaml and, depending on the snitch, the cassandra-topology.properties or cassandra-rackdc.properties configuration files:
   • auto_bootstrap - If this option has been set to false, you must set it to true. This option is not listed in the default cassandra.yaml configuration file and defaults to true.
   • cluster_name - The name of the cluster the new node is joining.
   • listen_address/broadcast_address - Can usually be left blank. Otherwise, use IP address or host name that other Cassandra nodes use to connect to the new node.
   • endpoint_snitch - The snitch Cassandra uses for locating nodes and routing requests.
   • num_tokens - The number of vnodes to assign to the node. If the hardware capabilities vary among the nodes in your cluster, you can assign a proportional number of vnodes to the larger machines.
   • seed_provider - Make sure that the new node lists at least one node in the existing cluster. The -seeds list determines which nodes the new node should contact to learn about the cluster and establish the gossip process.
     Note: Seed nodes cannot bootstrap. Make sure the new node is not listed in the -seeds list. Do not make all nodes seed nodes. Please read Internode communications (gossip).
   • Change any other non-default settings you have made to your existing cluster in the cassandra.yaml file and cassandra-topology.properties or cassandra-rackdc.properties files. Use the diff command to find and merge (by head) any differences between existing and new nodes.
5. Start the replacement node with the replace_address option:
   • Package installations: Add the following option to cassandra-env.sh file:

     JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=address_of_dead_node

   • Tarball installations: Start Cassandra with this option:

     sudo bin/cassandra -Dcassandra.replace_address=address_of_dead_node

6. If using a packaged install, after the new node finishes bootstrapping, remove the option you added in step 5.