Replacing a dead node

Steps to replace a node that has died for some reason, such as hardware failure.

You must prepare and start the replacement node, integrate it into the cluster, and then remove the dead node.

Procedure

  1. Confirm that the node is dead using nodetool status:

    The nodetool command shows a down status for the dead node (DN):

  2. Note the Address of the dead node; it is used in the last step.
  3. Install Cassandra on the new node, but do not start Cassandra.

    If you used the Debian/Ubuntu install, Cassandra starts automatically and you must and stop the node and clear the data.

  4. Set the following properties in the cassandra.yaml and cassandra-topology.properties configuration files:
    • auto_bootstrap - If this option has been set to false, you must set it to true. This option is not listed in the default cassandra.yaml file and is set to true by default.
    • cluster_name - the name of the cluster the new node is joining.
    • listen_address/broadcast_address - the IP address or host name that other Cassandra nodes use to connect to the new node.
    • endpoint_snitch -the snitch Cassandra uses for locating nodes and routing requests.
    • num_tokens -the number of vnodes to assign to the node. If the hardware capabilities vary among the nodes in your cluster, you can assign a proportional number of vnodes to the larger machines.
    • seed_provider - the - seeds list in this setting determines which nodes the new node should contact to learn about the cluster and establish the gossip process. Change other non-default settings you have made to your existing cluster in the cassandra.yaml file and cassandra-topology.properties files. Use the diff command to find and merge (by head) any differences between existing and new nodes.
  5. Start the replacement node with the replace_address option:
    • Packaged installs: Add the following option to the cassandra-env.sh file:
      JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=address_of_dead_node
    • Tarball installs: Start Cassandra with this option:
      sudo bin/cassandra -Dcassandra.replace_address=address_of_dead_node
  6. If using a packaged install, after the new node finishes bootstrapping, remove the option you added in the previous step.