Steps to replace a node that has died for some reason, such as hardware failure.
Be sure to install the same version of Cassandra as is installed on the other nodes in the cluster. See Installing prior releases.
Confirm that the node is dead using nodetool
The nodetool command shows a down status for the dead node (DN):
- Note the Address of the dead node; it is used in step 5.
- Install Cassandra on the new node, but do not start Cassandra.
Set the following properties in the
cassandra.yaml and, depending on the snitch,
the cassandra-topology.properties or
- auto_bootstrap - This property is not listed in the default cassandra.yaml configuration file, but it might have been added and set to false by other operations. If it is not defined in cassandra.yaml, Cassandra uses true as a default value. For this operation, search for this property in the cassandra.yaml file. If it is present, set it to true or delete it..
- cluster_name - The name of the cluster the new node is joining.
- listen_address/broadcast_address - Can usually be left blank. Otherwise, use IP address or host name that other Cassandra nodes use to connect to the new node.
- endpoint_snitch - The snitch Cassandra uses for locating nodes and routing requests.
- num_tokens - The number of vnodes to assign to the node. If the hardware capabilities vary among the nodes in your cluster, you can assign a proportional number of vnodes to the larger machines.
- seeds - Determines
which nodes the new node contacts to learn about the cluster and establish
the gossip process. Make sure that the -seeds list includes
the address of at least one node in the existing cluster.Note: This new node will not bootstrap if it is listed as a seed node. Make sure the new node's address is not listed in the -seeds list. For more information about seed nodes, see Internode communications (gossip).
To add the new node as a seed node, complete these steps, then go on to Promoting a new node to a seed node.
- Check the
cassandra.yaml file and
cassandra-rackdc.properties files in other nodes in
the cluster for any non-default settings, and make sure to replicate these
settings on the new node.Note: Use the diff command to find and merge (by head) any differences between existing and new nodes.
Start the replacement node with the replace_address option:
- Package installations: Add the following option to
- Tarball installations: Start Cassandra with this
$ sudo bin/cassandra -Dcassandra.replace_address=address_of_dead_node
- Package installations: Add the following option to /usr/share/cassandra/cassandra-env.sh file:
- If using a packaged install, after the new node finishes bootstrapping, remove the option you added in step 5.
- Wait at least 72 hours and then remove the old node's IP address from the cassandra-topology.properties or cassandra-rackdc.properties file.
Remove the node.
The location of the cassandra-topology.properties file depends on the type of installation:
Package installations /etc/cassandra/cassandra-topology.properties Tarball installations install_location/conf/cassandra-topology.propertiesThe location of the cassandra-rackdc.properties file depends on the type of installation: Package installations /etc/cassandra/cassandra-rackdc.properties Tarball installations install_location/conf/cassandra-rackdc.propertiesThe location of the cassandra.yaml file depends on the type of installation: Package installations /etc/cassandra/cassandra.yaml Tarball installations install_location/resources/cassandra/conf/cassandra.yaml