Steps to replace a node that has died for some reason, such as hardware
failure.
Steps to replace a node that has died for some reason, such as hardware failure.
Prepare and start the replacement node, then attach it to the cluster. After the
replacement node is running in the cluster, remove the dead
node.
Replacing a dead seed node
- Promote an existing node to a seed node by adding its IP address to -seeds list and remove
(demote) the IP address of the dead seed node from the
file for each node in the
cluster.
- Replace the dead node, as described in the next section.
Procedure
Be sure to install the same version of Cassandra as is
installed on the other nodes in the cluster. See Installing prior releases of DataStax Community.
-
Confirm that the node is dead using nodetool
status:
The nodetool command shows a down status for the dead node (DN):
-
Note the Address of the dead node; it is used in
step 5.
-
Install Cassandra on the new node, but do not start Cassandra.
If you used the Debian/Ubuntu install, Cassandra starts automatically and you
must and stop the node and clear the data.
-
Set the following properties in the
and, depending on the snitch,
the or
configuration
files:
-
Start the replacement node with the replace_address option:
-
If using a packaged install, after the new node finishes bootstrapping, remove
the option you added in step 5.
-
If using the cassandra-topology.properties wait at least
72 hours and then remove the old node's IP address from the file.
This ensures that old node information is removed from
gossip. If removed from the property
file too soon, problems may result. Use
nodetool
gossipinfo to check the gossip status. The node is still in gossip
until LEFT status disappears.
The location of the
cassandra-topology.properties file depends on the type of installation:
Package installations |
/etc/cassandra/cassandra-topology.properties |
Tarball installations |
install_location/conf/cassandra-topology.properties |
The location of the
cassandra-rackdc.properties file depends on the type of installation:
Package installations |
/etc/cassandra/cassandra-rackdc.properties |
Tarball installations |
install_location/conf/cassandra-rackdc.properties |
The location of the
cassandra.yaml file depends on the type of
installation:
Package installations |
/etc/cassandra/cassandra.yaml |
Tarball installations |
install_location/resources/cassandra/conf/cassandra.yaml |