Steps to replace a node that has died for some reason, such as hardware
failure.
Steps to replace a node that has died for some reason, such as hardware failure.
Prepare and start the replacement node, then attach it to the cluster. After the
replacement node is running in the cluster, remove the dead
node.
Replacing a dead seed node
- Promote an existing node to a seed node by adding its IP address to -seeds list and remove
(demote) the IP address of the dead seed node from the
file for each node in the
cluster.
- Replace the dead node, as described in the next section.
Replacing a dead node
You must prepare and start the replacement node, integrate it into the cluster, and
then remove the dead node.
Procedure
Be sure to install the same version of Cassandra as is
installed on the other nodes in the cluster. See Installing prior releases.
-
Confirm that the node is dead using nodetool
status:
The nodetool command shows a down status for the dead node (DN):
-
Note the Address of the dead node; it is used in
step 5.
-
Install Cassandra on the new node, but do not start Cassandra.
If using the Debian/Ubuntu install, Cassandra starts
automatically and you must stop the node
and clear the
data.
-
Set the following properties in the
and, depending on the snitch,
the or
configuration
files:
- auto_bootstrap
- If this option has been set to false, you must set it to true. This
option is not listed in the default cassandra.yaml configuration file and defaults to
true.
- cluster_name - The
name of the cluster the new node is joining.
- listen_address/broadcast_address
- Can usually be left blank. Otherwise, use IP address or host name that
other Cassandra nodes use to connect to the new node.
-
endpoint_snitch - The snitch
Cassandra uses for locating nodes and routing requests.
-
num_tokens - The number
of vnodes to assign to the node. If the hardware capabilities vary among the
nodes in your cluster, you can assign a proportional number of vnodes to the
larger machines.
- seed_provider - Make
sure that the new node lists at least one node in the existing cluster. The
-seeds list determines which nodes the new node should
contact to learn about the cluster and establish the gossip
process.
- Change any other non-default settings
you have made to your existing cluster in the
cassandra.yaml file and
cassandra-topology.properties or
cassandra-rackdc.properties files. Use the
diff command to find and merge (by head) any
differences between existing and new nodes.
-
Start the replacement node with the replace_address option:
-
If using a packaged install, after the new node finishes bootstrapping, remove
the option you added in step 5.
What's next
- Remove the old node's IP address from the
or
file.
CAUTION:
Wait at least 72 hours to ensure that old node
information is removed from
gossip. If removed from the property file too soon, problems may
result.
- Remove the node.
The location of the
cassandra-topology.properties file depends on the type of installation:
Package installations |
/etc/cassandra/cassandra-topology.properties |
Tarball installations |
install_location/conf/cassandra-topology.properties |
Windows installations |
C:\Program Files\DataStax
Community\apache-cassandra\conf\cassandra-topology.properties |
The location of the
cassandra-rackdc.properties file depends on the type of installation:
Package installations |
/etc/cassandra/cassandra-rackdc.properties |
Tarball installations |
install_location/conf/cassandra-rackdc.properties |
Windows installations |
C:\Program Files\DataStax
Community\apache-cassandra\conf\cassandra-rackdc.properties |
The location of the
cassandra.yaml file depends on the type of installation:
Package installations |
/etc/cassandra/cassandra.yaml |
Tarball installations |
install_location/resources/cassandra/conf/cassandra.yaml |
Windows installations |
C:\Program Files\DataStax
Community\apache-cassandra\conf\cassandra.yaml |
The location of the
cassandra-env.sh file depends on the type of installation:
Package installations |
/etc/cassandra/cassandra-env.sh |
Tarball installations |
install_location/conf/cassandra-env.sh |
The location of the
cassandra-env.ps1 is:
Windows installations |
C:\Program Files\DataStax
Community\apache-cassandra\conf\cassandra-env.ps1 |