Replacing a running node

Two methods for replacing a node with a new node, such as when updating to newer hardware or performing proactive maintenance.

Steps to replace a node with a new node, such as when updating to newer hardware or performing proactive maintenance.

You can replace a running node in two ways:
Note: To change the IP address of a node, simply change the IP of node and then restart Cassandra. If you change the IP address of a seed node, you must update the -seeds parameter in the seed_provider list in each node's cassandra.yaml file.
The location of the cassandra.yaml file depends on the type of installation:
Cassandra package installations /etc/cassandra/cassandra.yaml
Cassandra tarball installations install_location/cassandra/conf/cassandra.yaml

Adding a node and then decommissioning the old node

You must prepare and start the replacement node, integrate it into the cluster, and then decommission the old node.

Procedure

Be sure to use the same version of Cassandra on all nodes in the cluster.
  1. Prepare and start the replacement node, as described in Adding nodes to an existing cluster.
    Note: If not using vnodes, see Adding single-token nodes to a cluster.
  2. Confirm that the replacement node is alive:
    The status should show:
    • nodetool ring: Up
    • nodetool status: UN
  3. Note the Host ID of the original node; it is used in the next step.
  4. Using the Host ID of the original node, decommission the original node from the cluster using the nodetool decommission command.
  5. Run nodetool cleanup on all the other nodes in the same datacenter.

Using nodetool to replace a running node

This method allows you to replace a running node while avoiding streaming the data twice or running cleanup.

CAUTION: If using a consistency level of ONE, you risk losing data because the node might contain the only copy of a record. Be absolutely sure that no application uses consistency level ONE.

Procedure

  1. Stop Cassandra on the node to be replaced.
  2. Follow the instructions for replacing a dead node using the old node’s IP address for -Dcassandra.replace_address.
  3. Ensure that consistency level ONE is not used on this node.