Replace a running node

When replacing or adding nodes, only use new nodes. A new node is a system where DSE has never started. The node must have absolutely no previous data in the data, saved_caches, commitlog, and hints directories.

Data loss or corruption can occur if you use a recycled node, such as a node moved from another cluster, because the existing data on the recycled node is merged into the cluster.

You can replace a running DataStax Enterprise (DSE) node with a new node when updating to newer hardware or performing proactive maintenance. There are two ways to replace a running node:

Add a node and then remove the old node.
Replace a running node as though it were a dead node to avoid cleanup and repetitive streaming.

Add a node and then remove the old node

Prepare and start the replacement node, integrate it into the cluster, and then decommission the old node:

Prepare and start the replacement node according to your cluster architecture:
- Vnodes: Adding vnodes to an existing cluster
- Single-token architecture: Adding single-token nodes to a cluster
You must install the same DSE version (major, minor, and patch) on all nodes in a cluster.
Confirm that the replacement node is up and running:
- Vnodes: Run nodetool status, and make sure the replacement node’s status is UN (Up/Normal).
- Single-token architecture: Run nodetool ring, and make sure the replacement node’s status is Up.
From the output of the previous command, get the Host ID of the node that you are replacing, and then use that Host ID to decommission the node with the nodetool decommission command.
Run nodetool cleanup on all the other nodes in the same datacenter.

Failure to run nodetool cleanup after adding a node can result in data inconsistencies, including resurrection of previously deleted data.

Replace a running node without cleanup or repetitive streaming

To replace a running node without streaming the data twice or running cleanup, you can use the dead node replacement process:

Ensure that your applications and scripts don’t use consistency level ONE for any writes to the cluster where you are replacing a node.

Writes at consistency level ONE risk data loss because the replaced node might contain the only copy of a record. Be absolutely sure that no application uses consistency level ONE when using this replacement method.

Stop DSE on the node that will be replaced.
Follow the instructions for replacing a dead node using the old node’s IP address for -Dcassandra.replace_address.

Replace a running node

Add a node and then remove the old node

Replace a running node without cleanup or repetitive streaming

Was this helpful?

Give Feedback