Replace a dead node

The procedure for replacing a dead node is generally the same for clusters using vnodes or a single-token architecture. There are a few extra steps required for specific architectures and when replacing dead seed nodes. These extra steps are indicated where necessary.

When replacing or adding nodes, only use new nodes. A new node is a system where DSE has never started. The node must have absolutely no previous data in the data, saved_caches, commitlog, and hints directories.

Data loss or corruption can occur if you use a recycled node, such as a node moved from another cluster, because the existing data on the recycled node is merged into the cluster.

Verify the node’s status

Run nodetool status to check the condition of the node that you want to replace. This command returns a two-letter code describing the status and state of each node in the cluster. For example, UN means a node is in Up status and Normal state.

To replace a node, the following conditions must be true:

  • The output of nodetool status shows the node in Down (D) status and Normal (N) state. The node you want to replace must not report a transitional state like Joining (J) or Leaving (L).

  • DSE isn’t running on the node. Specifically, the DSE Java process must be stopped or the host itself must be offline.

A node in Stopped (S) state isn’t the same as stopping DSE on the node. A node enters the Stopped state if the nodetool drain command is issued on the node itself, or if the disk policy is set to disk_failure_policy: stop and the policy is triggered due to disk issues. The Stopped state means that the DSE process is still running, and it can still respond to JMX commands, but connections are stopped for gossip (port 7000) and clients (port 9042).

The output of nodetool status depends on your cluster architecture, your version of DSE, and the node where you run the command. Nodes can report their own condition differently than the condition seen by other nodes in the cluster. Review the following information for your version of DSE to understand what to expect when a node is in Down status. To assist with migrations and upgrades, the following scenarios span patch releases of DSE 5.1, 6.0, 6.7, 6.8, and 6.9.

DSE versions Identifying nodes in Down (D) status

6.9.0 and later

6.8.26 and later

5.1.33 and later

A Down node can have one of the following states:

  • Normal (N)

  • Leaving (L)

  • Joining (J)

  • Moving (M)

  • Stopped (S)

If a node is Down and Stopped, the output of nodetool status shows DS when run on the node itself and DN when run from any other node in the cluster.

6.8 to 6.8.25

6.7.8 and later

6.0.12 and later

5.1.18 to 5.1.32

A node in Down status only reports Stopped (S) state in gossip and the nodetool status command.

Run nodetool ring to get detailed state information for Down nodes. You can check the status reported for the node’s IP on any token belonging to the node. In the following example, the node at 1.2.3.12 is Down and Leaving:

Datacenter: DC1
==========
Address   Rack   Status State   Load        Owns  Token
                                                   8932492
1.2.3.10  RACK3  Up     Normal  105.33 GiB  ?     -8332242
1.2.3.11  RACK2  Up     Normal  102.20 GiB  ?     -8236178
1.2.3.12  RACK1  Down   Leaving 110.43 GiB  ?     -8053138
1.2.3.10  RACK3  Up     Normal  105.33 GiB  ?     -7195762
...

6.7.0 to 6.7.7

6.0.0 to 6.0.11

5.1.0 to 5.1.17

A Down node can have one of the following states:

  • Normal (N)

  • Leaving (L)

  • Joining (J)

  • Moving (M)

The Stopped state did not exist in these versions. If a node was Down and in a condition equivalent to the future Stopped state, the output of nodetool status showed UN when run on the node itself and DN when run from any other node in the cluster.

Replace the dead node

  1. Get the following information from the dead node:

    • IP address.

    • Datacenter and rack information:

    • Single-token architecture only (not vnodes): initial_token value from cassandra.yaml.

  2. Add the replacement node to the network and record its IP address.

    Make sure the replacement node meets the hardware and system requirements for your workloads.

  3. Install DSE on the new node but don’t start DSE.

    You must install the same DSE version (major, minor, and patch) on all nodes in a cluster.

  4. If DSE starts automatically on the node, stop DSE and then clear the data that was added automatically on startup.

    The node must be clear of data before joining the cluster as a replacement node.

  5. If the dead node was a seed node, change the cluster’s seed node configuration on each node:

    1. In the cassandra.yaml file for each node, remove the IP address of the dead node from the - seeds list in the seed-provider property.

    2. If a replacement seed node is needed, add the new node’s IP address to the - seeds list of the other nodes.

    3. Run nodetool reloadseeds to force the node to read the changes to the - seeds list in the cassandra.yaml file.

    Don’t make every node a seed node because it increases maintenance and reduces gossip performance. Gossip optimization isn’t critical, but DataStax recommends a small seed list with two or three seed nodes per datacenter.

  6. From a healthy node’s cassandra.yaml file, get the cluster_name, endpoint_snitch, and any other non-default values that you need to configure the new node.

    Use the diff command to compare configuration files for healthy nodes with the default configuration files.

  7. Configure the new node’s cassandra.yaml file using the values that you gathered earlier.

    In particular, make sure the following settings are configured correctly:

    • auto_bootstrap: If set to false for other nodes in the cluster, set it to true on the replacement node. This option isn’t explicitly set in the default cassandra.yaml configuration file, and it defaults to true.

    • cluster_name

    • initial_token: Only required for single-token architectures.

    • - seeds list

      If the new node is a seed node, make sure it is not listed in its own - seeds list.

  8. Add the rack and datacenter configuration to the snitch configuration files on all nodes in the cluster:

    Only add the new node to the snitch configuration files. Do not remove the original entry for the dead node yet.

    • If the cluster uses the GossipingPropertyFileSnitch, Ec2Snitch, Ec2MultiRegionSnitch, or GoogleCloudSnitch: Add the dead node’s rack and datacenter assignments to the cassandra-rackdc.properties file on the replacement node.

    • If the cluster uses the PropertyFileSnitch:

      1. Copy the cassandra-topology.properties file from an existing node to the replacement node.

      2. In each node’s cassandra-topology.properties file, add an entry with the new node’s IP address and the dead node’s rack and datacenter assignments.

      3. Delete the cassandra-topology.properties file from the dead node.

  9. Start DSE on the replacement node with the following start up options in addition to any other startup options that you typically used for the dead node, such as advanced workload types:

    • -Dcassandra.replace_address_first_boot: The IP address of the dead node.

    • -Ddse.consistent_replace: The consistency level to use when replacing the node. Allowed values are ONE, QUORUM (default), or LOCAL_QUORUM. Typically, this should match the consistency level of read requests.

      ONE requires the least system resources but the replacement node might stream data from an inconsistent replica, rather than comparing data from a quorum of replicas. Until you run repair, reads against the replacement node might return stale data.

    For additional control over the data streamed to the replacement node or streaming performance, you can set consistent_replace.parallelism, consistent_replace.retries, and consistent_replace.whitelist. For more information, see Apache Cassandra system properties.

    Package installations

    Add the options to the jvm-server.options file, and then start the node:

    JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address_first_boot=ADDRESS_OF_DEAD_NODE"
JVM_OPTS="$JVM_OPTS -Ddse.consistent_replace=QUORUM"
    Tarball installations

    Start the node with the replacement options:

    sudo INSTALL_DIRECTORY/bin/dse cassandra \
  -Dcassandra.replace_address_first_boot=ADDRESS_OF_DEAD_NODE \
  -Ddse.consistent_replace=QUORUM

  10. Run nodetool status to verify that the new node has bootstrapped successfully.

  11. For package installations, remove replace_address_first_boot and any consistent_replace options from jvm-server.options.

  12. If using the PropertyFileSnitch, wait at least 72 hours, and then remove the dead node’s IP address from the cassandra-topology.properties file on all nodes in the cluster.

    This ensures that dead node’s information is removed from gossip. If removed from the property file too soon, problems can occur. Use nodetool gossipinfo to check the gossip status. The node is still in gossip until LEFT status disappears.

    This step isn’t required when using other snitches because the cassandra-rackdc.properties file doesn’t contain IP addresses.

Was this helpful?

Give Feedback

How can we improve the documentation?

© Copyright IBM Corporation 2026 | Privacy policy | Terms of use |  Manage Privacy Choices

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: Contact IBM