Replace a dead node

The procedure for replacing a dead node is generally the same for clusters using vnodes or a single-token architecture. There are a few extra steps required for specific architectures and when replacing dead seed nodes. These extra steps are indicated where necessary.

When replacing or adding nodes, only use new nodes. A new node is a system where HCD has never started. The node must have absolutely no previous data in the data, saved_caches, commitlog, and hints directories.

Data loss or corruption can occur if you use a recycled node, such as a node moved from another cluster, because the existing data on the recycled node is merged into the cluster.

Verify the node’s status

Run nodetool status to check the condition of the node that you want to replace. This command returns a two-letter code describing the status and state of each node in the cluster. For example, UN means a node is in Up status and Normal state.

To replace a node, the following conditions must be true:

  • The output of nodetool status shows the node in Down (D) status and Normal (N) state. The node you want to replace must not report a transitional state like Joining (J) or Leaving (L).

  • HCD isn’t running on the node. Specifically, the HCD Java process must be stopped or the host itself must be offline.

A node in Stopped (S) state isn’t the same as stopping HCD on the node. A node enters the Stopped state if the nodetool drain command is issued on the node itself, or if the disk policy is set to disk_failure_policy: stop and the policy is triggered due to disk issues. The Stopped state means that the HCD process is still running, and it can still respond to JMX commands, but connections are stopped for gossip (port 7000) and clients (port 9042).

The output of nodetool status depends on your cluster architecture and the node where you run the command. Nodes can report their own condition differently than the condition seen by other nodes in the cluster.

A HCD node in Down status can have one of the following states:

  • Normal (N)

  • Leaving (L)

  • Joining (J)

  • Moving (M)

  • Stopped (S)

If a node is Down and Stopped, the output of nodetool status shows DS when run on the node itself and DN when run from any other node in the cluster.

Replace the dead node

  1. Get the following information from the dead node:

    • IP address.

    • Datacenter and rack information:

    • Single-token architecture only (not vnodes): initial_token value from cassandra.yaml.

  2. Add the replacement node to the network and record its IP address.

    Make sure the replacement node meets the hardware and system requirements for your workloads.

  3. Install HCD on the new node but don’t start HCD.

    You must install the same HCD version (major, minor, and patch) on all nodes in a cluster.

  4. If HCD starts automatically on the node, stop HCD and then clear the data that was added automatically on startup.

    The node must be clear of data before joining the cluster as a replacement node.

  5. If the dead node was a seed node, change the cluster’s seed node configuration on each node:

    1. In the cassandra.yaml file for each node, remove the IP address of the dead node from the - seeds list in the seed-provider property.

    2. If a replacement seed node is needed, add the new node’s IP address to the - seeds list of the other nodes.

    3. Run nodetool reloadseeds to force the node to read the changes to the - seeds list in the cassandra.yaml file.

    Don’t make every node a seed node because it increases maintenance and reduces gossip performance. Gossip optimization isn’t critical, but DataStax recommends a small seed list with two or three seed nodes per datacenter.

  6. From a healthy node’s cassandra.yaml file, get the cluster_name, endpoint_snitch, and any other non-default values that you need to configure the new node.

    Use the diff command to compare configuration files for healthy nodes with the default configuration files.

  7. Configure the new node’s cassandra.yaml file using the values that you gathered earlier.

    In particular, make sure the following settings are configured correctly:

    • auto_bootstrap: If set to false for other nodes in the cluster, set it to true on the replacement node. This option isn’t explicitly set in the default cassandra.yaml configuration file, and it defaults to true.

    • cluster_name

    • initial_token: Only required for single-token architectures.

    • - seeds list

      If the new node is a seed node, make sure it is not listed in its own - seeds list.

  8. Add the rack and datacenter configuration to the snitch configuration files on all nodes in the cluster:

    Only add the new node to the snitch configuration files. Do not remove the original entry for the dead node yet.

    • If the cluster uses the GossipingPropertyFileSnitch, Ec2Snitch, Ec2MultiRegionSnitch, or GoogleCloudSnitch: Add the dead node’s rack and datacenter assignments to the cassandra-rackdc.properties file on the replacement node.

    • If the cluster uses the PropertyFileSnitch:

      1. Copy the cassandra-topology.properties file from an existing node to the replacement node.

      2. In each node’s cassandra-topology.properties file, add an entry with the new node’s IP address and the dead node’s rack and datacenter assignments.

      3. Delete the cassandra-topology.properties file from the dead node.

  9. Start the new node with the -Dcassandra.replace_address_first_boot start up option in addition to any other startup options that you typically used for the dead node.

    The value of replace_address_first_boot is the dead node’s IP address.

    Package installations

    Add the option to the jvm-server.options file, and then start the node:

    JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address_first_boot=ADDRESS_OF_DEAD_NODE"
    Tarball installations
    sudo INSTALL_DIRECTORY/bin/hcd cassandra \
  -Dcassandra.replace_address_first_boot=ADDRESS_OF_DEAD_NODE

  10. Run nodetool status to verify that the new node has bootstrapped successfully.

  11. For package installations, remove replace_address_first_boot from jvm-server.options.

  12. If using the PropertyFileSnitch, wait at least 72 hours, and then remove the dead node’s IP address from the cassandra-topology.properties file on all nodes in the cluster.

    This ensures that dead node’s information is removed from gossip. If removed from the property file too soon, problems can occur. Use nodetool gossipinfo to check the gossip status. The node is still in gossip until LEFT status disappears.

    This step isn’t required when using other snitches because the cassandra-rackdc.properties file doesn’t contain IP addresses.

Was this helpful?

Give Feedback

How can we improve the documentation?

© Copyright IBM Corporation 2026 | Privacy policy | Terms of use |  Manage Privacy Choices

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: Contact IBM