Replacing a dead node in a single-token architecture cluster

Steps for replacing nodes in single-token architecture clusters, not vnodes.

Only add new nodes to the cluster. A new node is a system in which DataStax Enterprise (DSE) has never started. The node must have absolutely NO PREVIOUS DATA in the data directory, saved_caches, commitlog, and hints. Adding nodes previously used for testing or that have been removed from another cluster, merges the older data into the cluster and may cause data loss or corruption.

Where is the cassandra.yaml file?

The location of the cassandra.yaml file depends on the type of installation:

Installation Type Location

Package installations + Installer-Services installations


Tarball installations + Installer-No Services installations



  1. Run nodetool status to verify that the node is dead (DN).

    nodetool status arrow
    Verify that the node is dead
  2. Record the datacenter, address, and rack settings of the dead node to use later.

  3. Record the existing initial_token setting from the dead node’s cassandra.yaml.

  4. Add the replacement node to the network and record its IP address.

  5. If the dead node was a seed node, change the cluster’s seed node configuration on each node:

    1. In the cassandra.yaml file for each node, remove the IP address of the dead node from the - seeds list in the seed-provider property.

    2. If the cluster needs a new seed node to replace the dead node, add the new node’s IP address to the - seeds list of the other nodes.

      Making every node a seed node is not recommended because of increased maintenance and reduced gossip performance. Gossip optimization is not critical, but it is recommended to use a small seed list (approximately three nodes per datacenter).

  6. On an existing node, gather setting information for the new node from the cassandra.yaml file:

    • cluster_name

    • endpoint_snitch

    • Other non-default settings: Use the diff tool to compare current settings with default settings.

  7. Gather rack and datacenter information:

  8. Make sure that the new node meets all prerequisites and then install DSE on the new node, but do not start DSE.

    Be sure to install the same version of DSE as is installed on the other nodes in the cluster. If not using the latest version, see Installing DataStax Enterprise 5.1.x patch releases.

  9. If DSE automatically started on the node, stop and clear the data that was added automatically on startup.

  10. Add values to the following properties in cassandra.yaml file from the information gathered earlier:

    • auto_bootstrap: If this setting exists and is set to false, set it to true. (This setting is not included in the default cassandra.yaml configuration file.)

    • cluster_name

    • initial token

    • seed list

      If the new node is a seed node, make sure it is not listed in its own - seeds list.

  11. Add the rack and datacenter configuration:

    • If the cluster uses the GossipingPropertyFileSnitch, Ec2Snitch, and Ec2MultiRegionSnitch or GoogleCloudSnitch:

      1. Add the dead node’s rack and datacenter assignments to the file on the replacement node.

        Do not remove the entry for the dead node’s IP address yet.

      2. Delete the file.

    • If the cluster uses the PropertyFileSnitch:

      1. Copy the file from an existing node, or add the settings to the local copy.

      2. Edit the file to add an entry with the new node’s IP address and the dead node’s rack and datacenter assignments.

  12. Start the new node with the replace_address option, passing in the IP address of the dead node.

    • Package and Installer-Services installations:

      1. Add the following option to file:

        JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=address_of_dead_node
      2. Start the node.

      3. After the node bootstraps, remove the replace-address parameter from

      4. Restart the node.

    • Tarball and Installer-No Services installations:

      • Start DataStax Enterprise from the <installation_location> with this option:

        sudo bin/dse cassandra -Dcassandra.replace_address=address_of_dead_node
      • Start DataStax Enterprise from the <installation_location> with this option:

        sudo bin/dse cassandra -Dcassandra.replace_address=address_of_dead_node
  1. Run nodetool status to verify that the new node has bootstrapped successfully.

    Tarball and Installer No-Services path:

  2. In environments that use the PropertyFileSnitch, wait at least 72 hours and then remove the old node’s IP address from the file.

    This ensures that old node’s information is removed from gossip. If removed from the property file too soon, problems may result. Use nodetool gossipinfo to check the gossip status. The node is still in gossip until LEFT status disappears.

    The file does not contain IP information; therefore this step is not required when using other snitches, such as GossipingPropertyFileSnitch.

Was this helpful?

Give Feedback

How can we improve the documentation?

© 2024 DataStax | Privacy policy | Terms of use

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: +1 (650) 389-6000,