Adding or replacing single-token nodes

About adding Capacity to an Existing Cluster 

Cassandra allows you to add capacity to a cluster by introducing new nodes to the cluster in stages and by adding an entire datacenter. When a new node joins an existing cluster, it needs to know:

• Its position in the ring and the range of data it is responsible for, which is assigned by the initial_token and the partitioner.
• The seed nodes to contact for learning about the cluster and establish the gossip process.
• The name of the cluster it is joining and how the node should be addressed within the cluster.
• Any other non-default settings made to cassandra.yaml on your existing cluster.

When you add one or more nodes to a cluster, you must calculate the tokens for the new nodes. Use one of the following approaches:

Add capacity by doubling the cluster size 

Adding capacity by doubling (or tripling or quadrupling) the number of nodes is less complicated when assigning tokens. Existing nodes can keep their existing token assignments, and new nodes are assigned tokens that bisect (or trisect) the existing token ranges. For example, when you generate tokens for six nodes, three of the generated token values will be the same as if you generated for three nodes. To clarify, you first obtain the token values that are already in use, and then assign the newly calculated token values to the newly added nodes.

Recalculate new tokens for all nodes and move nodes around the ring 

When increases capacity by a non-uniform number of nodes, you must recalculate tokens for the entire cluster, and then use nodetool move to assign the new tokens to the existing nodes. After all nodes are restarted with their new token assignments, run a nodetool cleanup to remove unused keys on all nodes. These operations are resource intensive and should be done during low-usage times.

Adding Nodes to a Cluster 

1. Install Cassandra on the new nodes, but do not start them.
2. Calculate the tokens for the nodes based on the expansion strategy you are using the Token Generating Tool.
3. Set the cassandra.yaml for the new nodes.
4. Set the initial_token according to your token calculations.
5. Start Cassandra on each new node. Allow two minutes between node initializations. You can monitor the startup and data streaming process using nodetool netstats.
6. After the new nodes are fully bootstrapped, assign the new initial_token property value to the nodes that required new tokens, and then run nodetool move new_token, one node at a time.
7. After all nodes have their new tokens assigned, run nodetool cleanup one node at a time for each node. Wait for cleanup to complete before doing the next node. This step removes the keys that no longer belong to the previously existing nodes.
   Note: Cleanup may be safely postponed for low-usage hours.

Adding a datacenter to a Cluster 

Before starting this procedure, please read the guidelines in Adding Capacity to an Existing Cluster above.

1. Ensure that you are using NetworkTopologyStrategy for all of your keyspaces.
2. For each new node, edit the configuration properties in the cassandra.yaml file:
   • Set auto_bootstrap to False.
   • Set the initial_token. Be sure to offset the tokens in the new datacenter, see Generating tokens.
   • Set the cluster name.
   • Set any other non-default settings.
   • Set the seed lists. Every node in the cluster must have the same list of seeds and include at least one node from each datacenter. Typically one to three seeds are used per datacenter.
3. Update either the cassandra-topology.properties (PropertyFileSnitch) or cassandra-rackdc.properties (GossipingPropertyFileSnitch) on all servers to include the new nodes. You do not need to restart.
   The location of the conf directory depends on the type of installation:

   • Cassandra Package installations: /etc/cassandra/conf
   • Cassandra Tarball installations: install_location/conf
4. Ensure that your client does not auto-detect the new nodes so that they aren't contacted by the client until explicitly directed. For example in Hector, set

   hostConfig.setAutoDiscoverHosts(false);

5. If using a QUORUM consistency level for reads or writes, check the LOCAL_QUORUM or EACH_QUORUM consistency level to make sure that the level meets the requirements for multiple datacenters.
6. Start the new nodes.
7. After all nodes are running in the cluster:
   1. Change the replication factor for your keyspace for the expanded cluster.
   2. Run nodetool rebuild on each node in the new datacenter.

Replacing a Dead Node 

1. Confirm that the node is dead using the nodetool ring command on any live node in the cluster.

   The nodetool ring command shows a Down status for the token value of the dead node:

   

2. Install Cassandra on the replacement node.
3. Remove any preexisting Cassandra data on the replacement node:

   sudo rm -rf /var/lib/cassandra/* 

4. Set auto_bootstrap: true. (If auto_bootstrap is not in the cassandra.yaml file, it automatically defaults to true.)
5. Set the initial_token in the cassandra.yaml file to the value of the dead node's token -1. Using the value from the above graphic, this is 28356863910078205288614550619314017621-1:

   initial_token: 28356863910078205288614550619314017620

6. Configure any non-default settings in the node's cassandra.yaml to match your existing cluster.
7. Start the new node.
8. After the new node has finished bootstrapping, check that it is marked up using the nodetool ring command.
9. Run nodetool repair on each keyspace to ensure the node is fully consistent. For example:

   nodetool repair -h 10.46.123.12 keyspace_name

10. Remove the dead node.