Adding single-token nodes to a cluster

Steps for adding nodes in single-token architecture clusters, not clusters using Virtual nodes.

To add capacity to a cluster, introduce new nodes in stages or by adding an entire datacenter. Use one of the following methods:

Add capacity by doubling the cluster size: Adding capacity by doubling (or tripling or quadrupling) the number of nodes is less complicated when assigning tokens. Using this method, existing nodes keep their existing token assignments, and the new nodes are assigned tokens that bisect (or trisect) the existing token ranges.
Add capacity for a non-uniform number of nodes: When increasing capacity with this method, you must recalculate tokens for the entire cluster, and assign the new tokens to the existing nodes.

Only add new nodes to the cluster. A new node is a system in which DataStax Enterprise (DSE) has never started. The node must have absolutely NO PREVIOUS DATA in the data directory, saved_caches, commitlog, and hints. Adding nodes previously used for testing or that have been removed from another cluster, merges the older data into the cluster and may cause data loss or corruption.

For DataStax Enterprise clusters, you can use OpsCenter to rebalance a cluster.

Where is the cassandra.yaml file?

The location of the cassandra.yaml file depends on the type of installation:

Installation Type Location

Installation Type	Location
Package installations + Installer-Services installations	`/etc/dse/cassandra/cassandra.yaml`
Tarball installations + Installer-No Services installations	`<installation_location>/resources/cassandra/conf/cassandra.yaml`

Package installations + Installer-Services installations

/etc/dse/cassandra/cassandra.yaml

Tarball installations + Installer-No Services installations

<installation_location>/resources/cassandra/conf/cassandra.yaml

Where is the cassandra-topology.properties file?

The location of the cassandra-topology.properties file depends on the type of installation:

Installation Type Location

Installation Type	Location
Package installations + Installer-Services installations	`/etc/dse/cassandra/cassandra-topology.properties`
Tarball installations + Installer-No Services installations	`<installation_location>/resources/cassandra/conf/cassandra-topology.properties`

Package installations + Installer-Services installations

/etc/dse/cassandra/cassandra-topology.properties

Tarball installations + Installer-No Services installations

<installation_location>/resources/cassandra/conf/cassandra-topology.properties

Where is the cassandra-rackdc.properties file?

The location of the cassandra-rackdc.properties depends on the type of installation:

Installation Type Location

Installation Type	Location
Package installations + Installer-Services installations	`/etc/dse/cassandra/cassandra-rackdc.properties`
Tarball installations + Installer-No Services	`<installation_location>/resources/cassandra/conf/cassandra-rackdc.properties`

Package installations + Installer-Services installations

/etc/dse/cassandra/cassandra-rackdc.properties

Tarball installations + Installer-No Services

<installation_location>/resources/cassandra/conf/cassandra-rackdc.properties

Procedure

Calculate the tokens for the nodes based on your expansion strategy using the Token Generating Tool.
Install DSE and configure DataStax Enterprise on each new node.
If DataStax Enterprise starts automatically, stop the node and clear the data.
Configure cassandra.yaml on each new node:
- auto_bootstrap: If false, set it to true.
  
  This option is not listed in the default cassandra.yaml configuration file and defaults to true.
- cluster_name
- listen_address/broadcast_address: Usually leave blank. Otherwise, use the IP address or host name that other nodes use to connect to the new node.
- endpoint_snitch
- initial_token: Set according to your token calculations.
  
  If this property has no value, the database assigns the node a random token range and results in a badly unbalanced ring.
- seed_provider: Make sure that the new node lists at least one seed node in the existing cluster.
  
  Seed nodes cannot bootstrap. Make sure the new nodes are not listed in the -seeds list. Do not make all nodes seed nodes. See Internode communications (gossip).
- Change any other non-default settings in the new nodes to match the existing nodes. Use the diff command to find and merge any differences between the nodes.
Depending on the snitch, assign the datacenter and rack names in the cassandra-topology.properties or cassandra-rackdc.properties for each node.
Start DSE on each new node in two minutes intervals with cassandra.consistent.rangemovement turned off:
```
dse cassandra -Dcassandra.consistent.rangemovement=false
```

The following operations are resource intensive and should be done during low-usage times.

After the new nodes are fully bootstrapped, use nodetool move to assign the new initial_token value to each node that requires one, one node at a time.
After all nodes have their new tokens assigned, run nodetool cleanup on each node in the cluster and wait for cleanup to complete on each node before doing the next node. This step removes the keys that no longer belong to the previously existing nodes.

Failure to run nodetool cleanup after adding a node may result in data inconsistencies including resurrection of previously deleted data.

Adding single-token nodes to a cluster

Procedure

Was this helpful?

Give Feedback