Adding vnodes to an existing cluster

Steps to add virtual nodes to a datacenter in an existing cluster.

Virtual nodes (vnodes) greatly simplify adding nodes to an existing cluster:

  • Calculating tokens and assigning them to each node is no longer required.
  • Rebalancing the nodes within a datacenter is no longer necessary because a node joining the datacenter assumes responsibility for an even portion of the data.

For a detailed explanation about how vnodes work, see Virtual nodes.

CAUTION: When adding multiple nodes to the cluster using the allocation algorithm, ensure that nodes are added one at a time. If nodes are added concurrently, the algorithm assigns the same tokens to different nodes.

Procedure

Be sure to use the same version of DataStax Distribution of Apache Cassandra™ (DDAC) on all nodes in the cluster. See Installing DataStax Distribution of Apache Cassandra patch releases.

  1. Install DataStax Enterprise on the new nodes, but do not start DDAC.
    Warning: If your DDAC installation started automatically, you must stop the node and clear the data.
  2. Copy the snitch properties file from another node in the same center datacenter to the node you are adding.

    cassandra-topology.properties

    • The cassandra-topology.properties file is located in the installation_location/conf directory.

    cassandra-rackdc.properties

    • The cassandra-rackdc.properties file is located in the installation_location/conf directory.

    cassandra.yaml

    • The cassandra.yaml file is located in the installation_location/conf directory.
  3. Set the following properties in the cassandra.yaml file:

    Manually add the auto_bootstrap setting if it does not exist in the cassandra.yaml. The other settings should exist in the default cassandra.yaml file, ensure that you uncomment and set.

    Warning: Seed nodes cannot bootstrap. Make sure the new node is not listed in the -seeds list. Do not make all nodes seed nodes. See Internode communications (gossip).
  4. Change any other non-default settings you have made to your existing cluster in the cassandra.yaml file and cassandra-topology.properties or cassandra-rackdc.properties files. Use the diff command to find and merge any differences between existing and new nodes.
  5. Start the bootstrap node, see Starting DataStax Distribution of Apache Cassandra.
  6. Verify that the node is fully bootstrapped using nodetool status. All other nodes must be up (UN) and not in any other state.
  7. After all new nodes are running, run nodetool cleanup on each of the previously existing nodes to remove the keys that no longer belong to those nodes. Wait for cleanup to complete on one node before running nodetool cleanup on the next node.

    Cleanup can be safely postponed for low-usage hours.