Configure virtual nodes (vnodes)
When partitions are distributed across multiple nodes through replicas, you can use virtual nodes (vnodes) to automatically allocate token ranges to each replica node.
Virtual nodes make it easier to scale clusters because you don’t need to manually calculate and assign token ranges to each node. The vnode architecture automatically rebalances token ranges across replicas when you add, remove, or replace nodes.
Requirements for vnodes
- Token architecture must be consistent within a datacenter
-
All nodes in a datacenter must use the same token architecture: vnodes or single-token. In a multi-datacenter cluster, each datacenter can vary. For example, one cluster with three datacenters could have two datacenters that use vnodes and one that uses a single-token architecture.
- When using vnodes, each node can have a different number of vnodes
-
For most use cases, DataStax recommends the same number of vnodes (
num_tokens) for all nodes in a datacenter. If your cluster design includes a mix of small, medium, and large machines, you might need to balance vnodes according to each node’s capacity. For example, nodes with less CPU power or memory can be assigned fewer vnodes than more powerful nodes in the same cluster. When the number of vnodes varies between nodes in a datacenter, the algorithm assigns a proportional number of ranges relative to other nodes in the datacenter. - Add nodes to the cluster sequentially
-
When adding multiple nodes to a cluster that uses vnodes, add nodes one at a time to ensure proper token allocation. Adding nodes concurrently can cause the algorithm to assign the same tokens to multiple nodes.
- Murmur3Partitioner is required
-
Legacy partitioners cannot be used with vnodes.
Determine the number of vnodes (num_tokens)
The number of vnodes is the number of token ranges to allocate to a given node.
This is set by num_tokens in cassandra.yaml.
For most use cases, DataStax recommends 8 or 16 vnodes. For clusters with more than 30 nodes, DataStax recommends 4 vnodes for better performance.
|
DataStax doesn’t recommend vnodes with DSE Search.
If you choose to use them, don’t exceed 8 vnodes, and make sure you set |
num_tokens |
Use case | Performance notes |
|---|---|---|
1 |
Maximum availability |
Maximum availability (fewest peers) and maximum data size per node, but inflexible scaling. Must always double or halve the cluster size to remain balanced. Fewer than 4 vnodes makes the cluster less elastic because each node is responsible for a significantly larger portion of token ranges. |
4 |
General purpose or clusters with more than 30 nodes |
A balance of elasticity and availability. Scaling requires adding or removing approximately 20 percent of the nodes to remain balanced. |
8 |
Performance focused |
In performance tests, 8 vnodes distributed token ranges between nodes with approximately 10 percent variance and minimal impact on performance. |
16 |
Highly elastic clusters |
Provides the greatest elasticity for frequent scaling but can be suboptimal for larger clusters. Not recommended for clusters over 50 nodes. |
Enable vnodes in a new node, datacenter, or cluster
To enable vnodes in a new node, datacenter, or cluster, set num_tokens and allocate_tokens_for_local_replication_factor in cassandra.yaml.
If you are adding a node to an existing datacenter, align these settings with the existing nodes.
-
Edit
cassandra.yaml. -
Uncomment
num_tokens, and then set the value to the appropriate number of vnodes. -
Uncomment
allocate_tokens_for_local_replication_factor, and then set it to the replication factor for the keyspaces in the local datacenter. If the replication factor varies, you alternate this setting for each node.This setting isn’t required but it is recommended for more balanced token range allocation.
allocate_tokens_for_local_replication_factoris recommended overallocate_tokens_for_keyspacebecause it doesn’t require a preexisting keyspace. By default (if theallocate_tokens_*parameters are not set), allocation is random. -
Make sure
initial_tokenis commented out or not set.
Migrate an existing cluster from a single-token architecture to vnodes
You cannot convert a cluster directly from a single-token architecture to vnodes. Instead, deploy a new datacenter in the cluster with vnodes enabled, and allow the existing datacenter to replicate to the new datacenter:
-
Add a new datacenter to the cluster with vnodes enabled, and allow replication to the new datacenter.
-
Once the nodes in the new datacenter are up and ready to receive traffic, switch your client applications to connect to the new datacenter.
-
Run a full repair with
nodetool repair.This ensures that any ongoing writes are replicated to the new datacenter and that maintenance objects, like hints, aren’t dropped when you remove the original datacenter.
-
Update your schema to no longer use the original datacenter. Only reference the new datacenter in schema properties like
replication.
Disable vnodes
|
If you do not use vnodes, you must make sure that each node is responsible for approximately the same amount of data.
To ensure that each node is responsible for an equal amount of data, assign each node an |
-
Edit
cassandra.yaml. -
Comment out
num_tokens,allocate_tokens_for_local_replication_factor, andallocate_tokens_for_keyspace. -
Uncomment
initial_token, and then set it to1or, for a multi-node cluster, a generated token.
