Enabling virtual nodes on an existing production cluster

Steps and recommendations for enabling virtual nodes (vnodes) on an existing production cluster.

For production clusters, enabling virtual nodes (vnodes) has less impact on performance if you bring up a another data center configured with vnodes already enabled and let Cassandra automatic mechanisms distribute the existing data into the new nodes. Using the shuffle utility can be an expensive process on a running production system because the data to be shuffled around. Bootstrapping a new data center is a much safer way to enable vnodes.

Warning: Using the shuffle utility is not recommended.

Procedure

  1. Add a new data center to the cluster.
  2. Once the new data center with vnodes enabled is up, switch your clients to use the new data center.
  3. Run a full repair with nodetool repair.

    This step ensures that after you move the client to the new data center that any previous writes are added to the new data center and that nothing else, such as hints, is dropped when you remove the old data center.

  4. Update your schema to no longer reference the old data center.
  5. Remove the old data center from the cluster.