Adding a datacenter to a cluster

Add a datacenter to an existing cluster.

Complete the following steps to add a datacenter to an existing cluster.

cassandra-rackdc.properties

  • The cassandra-rackdc.properties file is located in the installation_location/conf directory.

cassandra.yaml

  • The cassandra.yaml file is located in the installation_location/conf directory.

cassandra-topology.properties

  • The cassandra-topology.properties file is located in the installation_location/conf directory.

Prerequisites

Important: Complete the prerequisite tasks outlined in Initializing a Cassandra cluster to prepare the environment.
If the new datacenter uses existing nodes from another datacenter or cluster, complete the following steps to ensure that old data will not interfere with the new cluster:
  1. If the nodes are behind a firewall, open the required portsopen the required ports for internal/external communication.
  2. Decommission each node that will be added to the new datacenter.
  3. Clear the data from Cassandra to completely remove application directories.
  4. Install Cassandra on each node.

Procedure

  1. Complete the following steps to prevent client applications from prematurely connecting to the new datacenter, and to ensure that the consistency level for reads or writes does not query the new datacenter:
    Warning: If client applications are not properly configured, they might connect to the new datacenter before it is online. Incorrect configuration results in connection exceptions, timeouts, and/or inconsistent data.
    1. Configure client applications to use the DCAwareRoundRobinPolicy.
    2. Direct clients to an existing datacenter. Otherwise, clients might try to access the new datacenter, which might not have any data.
    3. If using the QUORUM consistency level, change to LOCAL_QUORUM.
    4. If using the ONE consistency level, set to LOCAL_ONE.

    See the programming instructions for your driver.

  2. Configure every keyspace using SimpleStrategy to use the NetworkTopologyStrategy replication strategy, including (but not restricted to) the following keyspaces.

    If SimpleStrategy was used previously, this step is required to configure NetworkTopologyStrategy.

    1. Use ALTER KEYSPACEALTER KEYSPACE to change the keyspace replication strategy to NetworkTopologyStrategy for the following keyspaces.
      ALTER KEYSPACE keyspace_name WITH REPLICATION = 
      {'class' : 'NetworkTopologyStrategy', 'ExistingDC1' : 3};
      • All keyspaces created by users
    2. Use DESCRIBE SCHEMA to check the replication strategy of keyspaces in the cluster. Ensure that any existing keyspaces use the NetworkTopologyStrategy replication strategy.
      DESCRIBE SCHEMA ;
  3. In the new datacenter, install Cassandra on each new node. Do not start the service or restart the node.
    Important: Use the same version of DDAC on all nodes in the cluster.
  4. Configure properties in cassandra.yaml on each new node, following the configuration of the other nodes in the cluster.
    1. Configure node properties:
    2. Configure node architecture (all nodes in the datacenter must use the same type):

      Virtual node (vnode) allocation algorithm settings

      For more information, refer to Virtual node (vnode) configuration.

      Single-token architecture settings

  5. In the cassandra-rackdc.properties (GossipingPropertyFileSnitch) or cassandra-topology.properties (PropertyFileSnitch) file, assign datacenter and rack names to the IP addresses of each node, and assign a default datacenter name and rack name for unknown nodes.
    Note: Migration information: The GossipingPropertyFileSnitch always loads cassandra-topology.properties when the file is present. Remove the file from each node on any new cluster, or any cluster migrated from the PropertyFileSnitch.
    # Transactional Node IP=Datacenter:Rack
    110.82.155.0=DC_Transactional:RAC1
    110.82.155.1=DC_Transactional:RAC1
    110.54.125.1=DC_Transactional:RAC2
    
    # default for unknown nodes
    default=DC1:RAC1
    Note: After making any changes in the configuration files, you must the restart the node for the changes to take effect.
  6. Make the following changes in the existing datacenters.
    1. On nodes in the existing datacenters, update the -seeds property in cassandra.yaml to include the seed nodes in the new datacenter.
    2. Add the new datacenter definition to the cassandra.yaml properties file for the type of snitch used in the cluster. If changing snitches, see Switching snitches.
  7. After you have installed and configured Cassandra on all nodes, start the seed nodes one at a time, and then start the rest of the nodes:
  8. Rotate starting Cassandra through the racks until all the nodes are up.
  9. After all nodes are running in the cluster and the client applications are datacenter aware, use cqlsh to alter the keyspaces to add the desired replication in the new datacenter.
    ALTER KEYSPACE keyspace_name WITH REPLICATION = 
    {'class' : 'NetworkTopologyStrategy', 'ExistingDC1' : 3, 'NewDC2' : 2};
    Warning: If client applications, are not properly configured, they might connect to the new datacenter before it is online. Incorrect configuration results in connection exceptions, timeouts, and/or inconsistent data.
  10. Run nodetool rebuild on each node in the new datacenter, specifying the datacenter to rebuild from. This step replicates the data to the new datacenter in the cluster.
    nodetool rebuild -- datacenter_name
    CAUTION: You must specify an existing datacenter in the command line, or the new nodes will appear to rebuild successfully, but might not contain all anticipated data.

    Requests to the new datacenter with LOCAL_ONE or ONE consistency levels can fail if the existing datacenters are not completely in-sync.

    1. Use nodetool rebuild on one or more nodes at the same time. Run on one node at a time to reduce the impact on the existing cluster.
    2. Alternatively, run the command on multiple nodes simultaneously when the cluster can handle the extra I/O and network pressure.
  11. Check that your cluster is up and running:
    installation_location/bin/nodetool status
    Note: If the cluster has problems starting, look for starting DSE troubleshooting and other articles in the Support Knowledge Center.

Results

The datacenters in the cluster are now replicating with each other.
DC: datacenter1
==============================================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address         Load        Tokens    Owns    Host ID             Rack
UN 110.82.155.0    21.33 KB    256       33.3%   a9fa31c7-f3c0-...   RAC1
UN 110.82.155.1    21.33 KB    256       33.3%   f5bb416c-db51-...   RAC1
UN 110.54.125.1    21.33 KB    256       16.7%   b836748f-c94f-...   RAC1
datacenter2
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address         Load        Owns      Host ID               Tokens         Rack
UN 110.54.125.2    28.44 KB    13.0.%    e2451cdf-f070- ...    -922337....    RAC1
UN 110.82.155.2    44.47 KB    16.7%     f9fa427c-a2c5- ...    30745512...    RAC1 
UN 110.82.155.3    54.33 KB    23.6%     b9fc31c7-3bc0- ..-    45674488...    RAC1

datacenter2
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address         Load        Owns      Host ID               Tokens         Rack
UN 110.54.125.3    15.44 KB    50.2.%    e2451cdf-f070- ...    9243578....    RAC1
UN 110.82.155.4    18.78 KB    49.8.%    e2451cdf-f070- ...    10000          RAC1