Add a datacenter to an existing DataStax Enterprise (DSE) cluster

Database administrators can add one or more datacenters to:

Support additional workloads.
Solve latency issues.
Expand into new markets.
Add capacity so that their applications remain available.
Support new functionality.

Use Mission Control to create one or more datacenters. Do this from an existing cluster and bootstrap one datacenter at a time. With multiple datacenters, sort them in ascending order by datacenter name and select each datacenter (DC) in the list to create until all are added to the cluster.

Prerequisites

A prepared environment on either bare-metal/VM or an existing Kubernetes cluster.

Example

This example uses a Kubernetes cluster with a single east region datacenter set up in the Data Plane. The example extends it to include a second, west region datacenter set up in the Data Plane.

Workflow for User and operators

User submits a modified MissionControlCluster specifying a new datacenter in the west region and defining it to the Control Plane Kubernetes cluster.
Cluster-level operator picks up the modification and creates datacenter-level resources in the Kubernetes cluster in the east region where the nodes are to be created.
DC-level operator picks up datacenter-level resources and creates native Kubernetes objects representing the DSE nodes.
DC-level operator bootstraps one node at a time, balancing operations across racks and reporting progress.
Cluster-level operator updates keyspace replication settings on system keyspaces. User updates keyspace replication settings on user keyspaces.
Cluster-level operator runs a rebuild operation on system keyspaces.

Define and add a new datacenter to a cluster

Define the new datacenter (DC) criteria, for example:
- Target the region or Kubernetes (k8s) cluster, availability zones, etc.
- Target the workload (core Cassandra, Search, graph)

Modify the existing MissionControlCluster YAML file in the Control Plane cluster to add a new DC definition with three (3) nodes in the west region:

apiVersion: missioncontrol.datastax.com/v1beta2
kind: MissionControlCluster
metadata:
  name: demo
spec:
...
    datacenters:
      - metadata:
          name: dc1
        k8sContext: east
        size: 3
...
      - metadata:
          name: dc2
        k8sContext: west
        size: 3
        racks:
          - name: rack1
            nodeAffinityLabels:
              topology.kubernetes.io/zone: us-west1-c
          - name: rack2
            nodeAffinityLabels:
              topology.kubernetes.io/zone: us-west1-b
          - name: rack3
            nodeAffinityLabels:
              topology.kubernetes.io/zone: us-west1-a

Submit the updated MissionControlCluster YAML file to Kubernetes.
```
kubectl apply -f demo-dse.cassandratask.yaml
```
The K8ssandraCluster is updated. DC-level operators then create a CassandraDatacenter named dc2 in the west cluster.

Monitor the progress of adding a datacenter with the following command:

kubectl get k8ssandracluster demo -o yaml

Sample results

...
status:
  conditions:
  - lastTransitionTime: "2022-10-20T16:07:17Z"
    status: "True"
    type: CassandraInitialized
  datacenters:
    dc1:
      cassandra:
        cassandraOperatorProgress: Ready
        conditions:
        - lastTransitionTime: "2022-10-20T16:07:10Z"
          message: ""
          reason: ""
          status: "True"
          type: Healthy
        - lastTransitionTime: "2022-10-20T16:07:10Z"
          message: ""
          reason: ""
          status: "False"
          type: Stopped
        - lastTransitionTime: "2022-10-20T16:07:10Z"
          message: ""
          reason: ""
          status: "False"
          type: ReplacingNodes
        - lastTransitionTime: "2022-10-20T17:33:34Z"
          message: ""
          reason: ""
          status: "False"
          type: Updating
        - lastTransitionTime: "2022-10-20T16:07:10Z"
          message: ""
          reason: ""
          status: "False"
          type: RollingRestart
        - lastTransitionTime: "2022-10-20T16:07:10Z"
          message: ""
          reason: ""
          status: "False"
          type: Resuming
        - lastTransitionTime: "2022-10-20T16:07:10Z"
          message: ""
          reason: ""
          status: "False"
          type: ScalingDown
        - lastTransitionTime: "2022-10-20T16:07:10Z"
          message: ""
          reason: ""
          status: "True"
          type: Valid
        - lastTransitionTime: "2022-10-20T16:07:11Z"
          message: ""
          reason: ""
          status: "True"
          type: Initialized
        - lastTransitionTime: "2022-10-20T16:07:11Z"
          message: ""
          reason: ""
          status: "True"
          type: Ready
        lastServerNodeStarted: "2022-10-20T17:32:21Z"
        nodeStatuses:
          demo-dc1-rack1-sts-0:
            hostID: 772b67f5-ee00-4eab-ab84-61f430d376ea
          demo-dc1-rack2-sts-0:
            hostID: 9ecd5c6b-f062-454d-8411-7cb3a2e9283a
          demo-dc1-rack3-sts-0:
            hostID: cf3a4951-f554-43b2-9c42-290c0301d47d
        observedGeneration: 2
        quietPeriod: "2022-10-20T20:26:47Z"
        superUserUpserted: "2022-10-20T20:26:42Z"
        usersUpserted: "2022-10-20T20:26:42Z"
    dc2:
      cassandra:
        cassandraOperatorProgress: Updating
        lastServerNodeStarted: "2022-10-20T20:30:18Z"
        nodeStatuses:
          demo-dc2-rack1-sts-0:
            hostID: 53489fcd-7ac5-4e60-b231-e152efed736d
          demo-dc2-rack2-sts-0:
            hostID: 2ba9874a-3d7c-4033-8ab7-9653c48274df
  error: None
...

The sample output indicates that two DSE nodes are online at this point in the monitoring. The new CassandraDatacenter dc2 is ready when all of the Ready and Initialized conditions:status are set to "True".

Operators running on the cluster automatically modify system keyspaces to include the new datacenter. Replication of user-defined keyspaces remains unchanged.

The following keyspaces are updated:

system_traces
system_distributed
system_auth
dse_leases
dse_perf
dse_security

Now users can run workloads across the east and west region datacenters.

Next steps

The user configures the replication factor of any user keyspaces. In this example RF=3. If the number of nodes in the datacenter is less than 3, the RF is set equal to the number of nodes. See Cleanup DSE nodes in a datacenter. See Changing keyspace replication strategy using CQLSH commands.
Run a rebuild DSE operation on all nodes in the newly created datacenter, using the original datacenter as the streaming source. See Rebuild DSE.