Add a datacenter to an existing DataStax Enterprise (DSE) cluster
Database administrators can add or terminate one or more datacenters to:
-
support additional workloads
-
solve latency issues
-
expand into new markets
-
add capacity so that their applications remain available
-
support new functionality.
Use Mission Control to create one or more datacenters. Do this from an existing cluster and bootstrap one datacenter at a time. With multiple datacenters, sort them in ascending order by datacenter name and select each datacenter (DC) in the list to create until all are added to the cluster.
Prerequisites
-
A prepared environment on either bare-metal/VM or an existing Kubernetes cluster.
Example
This example uses a Kubernetes cluster with a single east region datacenter set up in the Data Plane
.
The example extends it to include a second, west region datacenter set up in the Data Plane
.
Workflow for User and operators
-
User submits a modified
MissionControlCluster
specifying a new datacenter in the west region and defining it to theControl Plane
Kubernetes cluster. -
Cluster-level operator picks up the modification and creates datacenter-level resources in the Kubernetes cluster in the east region where the nodes are to be created.
-
DC-level operator picks up datacenter-level resources and creates native Kubernetes objects representing the DSE nodes.
-
DC-level operator bootstraps one node at a time, balancing operations across racks and reporting progress.
-
Cluster-level operator updates keyspace replication settings on system keyspaces. User updates keyspace replication settings on user keyspaces.
-
Cluster-level operator runs a rebuild operation on system keyspaces.
Define and add a new datacenter to a cluster
-
Define the new datacenter (DC) criteria, for example:
-
Target the region or Kubernetes (k8s) cluster, availability zones, etc.
-
Target the workload (core Cassandra, Search, graph)
-
-
Modify the existing
MissionControlCluster
YAML file in theControl Plane
cluster to add a new DC definition with three (3) nodes in the west region:apiVersion: missioncontrol.datastax.com/v1beta1 kind: MissionControlCluster metadata: name: demo spec: ... datacenters: - metadata: name: dc1 k8sContext: east size: 3 ... - metadata: name: dc2 k8sContext: west size: 3 racks: - name: rack1 nodeAffinityLabels: topology.kubernetes.io/zone: us-west1-c - name: rack2 nodeAffinityLabels: topology.kubernetes.io/zone: us-west1-b - name: rack3 nodeAffinityLabels: topology.kubernetes.io/zone: us-west1-a
-
Submit the updated
MissionControlCluster
YAML file to Kubernetes.kubectl apply -f demo-dse.cassandratask.yaml
The K8ssandraCluster is updated. DC-level operators then create a CassandraDatacenter named
dc2
in the west cluster. -
Monitor the progress of adding a datacenter with the following command:
kubectl get k8ssandracluster demo -o yaml
Sample results
... status: conditions: - lastTransitionTime: "2022-10-20T16:07:17Z" status: "True" type: CassandraInitialized datacenters: dc1: cassandra: cassandraOperatorProgress: Ready conditions: - lastTransitionTime: "2022-10-20T16:07:10Z" message: "" reason: "" status: "True" type: Healthy - lastTransitionTime: "2022-10-20T16:07:10Z" message: "" reason: "" status: "False" type: Stopped - lastTransitionTime: "2022-10-20T16:07:10Z" message: "" reason: "" status: "False" type: ReplacingNodes - lastTransitionTime: "2022-10-20T17:33:34Z" message: "" reason: "" status: "False" type: Updating - lastTransitionTime: "2022-10-20T16:07:10Z" message: "" reason: "" status: "False" type: RollingRestart - lastTransitionTime: "2022-10-20T16:07:10Z" message: "" reason: "" status: "False" type: Resuming - lastTransitionTime: "2022-10-20T16:07:10Z" message: "" reason: "" status: "False" type: ScalingDown - lastTransitionTime: "2022-10-20T16:07:10Z" message: "" reason: "" status: "True" type: Valid - lastTransitionTime: "2022-10-20T16:07:11Z" message: "" reason: "" status: "True" type: Initialized - lastTransitionTime: "2022-10-20T16:07:11Z" message: "" reason: "" status: "True" type: Ready lastServerNodeStarted: "2022-10-20T17:32:21Z" nodeStatuses: demo-dc1-rack1-sts-0: hostID: 772b67f5-ee00-4eab-ab84-61f430d376ea demo-dc1-rack2-sts-0: hostID: 9ecd5c6b-f062-454d-8411-7cb3a2e9283a demo-dc1-rack3-sts-0: hostID: cf3a4951-f554-43b2-9c42-290c0301d47d observedGeneration: 2 quietPeriod: "2022-10-20T20:26:47Z" superUserUpserted: "2022-10-20T20:26:42Z" usersUpserted: "2022-10-20T20:26:42Z" dc2: cassandra: cassandraOperatorProgress: Updating lastServerNodeStarted: "2022-10-20T20:30:18Z" nodeStatuses: demo-dc2-rack1-sts-0: hostID: 53489fcd-7ac5-4e60-b231-e152efed736d demo-dc2-rack2-sts-0: hostID: 2ba9874a-3d7c-4033-8ab7-9653c48274df error: None ...
The sample output indicates that two DSE nodes are online at this point in the monitoring. The new CassandraDatacenter
dc2
is ready when all of theReady
andInitialized
conditions:status
are set to "True".
Operators running on the cluster automatically modify system keyspaces to include the new datacenter. Replication of user-defined keyspaces remains unchanged. |
The following keyspaces are updated:
-
system_traces
-
system_distributed
-
system_auth
-
dse_leases
-
dse_perf
-
dse_security
Now users can run workloads across the east and west region datacenters.
What’s next
-
The user configures the replication factor of any user keyspaces. In this example RF=3. If the number of nodes in the datacenter is less than 3, the RF is set equal to the number of nodes. See Cleanup DSE nodes in a datacenter. See Changing keyspace replication strategy using CQLSH commands.
-
Run a
rebuild
DSE operation on all nodes in the newly created datacenter, using the original datacenter as the streaming source. See Rebuild DSE.