Add a Datacenter to an Existing DataStax Enterprise Cluster
DataStax Mission Control is currently in Public Preview. DataStax Mission Control is not intended for production use, has not been certified for production workloads, and might contain bugs and other functional issues. There is no guarantee that DataStax Mission Control will ever become generally available. DataStax Mission Control is provided on an “AS IS” basis, without warranty or indemnity of any kind. If you are interested in trying out DataStax Mission Control please join the Public Preview. |
Database administrators can add or terminate one or more datacenters to: * support additional workloads * solve latency issues * expand into new markets * add capacity so that their applications remain available * support new functionality.
Use DataStax Mission Control to create one or more datacenters. Do this from an existing cluster and bootstrap one datacenter at a time. With multiple datacenters, sort them in ascending order by datacenter name and select each datacenter (DC) in the list to create until all are added to the cluster.
Prerequisites
-
The
kubectl
CLI tool. -
Kubeconfig file
orcontext
pointing to aControl Plane
Kubernetes cluster. -
A Kubernetes cluster must exist.
Example
This example uses a Kubernetes cluster with a single east region datacenter set up in the Data Plane
. The example extends it to include a second, west region datacenter set up in the Data Plane
.
Workflow for User and operators
-
User submits a modified
MissionControlCluster
specifying a new datacenter in the west region and defining it to theControl Plane
Kubernetes cluster. -
Cluster-level operator picks up the modification and creates datacenter-level resources in the Kubernetes cluster in the east region where the nodes are to be created.
-
DC-level operator picks up datacenter-level resources and creates native Kubernetes objects representing the DSE nodes.
-
DC-level operator bootstraps one node at a time, balancing operations across racks and reporting progress.
-
Cluster-level operator updates keyspace replication settings on system keyspaces. User updates keyspace replication settings on user keyspaces.
-
Cluster-level operator runs a rebuild operation on system keyspaces.
Procedure
-
Define the new datacenter (DC) criteria, for example:
-
Target the region or Kubernetes (k8s) cluster, availability zones, etc.
-
Target the workload (core Cassandra, Search, graph)
-
-
Modify the existing
MissionControlCluster
YAML file in theControl Plane
cluster to add a new DC definition with three (3) nodes in the west region:apiVersion: missioncontrol.datastax.com/v1beta1 kind: MissionControlCluster metadata: name: demo spec: ... datacenters: - metadata: name: dc1 k8sContext: east size: 3 ... - metadata: name: dc2 k8sContext: west size: 3 racks: - name: rack1 nodeAffinityLabels: topology.kubernetes.io/zone: us-west1-c - name: rack2 nodeAffinityLabels: topology.kubernetes.io/zone: us-west1-b - name: rack3 nodeAffinityLabels: topology.kubernetes.io/zone: us-west1-a
-
Submit the updated
MissionControlCluster
YAML file to Kubernetes.kubectl apply -f demo-dse.cassandratask.yaml
The K8ssandraCluster is updated. DC-level operators then create a CassandraDatacenter named
dc2
in the west cluster. -
Monitor the progress of adding a datacenter with the following command:
kubectl get k8ssandracluster demo -o yaml
Sample output:
... status: conditions: - lastTransitionTime: "2022-10-20T16:07:17Z" status: "True" type: CassandraInitialized datacenters: dc1: cassandra: cassandraOperatorProgress: Ready conditions: - lastTransitionTime: "2022-10-20T16:07:10Z" message: "" reason: "" status: "True" type: Healthy - lastTransitionTime: "2022-10-20T16:07:10Z" message: "" reason: "" status: "False" type: Stopped - lastTransitionTime: "2022-10-20T16:07:10Z" message: "" reason: "" status: "False" type: ReplacingNodes - lastTransitionTime: "2022-10-20T17:33:34Z" message: "" reason: "" status: "False" type: Updating - lastTransitionTime: "2022-10-20T16:07:10Z" message: "" reason: "" status: "False" type: RollingRestart - lastTransitionTime: "2022-10-20T16:07:10Z" message: "" reason: "" status: "False" type: Resuming - lastTransitionTime: "2022-10-20T16:07:10Z" message: "" reason: "" status: "False" type: ScalingDown - lastTransitionTime: "2022-10-20T16:07:10Z" message: "" reason: "" status: "True" type: Valid - lastTransitionTime: "2022-10-20T16:07:11Z" message: "" reason: "" status: "True" type: Initialized - lastTransitionTime: "2022-10-20T16:07:11Z" message: "" reason: "" status: "True" type: Ready lastServerNodeStarted: "2022-10-20T17:32:21Z" nodeStatuses: demo-dc1-rack1-sts-0: hostID: 772b67f5-ee00-4eab-ab84-61f430d376ea demo-dc1-rack2-sts-0: hostID: 9ecd5c6b-f062-454d-8411-7cb3a2e9283a demo-dc1-rack3-sts-0: hostID: cf3a4951-f554-43b2-9c42-290c0301d47d observedGeneration: 2 quietPeriod: "2022-10-20T20:26:47Z" superUserUpserted: "2022-10-20T20:26:42Z" usersUpserted: "2022-10-20T20:26:42Z" dc2: cassandra: cassandraOperatorProgress: Updating lastServerNodeStarted: "2022-10-20T20:30:18Z" nodeStatuses: demo-dc2-rack1-sts-0: hostID: 53489fcd-7ac5-4e60-b231-e152efed736d demo-dc2-rack2-sts-0: hostID: 2ba9874a-3d7c-4033-8ab7-9653c48274df error: None ...
The sample output indicates that two DSE nodes are online at this point in the monitoring. The new CassandraDatacenter
dc2
is ready when all of theReady
andInitialized
conditions:status
are set to "True".
Operators running on the cluster automatically modify system keyspaces to include the new datacenter. Replication of user-defined keyspaces remains unchanged. |
The following keyspaces are updated:
-
system_traces
-
system_distributed
-
system_auth
-
dse_leases
-
dse_perf
-
dse_security
Now users can run workloads across the east and west region datacenters.
What’s next
-
The user configures the replication factor of any user keyspaces. In this example RF=3. If the number of nodes in the datacenter is less than 3, the RF is set equal to the number of nodes. See Cleanup DSE nodes in a datacenter. See Changing keyspace replication strategy using CQLSH commands.
-
Run a
rebuild
DSE operation on all nodes in the newly created datacenter, using the original datacenter as the streaming source. See Rebuild DSE.