Add a datacenter to an existing DataStax Enterprise (DSE) cluster

Database administrators can add or terminate one or more datacenters to:

  • support additional workloads

  • solve latency issues

  • expand into new markets

  • add capacity so that their applications remain available

  • support new functionality.

Use Mission Control to create one or more datacenters. Do this from an existing cluster and bootstrap one datacenter at a time. With multiple datacenters, sort them in ascending order by datacenter name and select each datacenter (DC) in the list to create until all are added to the cluster.

Example

This example uses a Kubernetes cluster with a single east region datacenter set up in the Data Plane. The example extends it to include a second, west region datacenter set up in the Data Plane.

Workflow for User and operators

  1. User submits a modified MissionControlCluster specifying a new datacenter in the west region and defining it to the Control Plane Kubernetes cluster.

  2. Cluster-level operator picks up the modification and creates datacenter-level resources in the Kubernetes cluster in the east region where the nodes are to be created.

  3. DC-level operator picks up datacenter-level resources and creates native Kubernetes objects representing the DSE nodes.

  4. DC-level operator bootstraps one node at a time, balancing operations across racks and reporting progress.

  5. Cluster-level operator updates keyspace replication settings on system keyspaces. User updates keyspace replication settings on user keyspaces.

  6. Cluster-level operator runs a rebuild operation on system keyspaces.

Define and add a new datacenter to a cluster

  1. Define the new datacenter (DC) criteria, for example:

    • Target the region or Kubernetes (k8s) cluster, availability zones, etc.

    • Target the workload (core Cassandra, Search, graph)

  2. Modify the existing MissionControlCluster YAML file in the Control Plane cluster to add a new DC definition with three (3) nodes in the west region:

    apiVersion: missioncontrol.datastax.com/v1beta1
    kind: MissionControlCluster
    metadata:
      name: demo
    spec:
    ...
        datacenters:
          - metadata:
              name: dc1
            k8sContext: east
            size: 3
    ...
          - metadata:
              name: dc2
            k8sContext: west
            size: 3
            racks:
              - name: rack1
                nodeAffinityLabels:
                  topology.kubernetes.io/zone: us-west1-c
              - name: rack2
                nodeAffinityLabels:
                  topology.kubernetes.io/zone: us-west1-b
              - name: rack3
                nodeAffinityLabels:
                  topology.kubernetes.io/zone: us-west1-a
  3. Submit the updated MissionControlCluster YAML file to Kubernetes.

    kubectl apply -f demo-dse.cassandratask.yaml

    The K8ssandraCluster is updated. DC-level operators then create a CassandraDatacenter named dc2 in the west cluster.

  4. Monitor the progress of adding a datacenter with the following command:

     kubectl get k8ssandracluster demo -o yaml
    Sample results
    ...
    status:
      conditions:
      - lastTransitionTime: "2022-10-20T16:07:17Z"
        status: "True"
        type: CassandraInitialized
      datacenters:
        dc1:
          cassandra:
            cassandraOperatorProgress: Ready
            conditions:
            - lastTransitionTime: "2022-10-20T16:07:10Z"
              message: ""
              reason: ""
              status: "True"
              type: Healthy
            - lastTransitionTime: "2022-10-20T16:07:10Z"
              message: ""
              reason: ""
              status: "False"
              type: Stopped
            - lastTransitionTime: "2022-10-20T16:07:10Z"
              message: ""
              reason: ""
              status: "False"
              type: ReplacingNodes
            - lastTransitionTime: "2022-10-20T17:33:34Z"
              message: ""
              reason: ""
              status: "False"
              type: Updating
            - lastTransitionTime: "2022-10-20T16:07:10Z"
              message: ""
              reason: ""
              status: "False"
              type: RollingRestart
            - lastTransitionTime: "2022-10-20T16:07:10Z"
              message: ""
              reason: ""
              status: "False"
              type: Resuming
            - lastTransitionTime: "2022-10-20T16:07:10Z"
              message: ""
              reason: ""
              status: "False"
              type: ScalingDown
            - lastTransitionTime: "2022-10-20T16:07:10Z"
              message: ""
              reason: ""
              status: "True"
              type: Valid
            - lastTransitionTime: "2022-10-20T16:07:11Z"
              message: ""
              reason: ""
              status: "True"
              type: Initialized
            - lastTransitionTime: "2022-10-20T16:07:11Z"
              message: ""
              reason: ""
              status: "True"
              type: Ready
            lastServerNodeStarted: "2022-10-20T17:32:21Z"
            nodeStatuses:
              demo-dc1-rack1-sts-0:
                hostID: 772b67f5-ee00-4eab-ab84-61f430d376ea
              demo-dc1-rack2-sts-0:
                hostID: 9ecd5c6b-f062-454d-8411-7cb3a2e9283a
              demo-dc1-rack3-sts-0:
                hostID: cf3a4951-f554-43b2-9c42-290c0301d47d
            observedGeneration: 2
            quietPeriod: "2022-10-20T20:26:47Z"
            superUserUpserted: "2022-10-20T20:26:42Z"
            usersUpserted: "2022-10-20T20:26:42Z"
        dc2:
          cassandra:
            cassandraOperatorProgress: Updating
            lastServerNodeStarted: "2022-10-20T20:30:18Z"
            nodeStatuses:
              demo-dc2-rack1-sts-0:
                hostID: 53489fcd-7ac5-4e60-b231-e152efed736d
              demo-dc2-rack2-sts-0:
                hostID: 2ba9874a-3d7c-4033-8ab7-9653c48274df
      error: None
    ...

    The sample output indicates that two DSE nodes are online at this point in the monitoring. The new CassandraDatacenter dc2 is ready when all of the Ready and Initialized conditions:status are set to "True".

Operators running on the cluster automatically modify system keyspaces to include the new datacenter. Replication of user-defined keyspaces remains unchanged.

The following keyspaces are updated:

  • system_traces

  • system_distributed

  • system_auth

  • dse_leases

  • dse_perf

  • dse_security

Now users can run workloads across the east and west region datacenters.

What’s next

  1. The user configures the replication factor of any user keyspaces. In this example RF=3. If the number of nodes in the datacenter is less than 3, the RF is set equal to the number of nodes. See Cleanup DSE nodes in a datacenter. See Changing keyspace replication strategy using CQLSH commands.

  2. Run a rebuild DSE operation on all nodes in the newly created datacenter, using the original datacenter as the streaming source. See Rebuild DSE.

Was this helpful?

Give Feedback

How can we improve the documentation?

© 2024 DataStax | Privacy policy | Terms of use

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: +1 (650) 389-6000, info@datastax.com