• Glossary
  • Support
  • Downloads
  • DataStax Home
Get Live Help
Expand All
Collapse All

DataStax Project Mission Control

    • Overview
      • Release notes
      • FAQs
      • Getting support
    • Installing DataStax Mission Control
      • Planning your install
      • Server-based Runtime Installer
        • Services setup with DataStax Mission Control Runtime Installer
      • Bring your own Kubernetes
        • Installing Control Plane
        • Installing Data Plane
    • Migrating
      • Migrating DSE Cluster to DataStax Mission Control
    • Managing
      • Managing DSE clusters
        • Configuring DSE
          • Authentication
          • Authorization
          • Securing DSE
          • DSE Unified Authorization
        • Cluster lifecycle
          • Creating a cluster
          • Creating a single-token cluster
          • Creating a multi-token cluster
          • Terminating a DSE cluster
          • Upgrading a DSE cluster
        • Datacenter lifecycle
          • Adding a DSE datacenter
          • Terminating a DSE datacenter
        • Node lifecycle
          • Adding DSE nodes
          • Terminating DSE nodes
          • Using per-node configurations
      • Managing DataStax Mission Control infrastructure
        • Adding a node to DataStax Mission Control clusters
        • Terminating a node from DataStax Mission Control clusters
        • Storage classes defined
      • Managing DataStax Mission Control resources
        • Accessing Admin Console
        • Configuring DataStax Mission Control
        • Generating a support bundle
    • Operating on DSE Clusters
      • Cleanup
      • Rebuilding
      • Replacing a node
      • Rolling restart
      • Upgrading SSTables
    • Reference
      • DSECluster manifest
      • CassandraTask manifest
  • DataStax Project Mission Control
  • Managing
  • Managing DSE clusters
  • Datacenter lifecycle
  • Adding a DSE datacenter

Adding a Datacenter to an Existing DataStax Enterprise Cluster

DataStax Mission Control is current in Private Preview. It is subject to the beta agreement executed between you and DataStax. DataStax Mission Control is not intended for production use, has not been certified for production workloads, and might contain bugs and other functional issues. There is no guarantee that DataStax Mission Control will ever become generally available. DataStax Mission Control is provided on an “AS IS” basis, without warranty or indemnity of any kind.

If you are interested in trying out DataStax Mission Control please contact your DataStax account team.

Database administrators can add or terminate one or more datacenters to: * support additional workloads * solve latency issues * expand into new markets * add capacity so that their applications remain available * support new functionality.

Use DataStax Mission Control to create one or more datacenters. Do this from an existing cluster and bootstrap one datacenter at a time. With multiple datacenters, sort them in ascending order by datacenter name and select each datacenter (DC) in the list to create until all are added to the cluster.

Prerequisites

  • The kubectl CLI tool.

  • Kubeconfig file or context pointing to a Control Plane Kubernetes cluster.

  • A Kubernetes cluster must exist.

Example

This example uses a Kubernetes cluster with a single east region datacenter set up in the Data Plane. The example extends it to include a second, west region datacenter set up in the Data Plane.

Workflow for User and operators

  1. User submits a modified DSECluster specifying a new datacenter in the west region and defining it to the Control Plane Kubernetes cluster.

  2. Cluster-level operator picks up the modification and creates datacenter-level resources in the Kubernetes cluster in the east region where the nodes are to be created.

  3. DC-level operator picks up datacenter-level resources and creates native Kubernetes objects representing the DSE nodes.

  4. DC-level operator bootstraps one node at a time, balancing operations across racks and reporting progress.

  5. Cluster-level operator updates keyspace replication settings on system keyspaces. User updates keyspace replication settings on user keyspaces.

  6. Cluster-level operator runs a rebuild operation on system keyspaces.

Procedure

  1. Define the new datacenter (DC) criteria, for example:

    • Target the region or Kubernetes (k8s) cluster, availability zones, etc.

    • Target the workload (core Cassandra, Search, graph)

  2. Modify the existing DSECluster YAML file in the Control Plane cluster to add a new DC definition with three (3) nodes in the west region:

    apiVersion: missioncontrol.datastax.com/v1alpha1
    kind: DSECluster
    metadata:
      name: demo
    spec:
    ...
      datacenters:
          - metadata:
              name: dc1
            k8sContext: east
            size: 3
    ...
    - metadata:
            name: dc2
          k8sContext: west
          size: 3
          racks:
            - name: rack1
              nodeAffinityLabels:
                topology.kubernetes.io/zone: us-west1-c
            - name: rack2
              nodeAffinityLabels:
                topology.kubernetes.io/zone: us-west1-b
            - name: rack3
              nodeAffinityLabels:
                topology.kubernetes.io/zone: us-west1-a
  3. Submit the updated DSECluster YAML file to Kubernetes.

    kubectl apply -f demo-dse.cassandratask.yaml

    The K8ssandraCluster is updated. DC-level operators then create a CassandraDatacenter named dc2 in the west cluster.

  4. Monitor the progress of adding a datacenter with the following command:

     kubectl get k8ssandracluster demo -o yaml

    Sample output:

    ...
    status:
      conditions:
      - lastTransitionTime: "2022-10-20T16:07:17Z"
        status: "True"
        type: CassandraInitialized
      datacenters:
        dc1:
          cassandra:
            cassandraOperatorProgress: Ready
            conditions:
            - lastTransitionTime: "2022-10-20T16:07:10Z"
              message: ""
              reason: ""
              status: "True"
              type: Healthy
            - lastTransitionTime: "2022-10-20T16:07:10Z"
              message: ""
              reason: ""
              status: "False"
              type: Stopped
            - lastTransitionTime: "2022-10-20T16:07:10Z"
              message: ""
              reason: ""
              status: "False"
              type: ReplacingNodes
            - lastTransitionTime: "2022-10-20T17:33:34Z"
              message: ""
              reason: ""
              status: "False"
              type: Updating
            - lastTransitionTime: "2022-10-20T16:07:10Z"
              message: ""
              reason: ""
              status: "False"
              type: RollingRestart
            - lastTransitionTime: "2022-10-20T16:07:10Z"
              message: ""
              reason: ""
              status: "False"
              type: Resuming
            - lastTransitionTime: "2022-10-20T16:07:10Z"
              message: ""
              reason: ""
              status: "False"
              type: ScalingDown
            - lastTransitionTime: "2022-10-20T16:07:10Z"
              message: ""
              reason: ""
              status: "True"
              type: Valid
            - lastTransitionTime: "2022-10-20T16:07:11Z"
              message: ""
              reason: ""
              status: "True"
              type: Initialized
            - lastTransitionTime: "2022-10-20T16:07:11Z"
              message: ""
              reason: ""
              status: "True"
              type: Ready
            lastServerNodeStarted: "2022-10-20T17:32:21Z"
            nodeStatuses:
              demo-dc1-rack1-sts-0:
                hostID: 772b67f5-ee00-4eab-ab84-61f430d376ea
              demo-dc1-rack2-sts-0:
                hostID: 9ecd5c6b-f062-454d-8411-7cb3a2e9283a
              demo-dc1-rack3-sts-0:
                hostID: cf3a4951-f554-43b2-9c42-290c0301d47d
            observedGeneration: 2
            quietPeriod: "2022-10-20T20:26:47Z"
            superUserUpserted: "2022-10-20T20:26:42Z"
            usersUpserted: "2022-10-20T20:26:42Z"
        dc2:
          cassandra:
            cassandraOperatorProgress: Updating
            lastServerNodeStarted: "2022-10-20T20:30:18Z"
            nodeStatuses:
              demo-dc2-rack1-sts-0:
                hostID: 53489fcd-7ac5-4e60-b231-e152efed736d
              demo-dc2-rack2-sts-0:
                hostID: 2ba9874a-3d7c-4033-8ab7-9653c48274df
      error: None
    ...

    The sample output indicates that two DSE nodes are online at this point in the monitoring. The new CassandraDatacenter dc2 is ready when all of the Ready and Initialized conditions:status are set to "True".

Operators running on the cluster automatically modify system keyspaces to include the new datacenter. Replication of user-defined keyspaces remains unchanged.

The following keyspaces are updated:

  • system_traces

  • system_distributed

  • system_auth

  • dse_leases

  • dse_perf

  • dse_security

Now users can run workloads across the east and west region datacenters.

What’s next

  1. The user configures the replication factor of any user keyspaces. In this example RF=3. If the number of nodes in the datacenter is less than 3, the RF is set equal to the number of nodes. See Cleanup DSE nodes in a datacenter. See Changing keyspace replication strategy using CQLSH commands.

  2. Run a rebuild DSE operation on all nodes in the newly created datacenter, using the original datacenter as the streaming source. See Rebuild DSE.

Datacenter lifecycle Terminating a DSE datacenter

General Inquiries: +1 (650) 389-6000 info@datastax.com

© DataStax | Privacy policy | Terms of use

DataStax, Titan, and TitanDB are registered trademarks of DataStax, Inc. and its subsidiaries in the United States and/or other countries.

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.

Kubernetes is the registered trademark of the Linux Foundation.

landing_page landingpage