• Glossary
  • Support
  • Downloads
  • DataStax Home
Get Live Help
Expand All
Collapse All

DataStax Project Mission Control

    • Overview
      • Release notes
      • FAQs
      • Getting support
    • Installing DataStax Mission Control
      • Planning your install
      • Server-based Runtime Installer
        • Services setup with DataStax Mission Control Runtime Installer
      • Bring your own Kubernetes
        • Installing Control Plane
        • Installing Data Plane
    • Migrating
      • Migrating DSE Cluster to DataStax Mission Control
    • Managing
      • Managing DSE clusters
        • Configuring DSE
          • Authentication
          • Authorization
          • Securing DSE
          • DSE Unified Authorization
        • Cluster lifecycle
          • Creating a cluster
          • Creating a single-token cluster
          • Creating a multi-token cluster
          • Terminating a DSE cluster
          • Upgrading a DSE cluster
        • Datacenter lifecycle
          • Adding a DSE datacenter
          • Terminating a DSE datacenter
        • Node lifecycle
          • Adding DSE nodes
          • Terminating DSE nodes
          • Using per-node configurations
      • Managing DataStax Mission Control infrastructure
        • Adding a node to DataStax Mission Control clusters
        • Terminating a node from DataStax Mission Control clusters
        • Storage classes defined
      • Managing DataStax Mission Control resources
        • Accessing Admin Console
        • Configuring DataStax Mission Control
        • Generating a support bundle
    • Operating on DSE Clusters
      • Cleanup
      • Rebuilding
      • Replacing a node
      • Rolling restart
      • Upgrading SSTables
    • Reference
      • DSECluster manifest
      • CassandraTask manifest
  • DataStax Project Mission Control
  • Managing
  • Managing DSE clusters
  • Node lifecycle
  • Adding DSE nodes

Add Nodes to a DataStax Enterprise Cluster or Datacenter

DataStax Mission Control is current in Private Preview. It is subject to the beta agreement executed between you and DataStax. DataStax Mission Control is not intended for production use, has not been certified for production workloads, and might contain bugs and other functional issues. There is no guarantee that DataStax Mission Control will ever become generally available. DataStax Mission Control is provided on an “AS IS” basis, without warranty or indemnity of any kind.

If you are interested in trying out DataStax Mission Control please contact your DataStax account team.

Adding nodes increases the capacity to service customer queries against the data.

This task information focuses on adding nodes to a single existing datacenter.

To scale-up the number of datacenters, follow the much more efficient task that adds a DSE datacenter instead.

Example

An existing k8ssandra datacenter with a cluster of 2 pods is installed based on the following k8ssandra-values.yaml properties:

cassandra:
  clusterName: org-k8ssandra
  datacenters:
  - name: dc1
    size: 2

Workflow of user and operators

  1. DataStax Mission Control User submits modified datacenter size parameter in DSECluster to the Control Plane Kubernetes cluster.

  2. Cluster-level operator detects dc-level change in the cluster object and modifies dc-level resources.

  3. DC-level operator detects size change in dc-level resource and provisions Kubernetes resources representing the new nodes.

  4. DC-level operator bootstraps DSE nodes on new pods.

    When commissioning nodes, DataStax Mission Control:

    • targets the rack with the lowest number of active nodes.

    • uses a bootstrap (self-starting process) that adds nodes without external input.

    • commissions multiple nodes in a single rack happens only after adjusting other racks in the datacenter to reflect the desired node count.

    • identifies the number of nodes being added.

      Limitations - You must increase the datacenter size by a multiple of the number of racks in the target datacenter. For example, with 3 racks you may scale up by 3, 6, or 9 nodes, and so on. Invalid size parameters are ignored.

Procedure

  1. Modify the DC size parameter in DSECluster object.

    Example: size: 3.

    By default, DataStax Mission Control configures the Cassandra pods so that Kubernetes is blocked from scheduling multiple DSE pods on the same worker node. An attempt to increase the cluster size beyond the number of available worker nodes may result in the non-deployment of additional pods.

    Here is a sample:

    apiVersion: control.k8ssandra.io/v1alpha1
    kind: CassandraTask
    metadata:
      name: cleanup-dc1
    spec:
      datacenter:
        name: dc1
        namespace: demo
      jobs:
        - name: cleanup-dc1
          command: cleanup
          args:
            keyspace_name: my_keyspace
  2. Submit that change to Kubernetes with the following command:

    helm upgrade org-k8ssandra k8ssandra/k8ssandra -f k8ssandra-values.yaml
  3. Monitor the bootstrap progress with this command to check the Running or Pending status of pods:

    kubectl get pods

    If the results show a pod with Pending status, issue this command to get more details about the pod:

    kubectl describe pod <pod-name>

    The results may indicate a FailedScheduling event. To override the default of only one Cassandra pod per Kubernetes worker node, set the option allowMultipleNodesPerWorker: true in the Helm chart file. Apply this configuration update to the cluster.

    Run the following command to check the status of the CassandraDatacenter object. In the output look for a ScalingUp condition with its status set to true.

     kubectl get cassandradatacenter dc1 -o yaml

    Sample output:

    ...
    status:
      cassandraOperatorProgress: Updating
      conditions:
      - lastTransitionTime: "2021-03-30T22:01:48Z"
        message: ""
        reason: ""
        status: "True"
        type: ScalingUp
    ...

After the new nodes are deployed and running, the K8ssandra cass-operator automatically runs nodetool cleanup only on the original nodes and not the new nodes. This removes keys and data that are no longer associated with those original nodes.

What’s next

Run Cleanup operation to recover disk space from previously provisioned nodes.

Node lifecycle Terminating DSE nodes

General Inquiries: +1 (650) 389-6000 info@datastax.com

© DataStax | Privacy policy | Terms of use

DataStax, Titan, and TitanDB are registered trademarks of DataStax, Inc. and its subsidiaries in the United States and/or other countries.

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.

Kubernetes is the registered trademark of the Linux Foundation.

landing_page landingpage