• Glossary
  • Support
  • Downloads
  • DataStax Home
Get Live Help
Expand All
Collapse All

DataStax Project Mission Control

    • Overview
      • Release notes
      • FAQs
      • Getting support
    • Installing DataStax Mission Control
      • Planning your install
      • Server-based Runtime Installer
        • Services setup with DataStax Mission Control Runtime Installer
      • Bring your own Kubernetes
        • Installing Control Plane
        • Installing Data Plane
    • Migrating
      • Migrating DSE Cluster to DataStax Mission Control
    • Managing
      • Managing DSE clusters
        • Configuring DSE
          • Authentication
          • Authorization
          • Securing DSE
          • DSE Unified Authorization
        • Cluster lifecycle
          • Creating a cluster
          • Creating a single-token cluster
          • Creating a multi-token cluster
          • Terminating a DSE cluster
          • Upgrading a DSE cluster
        • Datacenter lifecycle
          • Adding a DSE datacenter
          • Terminating a DSE datacenter
        • Node lifecycle
          • Adding DSE nodes
          • Terminating DSE nodes
          • Using per-node configurations
      • Managing DataStax Mission Control infrastructure
        • Adding a node to DataStax Mission Control clusters
        • Terminating a node from DataStax Mission Control clusters
        • Storage classes defined
      • Managing DataStax Mission Control resources
        • Accessing Admin Console
        • Configuring DataStax Mission Control
        • Generating a support bundle
    • Operating on DSE Clusters
      • Cleanup
      • Rebuilding
      • Replacing a node
      • Rolling restart
      • Upgrading SSTables
    • Reference
      • DSECluster manifest
      • CassandraTask manifest
  • DataStax Project Mission Control
  • Managing
  • Managing DSE clusters
  • Node lifecycle
  • Terminating DSE nodes

Removing Nodes from a DataStax Enterprise Cluster or Datacenter

DataStax Mission Control is current in Private Preview. It is subject to the beta agreement executed between you and DataStax. DataStax Mission Control is not intended for production use, has not been certified for production workloads, and might contain bugs and other functional issues. There is no guarantee that DataStax Mission Control will ever become generally available. DataStax Mission Control is provided on an “AS IS” basis, without warranty or indemnity of any kind.

If you are interested in trying out DataStax Mission Control please contact your DataStax account team.

When cloud costs are excessive, DataStax Mission Control allows the removal of a number of nodes in existing datacenters to better control those costs. Servicing customer queries continues uninterrupted in any datacenter because DataStax Mission Control controls and balances the decommission of nodes.

This task information focuses on removing nodes from a single datacenter.

To scale-down the number of datacenters, follow the much more efficient task that removes a DSE datacenter instead.

Example

There is one (1) datacenter (DC) with 6 nodes. In this datacenter the number of nodes is to be reduced to 3 nodes.

Workflow of user and operators

  1. User submits the modified DSECluster to Control Plane Kubernetes cluster with reduced size parameter.

  2. Cluster-level operator detects dc-level change in cluster object, modifies dc-level resources.

  3. DC-level operator detects size change in dc-level resource, decommissions nodes one by one.

    When decommissioning nodes, DataStax Mission Control considers:

    • any target datacenter.

    • a rejection of any desired cluster size when it is incompatible with the number of defined racks.

    • targeting the rack with the highest number of active nodes.

    • choosing the first rack name according to an ascending sort order in the case of a tie between racks and their number of nodes.

    • decommissioning multiple nodes in a single rack occurs only after adjusting the remaining racks in the datacenter to reflect the desired node count.

      DataStax Mission Control enlists cass-operator to check that the remaining nodes have enough capacity to handle the increased storage requirements. If cass-operator determines that there is insufficient capacity, then it logs a message, and reports units in bytes. Otherwise, cass-operator automatically runs nodetool decommission on the node to be removed. As a final step, the pod is terminated.

      Limitations - You must decrease the datacenter size by a multiple of the number of racks in the target datacenter. For example, with 3 racks you may scale down by 3, 6, or 9 nodes, and so on. Invalid size parameters are ignored.

Procedure

  1. Identify number of nodes to be removed; three (3) in this example.

  2. Modify DC size parameter in the DSECluster Custom Resource Definition (CRD).

  3. Submit the change to Kubernetes.

  4. Monitor the progress of decommission.

    Use this command to check the status of pod termination:

    kubectl get events

    Events such as ScalingDown may result.

    Run the following command to check the status of the CassandraDatacenter object. In the output look for conditions set indicating its status set to either false or true, depending on the capacity determined by cass-operator.

     kubectl get cassandradatacenter dc1 -o yaml

    Sample output:

    status:
       conditions:
      - lastTransitionTime: "2021-03-30T22:01:48Z"
        message: ""
        reason: ""
        status: "True"
        type: ScalingDown

    Alternate output:

    status:
       conditions:
      - lastTransitionTime: "2021-03-30T22:01:48Z"
    message: "Not enough free space available to decommission. my-k8ssandra-dc1-default-sts-3 has 12345 free space, but 67891 is needed."
        reason: "NotEnoughSpaceToScaleDown"
        status: "False"
        type: Valid
Adding DSE nodes Using per-node configurations

General Inquiries: +1 (650) 389-6000 info@datastax.com

© DataStax | Privacy policy | Terms of use

DataStax, Titan, and TitanDB are registered trademarks of DataStax, Inc. and its subsidiaries in the United States and/or other countries.

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.

Kubernetes is the registered trademark of the Linux Foundation.

landing_page landingpage