Removing Nodes from a DataStax Enterprise Cluster or Datacenter

DataStax Mission Control is current in Private Preview. It is subject to the beta agreement executed between you and DataStax. DataStax Mission Control is not intended for production use, has not been certified for production workloads, and might contain bugs and other functional issues. There is no guarantee that DataStax Mission Control will ever become generally available. DataStax Mission Control is provided on an “AS IS” basis, without warranty or indemnity of any kind.

If you are interested in trying out DataStax Mission Control please contact your DataStax account team.

When cloud costs are excessive, DataStax Mission Control allows the removal of a number of nodes in existing datacenters to better control those costs. Servicing customer queries continues uninterrupted in any datacenter because DataStax Mission Control controls and balances the decommission of nodes.

This task information focuses on removing nodes from a single datacenter.

To scale-down the number of datacenters, follow the much more efficient task that removes a DSE datacenter instead.

Example

There is one (1) datacenter (DC) with 6 nodes. In this datacenter the number of nodes is to be reduced to 3 nodes.

Workflow of user and operators

User submits the modified DSECluster to Control Plane Kubernetes cluster with reduced size parameter.
Cluster-level operator detects dc-level change in cluster object, modifies dc-level resources.

DC-level operator detects size change in dc-level resource, decommissions nodes one by one.

When decommissioning nodes, DataStax Mission Control considers:

any target datacenter.
a rejection of any desired cluster size when it is incompatible with the number of defined racks.
targeting the rack with the highest number of active nodes.
choosing the first rack name according to an ascending sort order in the case of a tie between racks and their number of nodes.

decommissioning multiple nodes in a single rack occurs only after adjusting the remaining racks in the datacenter to reflect the desired node count.

DataStax Mission Control enlists cass-operator to check that the remaining nodes have enough capacity to handle the increased storage requirements. If cass-operator determines that there is insufficient capacity, then it logs a message, and reports units in bytes. Otherwise, cass-operator automatically runs nodetool decommission on the node to be removed. As a final step, the pod is terminated.

Limitations - You must decrease the datacenter size by a multiple of the number of racks in the target datacenter. For example, with 3 racks you may scale down by 3, 6, or 9 nodes, and so on. Invalid size parameters are ignored.

Procedure

Identify number of nodes to be removed; three (3) in this example.
Modify DC size parameter in the DSECluster Custom Resource Definition (CRD).
Submit the change to Kubernetes.

Monitor the progress of decommission.

Use this command to check the status of pod termination:

kubectl get events

Events such as ScalingDown may result.

Run the following command to check the status of the CassandraDatacenter object. In the output look for conditions set indicating its status set to either false or true, depending on the capacity determined by cass-operator.

 kubectl get cassandradatacenter dc1 -o yaml

Sample output:

status:
   conditions:
  - lastTransitionTime: "2021-03-30T22:01:48Z"
    message: ""
    reason: ""
    status: "True"
    type: ScalingDown

Alternate output:

status:
   conditions:
  - lastTransitionTime: "2021-03-30T22:01:48Z"
message: "Not enough free space available to decommission. my-k8ssandra-dc1-default-sts-3 has 12345 free space, but 67891 is needed."
    reason: "NotEnoughSpaceToScaleDown"
    status: "False"
    type: Valid