Add Nodes to a DataStax Enterprise Cluster or Datacenter

DataStax Mission Control is currently in Public Preview. DataStax Mission Control is not intended for production use, has not been certified for production workloads, and might contain bugs and other functional issues. There is no guarantee that DataStax Mission Control will ever become generally available. DataStax Mission Control is provided on an “AS IS” basis, without warranty or indemnity of any kind.

If you are interested in trying out DataStax Mission Control please join the Public Preview.

Adding nodes increases the capacity to service customer queries against the data.

This task information focuses on adding nodes to a single existing datacenter.

To scale-up the number of datacenters, follow the much more efficient task that adds a DSE datacenter instead.

Example

An existing k8ssandra datacenter with a cluster of 2 pods is installed based on the following k8ssandra-values.yaml properties:

cassandra:
  clusterName: org-k8ssandra
  datacenters:
  - name: dc1
    size: 2

Workflow of user and operators

DataStax Mission Control User submits modified datacenter size parameter in MissionControlCluster to the Control Plane Kubernetes cluster.
Cluster-level operator detects dc-level change in the cluster object and modifies dc-level resources.
DC-level operator detects size change in dc-level resource and provisions Kubernetes resources representing the new nodes.

DC-level operator bootstraps DSE nodes on new pods.

When commissioning nodes, DataStax Mission Control:

targets the rack with the lowest number of active nodes.
uses a bootstrap (self-starting process) that adds nodes without external input.
commissions multiple nodes in a single rack happens only after adjusting other racks in the datacenter to reflect the desired node count.

identifies the number of nodes being added.

Limitations - You must increase the datacenter size by a multiple of the number of racks in the target datacenter. For example, with 3 racks you may scale up by 3, 6, or 9 nodes, and so on. Invalid size parameters are ignored.

Procedure

Modify the DC size parameter in MissionControlCluster object.

Example: size: 3.

By default, DataStax Mission Control configures the Cassandra pods so that Kubernetes is blocked from scheduling multiple DSE pods on the same worker node. An attempt to increase the cluster size beyond the number of available worker nodes may result in the non-deployment of additional pods.

Here is a sample:

apiVersion: control.k8ssandra.io/v1alpha1
kind: CassandraTask
metadata:
  name: cleanup-dc1
spec:
  datacenter:
    name: dc1
    namespace: demo
  jobs:
    - name: cleanup-dc1
      command: cleanup
      args:
        keyspace_name: my_keyspace

Submit that change to Kubernetes with the following command:

helm upgrade org-k8ssandra k8ssandra/k8ssandra -f k8ssandra-values.yaml

Monitor the bootstrap progress with this command to check the Running or Pending status of pods:
```
kubectl get pods
```
If the results show a pod with Pending status, issue this command to get more details about the pod:
```
kubectl describe pod <pod-name>
```
The results may indicate a FailedScheduling event. To override the default of only one Cassandra pod per Kubernetes worker node, set the option allowMultipleNodesPerWorker: true in the Helm chart file. Apply this configuration update to the cluster.

Run the following command to check the status of the CassandraDatacenter object. In the output look for a ScalingUp condition with its status set to true.
```
 kubectl get cassandradatacenter dc1 -o yaml
```
Sample output:
```
...
status:
  cassandraOperatorProgress: Updating
  conditions:
  - lastTransitionTime: "2021-03-30T22:01:48Z"
    message: ""
    reason: ""
    status: "True"
    type: ScalingUp
...
```

After the new nodes are deployed and running, the K8ssandra cass-operator automatically runs nodetool cleanup only on the original nodes and not the new nodes. This removes keys and data that are no longer associated with those original nodes.

What’s next

Run Cleanup operation to recover disk space from previously provisioned nodes.