Add Nodes to a DataStax Enterprise Cluster or Datacenter
DataStax Mission Control is current in Private Preview. It is subject to the beta agreement executed between you and DataStax. DataStax Mission Control is not intended for production use, has not been certified for production workloads, and might contain bugs and other functional issues. There is no guarantee that DataStax Mission Control will ever become generally available. DataStax Mission Control is provided on an “AS IS” basis, without warranty or indemnity of any kind.
If you are interested in trying out DataStax Mission Control please contact your DataStax account team.
Adding nodes increases the capacity to service customer queries against the data.
This task information focuses on adding nodes to a single existing datacenter.
To scale-up the number of datacenters, follow the much more efficient task that adds a DSE datacenter instead.
An existing k8ssandra datacenter with a cluster of 2 pods is installed based on the following
cassandra: clusterName: org-k8ssandra datacenters: - name: dc1 size: 2
Workflow of user and operators
DataStax Mission Control User submits modified datacenter
Control PlaneKubernetes cluster.
Cluster-level operator detects dc-level change in the cluster object and modifies dc-level resources.
DC-level operator detects
sizechange in dc-level resource and provisions Kubernetes resources representing the new nodes.
DC-level operator bootstraps DSE nodes on new pods.
When commissioning nodes, DataStax Mission Control:
targets the rack with the lowest number of active nodes.
uses a bootstrap (self-starting process) that adds nodes without external input.
commissions multiple nodes in a single rack happens only after adjusting other racks in the datacenter to reflect the desired node count.
identifies the number of nodes being added.
Limitations - You must increase the datacenter size by a multiple of the number of racks in the target datacenter. For example, with 3 racks you may scale up by 3, 6, or 9 nodes, and so on. Invalid size parameters are ignored.
Modify the DC
By default, DataStax Mission Control configures the Cassandra pods so that Kubernetes is blocked from scheduling multiple DSE pods on the same worker node. An attempt to increase the cluster size beyond the number of available worker nodes may result in the non-deployment of additional pods.
Here is a sample:
apiVersion: control.k8ssandra.io/v1alpha1 kind: CassandraTask metadata: name: cleanup-dc1 spec: datacenter: name: dc1 namespace: demo jobs: - name: cleanup-dc1 command: cleanup args: keyspace_name: my_keyspace
Submit that change to Kubernetes with the following command:
helm upgrade org-k8ssandra k8ssandra/k8ssandra -f k8ssandra-values.yaml
Monitor the bootstrap progress with this command to check the
Pendingstatus of pods:
kubectl get pods
If the results show a pod with
Pendingstatus, issue this command to get more details about the pod:
kubectl describe pod <pod-name>
The results may indicate a
FailedSchedulingevent. To override the default of only one Cassandra pod per Kubernetes worker node, set the option
allowMultipleNodesPerWorker: truein the Helm chart file. Apply this configuration update to the cluster.
Run the following command to check the status of the
CassandraDatacenterobject. In the output look for a
ScalingUpcondition with its
kubectl get cassandradatacenter dc1 -o yaml
... status: cassandraOperatorProgress: Updating conditions: - lastTransitionTime: "2021-03-30T22:01:48Z" message: "" reason: "" status: "True" type: ScalingUp ...
After the new nodes are deployed and running, the K8ssandra cass-operator automatically runs
nodetool cleanup only on the original nodes and not the new nodes. This removes keys and data that are no longer associated with those original nodes.
Run Cleanup operation to recover disk space from previously provisioned nodes.