Add Nodes to a DataStax Enterprise Cluster or Datacenter
DataStax Mission Control is currently in Public Preview. DataStax Mission Control is not intended for production use, has not been certified for production workloads, and might contain bugs and other functional issues. There is no guarantee that DataStax Mission Control will ever become generally available. DataStax Mission Control is provided on an “AS IS” basis, without warranty or indemnity of any kind. If you are interested in trying out DataStax Mission Control please join the Public Preview. |
Adding nodes increases the capacity to service customer queries against the data.
This task information focuses on adding nodes to a single existing datacenter. To scale-up the number of datacenters, follow the much more efficient task that adds a DSE datacenter instead. |
Example
An existing k8ssandra datacenter with a cluster of 2 pods is installed based on the following k8ssandra-values.yaml
properties:
cassandra:
clusterName: org-k8ssandra
datacenters:
- name: dc1
size: 2
Workflow of user and operators
-
DataStax Mission Control User submits modified datacenter
size
parameter inMissionControlCluster
to theControl Plane
Kubernetes cluster. -
Cluster-level operator detects dc-level change in the cluster object and modifies dc-level resources.
-
DC-level operator detects
size
change in dc-level resource and provisions Kubernetes resources representing the new nodes. -
DC-level operator bootstraps DSE nodes on new pods.
When commissioning nodes, DataStax Mission Control:
-
targets the rack with the lowest number of active nodes.
-
uses a bootstrap (self-starting process) that adds nodes without external input.
-
commissions multiple nodes in a single rack happens only after adjusting other racks in the datacenter to reflect the desired node count.
-
identifies the number of nodes being added.
Limitations - You must increase the datacenter size by a multiple of the number of racks in the target datacenter. For example, with 3 racks you may scale up by 3, 6, or 9 nodes, and so on. Invalid size parameters are ignored.
-
Procedure
-
Modify the DC
size
parameter inMissionControlCluster
object.Example:
size: 3
.By default, DataStax Mission Control configures the Cassandra pods so that Kubernetes is blocked from scheduling multiple DSE pods on the same worker node. An attempt to increase the cluster size beyond the number of available worker nodes may result in the non-deployment of additional pods.
Here is a sample:
apiVersion: control.k8ssandra.io/v1alpha1 kind: CassandraTask metadata: name: cleanup-dc1 spec: datacenter: name: dc1 namespace: demo jobs: - name: cleanup-dc1 command: cleanup args: keyspace_name: my_keyspace
-
Submit that change to Kubernetes with the following command:
helm upgrade org-k8ssandra k8ssandra/k8ssandra -f k8ssandra-values.yaml
-
Monitor the bootstrap progress with this command to check the
Running
orPending
status of pods:kubectl get pods
If the results show a pod with
Pending
status, issue this command to get more details about the pod:kubectl describe pod <pod-name>
The results may indicate a
FailedScheduling
event. To override the default of only one Cassandra pod per Kubernetes worker node, set the optionallowMultipleNodesPerWorker: true
in the Helm chart file. Apply this configuration update to the cluster.Run the following command to check the status of the
CassandraDatacenter
object. In the output look for aScalingUp
condition with itsstatus
set totrue
.kubectl get cassandradatacenter dc1 -o yaml
Sample output:
... status: cassandraOperatorProgress: Updating conditions: - lastTransitionTime: "2021-03-30T22:01:48Z" message: "" reason: "" status: "True" type: ScalingUp ...
After the new nodes are deployed and running, the K8ssandra cass-operator automatically runs nodetool cleanup
only on the original nodes and not the new nodes. This removes keys and data that are no longer associated with those original nodes.
What’s next
Run Cleanup operation to recover disk space from previously provisioned nodes.