Provision Cassandra or DSE in Kubernetes with the Kubernetes Operator for Apache Cassandra
After you apply the Kubernetes Operator for Apache Cassandra configuration, you can provision Apache Cassandra® or DataStax Enterprise (DSE) in your Kubernetes cluster. To do this, you must define a cluster topology for the Cassandra Operator to create and monitor.
In this guide, a three-node cluster is provisioned, with one datacenter made up of three racks, with one node per rack.
The steps show how to indicate whether the serverType for your Kubernetes environment is cassandra or dse, plus related values.
|
For all the parameters in this guide, DataStax provides example YAML files that you can download and customize for your purposes. |
Define the cluster and datacenter parameters
A logical datacenter is the primary resource managed by the Cassandra Operator. Within a single Kubernetes namespace:
-
A single
CassandraDatacenterresource defines a single-datacenter cluster. -
Two or more
CassandraDatacenterresources with differentclusterNamevalues define separate and unrelated single-datacenter clusters. The Cassandra Operator manages both clusters because they reside within the same Kubernetes namespace. -
Two or more
CassandraDatacenterresources that share the sameclusterNamedefine a multi-datacenter cluster. The Cassandra Operator joins the instances in each datacenter into a logical topology that acts as a single cluster.
The provisioning example in this topic defines a single-datacenter cluster.
The cluster name is cluster1, and the datacenter name is dc1.
Define the Rack parameters
Cassandra and DSE are rack aware, and the racks parameter configures the Cassandra Operator to set up pods in a rack-aware way.
To specify the availability zone for a given rack, the Kubernetes worker nodes must have labels matching failure-domain.beta.kubernetes.io/zone.
For more information, see the Kubernetes documentation.
Racks must have identifiers.
In this topic, the example configuration identifies the racks as r1, r2, and r3.
Ensure the number of racks match the replication factor in the keyspaces you plan to create.
|
It is not easy to change the number of racks after deploying the |
Define the node count parameters
The size parameter is the number of nodes to run in the datacenter.
A node is a worker machine in Kubernetes and may be a virtual machine (VM) or a physical machine, depending on the cluster.
Multiple pods can run on one node.
For optimal performance, DataStax recommends running only one Cassandra or DSE server instance per Kubernetes worker node. The Cassandra Operator enforces that limit, and pods may get stuck in the Pending status if there are insufficient Kubernetes workers available.
You can use kubectl to get information about the nodes in the Kubernetes cluster:
kubectl get node --selector='!node-role.kubernetes.io/master'
This topic and its examples assume you have at least three worker nodes available.
If you are working on a minikube or other setup with a single Kubernetes worker node, then you must reduce the size value accordingly or set the allowMultipleNodesPerWorker parameter to true.
Define the storage parameters
Define the storage with a combination of the previously provisioned storage class and size parameters. These values inform the storage provisioner how much room to require from the backend.
Configure the database
The config key in the CassandraDatacenter resource contains the parameters used to configure the server process running in each pod.
In general, it is not necessary to specify any parameters here.
Settings omitted from the config key receive reasonable default values, and it is common to run demo clusters with no custom configuration.
If you are familiar with configuring Cassandra or DSE outside of containers on traditional operating systems, then you recognize that some familiar configuration parameters have been specified elsewhere in the CassandraDatacenter resource, outside of the config section.
Do not repeat these parameters inside of the config section of the provisioning YAML file; the Cassandra Operator populates them automatically from the CassandraDatacenter resource.
For example:
-
cluster_nameparameter, which is normally specified in thecassandra.yamlfile. The location of this file depends on the type of installation:-
Package installations:
/etc/dse/cassandra/cassandra.yaml -
Tarball installations:
INSTALL_LOCATION/resources/cassandra/conf/cassandra.yaml
-
-
rackanddatacenterparameters.
Similarly, the Cassandra Operator automatically populates any values that are normally customized on a per Cassandra node basis.
Do not specify these in the CassandraDatacenter resource.
For example, in the basis key, don’t specify initial_token, listen_address, or other ip-addresses.
A large number of keys and values can be specified in the config section.
The config key data structure resembles the DSE OpsCenter Lifecycle Manager (LCM) configuration profiles.
If needed, extrapolate the parameters to use from the LCM config profile options.
View information about the superuser credentials
By default, the Cassandra Operator creates a Cassandra superuser.
A Kubernetes secret is created, named <clusterName>-superuser, which contains username and password keys.
To define the superuser with custom credentials, see Define superuser with custom credentials.
The following example assumes that you deployed the CassandraDatacenter resource, as explained in Configuring Cassandra or DSE in Kubernetes with the Kubernetes Operator for Apache Cassandra.
Use kubectl to get information about the superuser with the default output format:
kubectl -n my-db-ns get secret cluster1-superuser
NAME TYPE DATA AGE
cluster1-superuser Opaque 2 13m
Get information about the superuser with output in YAML format:
kubectl -n my-db-ns get secret cluster1-superuser -o yaml
apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: cluster1-superuser
data:
password: d0...c2Vy
Extract the username and password values from the Kubernetes secret:
echo Y2x1c3RlcjEtc3VwZXJ1c2Vy | base64 -d
echo 'd0g0U...IdUE=' | base64 -d
The output from each command is the decoded string value for the username and password respectively.
Define superuser with custom credentials
To define the superuser with your own custom credentials rather than the default values, create a secret with kubectl:
kubectl create secret generic superuser-secret -f my-secret.yaml
To use the new superuser secret, specify the name of the secret in the CassandraDatacenter configuration YAML file that you load into the cluster:
apiVersion: datastax.com/v1beta1
kind: CassandraDatacenter
metadata:
name: dtcntr
spec:
superuserSecretName: superuser-secret
Specify the server type and version
In the spec section of the CassandraDatacenter resource configuration YAML file, specify the following:
-
serverType: Required. Must becassandraordse. -
serverVersion: Required.-
If
serverType: dse, then this value must be6.8.0or later. -
If
serverType: cassandra, then this value must be3.11.7or later.
-
-
serverImage: Optional. Only used as an override.DataStax recommends that you don’t specify
serverImagein the provisioning example file. When unspecified, the Cassandra Operator provides the appropriate image path, filename, and supported version.- Use a default image with
serverImagenot set -
apiVersion: cassandra.datastax.com/v1beta1 kind: CassandraDatacenter metadata: name: dtcntr spec: serverType: dse serverVersion: 6.8.4When you are provisioning Apache Cassandra in the Kubernetes cluster and want to use the default image, set
serverType: cassandra, and then specify theserverVersionfor your version of Cassandra, such asserverVersion: 3.11.7.This approach isn’t supported for all versions of Cassandra. Version 3.11.8 and later require the fully qualified image name, as shown in Use a specific Cassandra image.
- Use a specific Cassandra image
-
apiVersion: cassandra.datastax.com/v1beta1 kind: CassandraDatacenter metadata: name: dtcntr spec: serverType: cassandra serverVersion: 3.11.7 serverImage: datastax/cassandra-mgmtapi-3_11_9:v0.1.23Set the version in
serverVersion, and then provide the fully qualified image name inserverImage.If you’re using a private registry, provide the fully qualified image name within your registry, such as
my-private-docker-registry.example.com/datastax/cassandra-mgmtapi-3_11_9:v0.1.23.Check the DataStax Docker Hub server images for the latest build numbers, such as v0.1.23, for each release.
management-apiimages taggedv0.1.22orv0.1.23are built to run Cassandra asnon-root. However, by default, the Cassandra Operator assumes Cassandra is run asroot. For this reason, you must also setdockerImageRunsAsCassandra: truein theCassandraDatacenterspec. This setting prompts cass-operator to add aPodSecurityContextthat explicitly configures the Cassandra pod to run asnon-rootuser/group. - Use a specific DSE image
-
apiVersion: datastax.com/v1beta1 kind: CassandraDatacenter metadata: name: dtcntr spec: serverType: dse serverVersion: 6.8.4 serverImage: my-private-docker-registry.example.com/datastax.dse-server:6.8.4-20200731
- Use a default image with
Configure a NodePort service
Use the following fields to request a NodePort service in your CassandraDatacenter configuration YAML file:
spec:
networking:
nodePort:
cql: 30001
broadcast: 30002
To request the SSL versions of the ports, use the following fields:
spec:
networking:
nodePort:
cqlSSL: 30010
broadcastSSL: 30020
If any of the nodePort fields are already configured, a NodePort service is created that routes from the specified external port to the identically numbered internal port.
Cassandra is configured to listen on the specified ports.
The following is a complete example of a nodeport-service-dc.yaml file:
apiVersion: cassandra.datastax.com/v1beta1
kind: CassandraDatacenter
metadata:
name: dc1
spec:
clusterName: cluster1
serverType: dse
serverVersion: "6.8.4"
managementApiAuth:
insecure: {}
networking:
nodePort:
cql: 30001
broadcast: 30002
size: 2
storageConfig:
cassandraDataVolumeClaimSpec:
storageClassName: server-storage
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
racks:
- name: r1
- name: r2
config:
jvm-server-options:
initial_heap_size: "800m"
max_heap_size: "800m"
cassandra-yaml:
file_cache_size_in_mb: 100
memtable_space_in_mb: 100
Configure a nodeSelector
To pin pods to labeled K8s worker nodes in the Kubernetes cluster, use node selectors.
Define node selectors in a Pod spec, similar to the following example:
apiVersion: cassandra.datastax.com/v1beta1
kind: Pod
metadata:
name: my-db-pod
labels:
env: mytest
spec:
containers:
- name: my-db-pod
image: my-db-pod
imagePullPolicy: IfNotPresent
nodeSelector:
disktype: ssd
Then, apply the YAML file.
The following command assumes the local configuration file is named my-db-pod-fast-storage.yaml and the namespace is cass-operator:
kubectl apply -n cass-operator -f my-db-pod-fast-storage.yaml
Encryption
The Cassandra Operator automates the creation of key stores and trust stores for client-to-node and internode encryption.
For each datacenter created with the Cassandra Operator, credentials are injected into the stateful set using secrets with the name <datacenter-name>-keystore.
To use client-to-node or internode encryption, reference only the injected keystores from the Cassandra parameters provided in the datacenter configuration.
For an example, see the datacenter encryption test yamls.
Due to limitations of Kubernetes stateful sets, the current strategy primarily focuses on internode encryption with ca-only verification (peer name verification is not currently available).
Peer verification can be achieved with init containers capable of leveraging external certificate issuance architecture to enable per-node and per-client peer name verification.
By storing the certificate authority (CA) in Kubernetes secrets, you can create secrets ahead of time from user-provided or organizational certificate authorities. Leverage a single CA across multiple datacenters by copying the secrets generated for one datacenter to the secondary datacenter prior to launching the secondary datacenter.
While you could change from encrypted internode communications to unencrypted internode communications and the reverse, this change as a rolling configuration is not currently supported. The entire cluster must be stopped and started to update these features.
Download, optionally customize, and apply the provisioning YAML
-
If you have not already done so, download an example provisioning YAML file from the Cassandra Operator GitHub repository.
-
Customize the example YAML to suit your requirements, and then save the file.
The following examples use the file name
cluster1-dc1.yaml. -
Apply the YAML file:
kubectl -n my-db-ns apply -f ./cluster1-dc1.yamlThe Cassandra Operator begins the deployment in your Kubernetes cluster. The total deployment time depends on the number of nodes because each node takes a few minutes to deploy and start.
-
During the deployment, monitor the pods using
kubectlcommands to check theStatusandEvents:-
Check the
STATUSof the pods:kubectl -n my-db-ns get podsWhen the deployment is complete, all pods should report as
Running:NAME READY STATUS RESTARTS AGE cass-operator-f74447c57-kdf2p 1/1 Running 0 13m gke-cluster1-dc1-r1-sts-0 1/1 Running 0 5m38s gke-cluster1-dc1-r2-sts-0 1/1 Running 0 42s gke-cluster1-dc1-r3-sts-0 1/1 Running 0 6m7s -
Check the
StatusandEvents:kubectl -n my-db-ns describe cassdc dc1This output is useful for inspecting an ongoing or errored deployment:
... Status: Cassandra Operator Progress: Updating Last Server Node Started: 2021-01-30T11:37:28Z Super User Upserted: 2021-01-30T11:38:37Z Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal CreatedResource 9m49s cassandra-operator Created service cluster1-dc1-service Normal CreatedResource 9m49s cassandra-operator Created service cluster1-seed-service Normal CreatedResource 9m49s cassandra-operator Created service cluster1-dc1-all-pods-service Normal CreatedResource 9m49s cassandra-operator Created statefulset cluster1-dc1-r1-sts Normal CreatedResource 9m49s cassandra-operator Created statefulset cluster1-dc1-r2-sts Normal CreatedResource 9m49s cassandra-operator Created statefulset cluster1-dc1-r3-sts
-