Upgrade Kubernetes Operator for Apache Cassandra and related resources in Kubernetes

Use these steps to do either of the following:

Upgrade the version of Kubernetes Operator for Apache Cassandra and its related Kubernetes resources
Upgrade the Kubernetes cluster itself

An existing K8s cluster is required.

The resource values used to define the K8s cluster are preserved.

Upgrade paths for Kubernetes Operator for Apache Cassandra

Updates are supported from previous versions of Kubernetes Operator for Apache Cassandra released from the k8ssandra/cass-operator repository.

This guide covers upgrades from 1.7.1 to any version from 1.8.0 through 1.10.1. For versions beyond 1.10.1, see the cass-operator GitHub repository for potential breaking changes and upgrade paths for those versions.

Depending on your current version of cass-operator, you might need to perform multiple upgrades or take additional steps to prepare for the upgrade.

Upgrade from 1.7.1

When upgrading from 1.7.1, you must take additional steps due to changes in the underlying controller-runtime and newer Kubernetes versions:

Stricter validation requirements in newer Kubernetes versions require removing the preserveUnknownFields global property from the CRD before you can update to a newer CRD.
The newer controller-runtime modifies the liveness, readiness ,and configuration options, which require deleting the existing deployment.

Run the following commands manually before upgrading Kubernetes Operator for Apache Cassandra. Change the -n parameter to the namespace where you installed Kubernetes Operator for Apache Cassandra. These commands don’t delete your running Cassandra instances.

kubectl -n cass-operator delete deployment.apps/cass-operator
kubectl -n cass-operator delete service/cassandradatacenter-webhook-service
kubectl patch crd cassandradatacenters.cassandra.datastax.com -p '{"spec":{"preserveUnknownFields":false}}'

You can now install a new version of cass-operator as a fresh install.

Upgrade from 1.7.0 or earlier

If upgrading from versions older than 1.7.0 (released under datastax/cass-operator), you must upgrade to version 1.7.1 first, and then follow the steps in Upgrade from 1.7.1.

Upgrade to Kubernetes 1.23

If you need to upgrade Kubernetes to 1.23 from and earlier version, you must upgrade cass-operator to, at minimum, version 1.8.0 before upgrading Kubernetes. This is because version 1.7.1 and earlier don’t support Kubernetes 1.23.

Upgrade clusters

To upgrade Cassandra or DSE clusters in Kubernetes, modify and apply the revised configuration.

Replace prior cluster configuration

Use these steps to replace a previously created configuration in a Kubernetes (K8s) cluster that is already running Kubernetes Operator for Apache Cassandra with a defined Storage class.

When upgrading the cluster, Cassandra datacenter configurations must be recreated.

Remove the existing CassandraDatacenter configuration:

kubectl -n cass-operator delete -f path/to/existing/cassdc/config.yaml

Result

cassandradatacenter.cassandra.datastax.com "dc1" deleted

Create the new CassandraDatacenter configuration.

kubectl -n cass-operator create -f path/to/new/cassdc/config.yaml

Result

cassandradatacenter.cassandra.datastax.com/dc1 created

When to use forceUpgradeRacks

For scenarios where a pod ultimately doesn’t start, use forceUpgradeRacks in an edited CassandraDatacenter YAML configuration file to restart the rack that is hosting the pod.

In this example, a three-rack, three-node CassandraDatacenter was created with the following command and its referenced sample YAML configuration file:

kubectl -n cass-operator create -f https://raw.githubusercontent.com/k8ssandra/cass-operator/v1.7.1/operator/example-cassdc-yaml/dse-6.8.x/example-cassdc-three-rack-three-node.yaml

Assume a scenario where the configured cluster2-dc1-rack1-sts-0 pod doesn’t start. This would be reported in the Google Cloud Console in the Kubernetes Engine section on the cluster’s Workloads tab.

On a local machine with an established a connection to a Kubernetes project and cluster, use a kubectl command to invoke an editor for the target configuration.

For example, run the following command for namespace cass-operator, datacenter cassdc, and cluster dc1:
```
kubectl -n cass-operator edit cassdc dc1
```
In the editing session, add entries to identify the rack hosting the pod that would not start.

In this case, the three racks were defined previously when the datacenter was created using example-cassdc-three-rack-three-node.yaml. The following editing example adds the two-line forceUpgradeRacks: entry:
```
spec:
   forceUpgradeRacks:
   - rack1
   racks:
   - name: rack1
   - name: rack2
   - name: rack3
```
When you save the edited cassdc configuration, Kubernetes Operator for Apache Cassandra directly applies the requested forceUpgradeRacks action to the target rack in the cluster. Upon completion, Kubernetes Operator for Apache Cassandra removes the forceUpgradeRacks entry from the cassdc YAML configuration file.

After allowing time for the pods to restart, check the pods status again to see if the upgraded rack solved the issue:

kubectl -n cass-operator get pod

A successful restart returns output similar to the following:

NAME                             READY   STATUS    RESTARTS   AGE
cass-operator-78c9999797-gdnwd   1/1     Running   0          18h
cluster2-dc1-rack1-sts-0         2/2     Running   0          4m23s
cluster2-dc1-rack2-sts-0         2/2     Running   0          4m23s
cluster2-dc1-rack3-sts-0         2/2     Running   0          4m23s