Deploy configuration changes with datacenter canary deployments

Mission Control allows you to deploy configuration changes or version upgrades to a single datacenter first to validate changes before you apply them cluster-wide. This approach minimizes risk by testing changes in a controlled subset of your infrastructure.

In this procedure, a canary datacenter is not a separate test cluster or a special deployment type. It is one of the datacenters in your actual cluster that you choose as the first target for a change. After you validate the change in that datacenter, you must still apply the corresponding cluster-level update to roll the change out to the remaining datacenters. The initial datacenter-level override affects only the selected datacenter.

Prerequisites

A prepared Mission Control environment on an existing Kubernetes cluster

Deploy a configuration change with the canary datacenter strategy

To demonstrate the canary deployment strategy, this example changes the compaction throughput setting from 0 to 200 MB per second.

The example dc-canary cluster has two datacenters, east and west, each with three racks and one node per rack, with the following initial cluster-level configuration:

# spec.k8ssandra.cassandra
serverVersion: 6.9.20
serverImage: REGISTRY_ADDRESS/dse-server:6.9.20
config:
  cassandraYaml:
    compaction_throughput_mb_per_sec: 0
    dynamic_snitch: false

Deploy changes to the canary datacenter

Deploy the configuration change to the east datacenter first to validate it before you apply the corresponding cluster-wide update:

Update the MissionControlCluster object YAML file to specify the configuration under the east datacenter.

You don’t need to specify the entire configuration because the values merge with the cluster-level configuration.
```
# Note: Specify the `east` datacenter first in the YAML
# spec.k8ssandra.cassandra.datacenters[0].config
cassandraYaml:
  compaction_throughput_mb_per_sec: 200
```
Apply the changes:
```
kubectl apply -f dc-canary.yaml
```
The operators perform a rolling restart of the east datacenter with the new configuration.

Validate the canary datacenter

Verify that the configuration change deploys successfully to the canary datacenter before you update the rest of the cluster:

Verify that all pods in the east datacenter are online and ready:
```
kubectl get pods -l cassandra.datastax.com/datacenter=east
```

Monitor the datacenter status:

kubectl get cassandradatacenter east -o yaml

Test the configuration change in the east datacenter. If you identify issues, continue making changes within the east datacenter until the configuration is correct.

In a test environment, this validation step required approximately 14 minutes. Your cluster size and configuration will affect your timing.

Deploy changes cluster-wide

After you validate the changes in the canary datacenter, update the cluster-level configuration to apply them to the entire cluster:

Remove the configuration block from the east datacenter and apply it at the cluster level:

# Remove the following path:
# spec.k8ssandra.cassandra.datacenters[0].config

# spec.k8ssandra.cassandra.config
cassandraYaml:
  compaction_throughput_mb_per_sec: 200

Apply the changes:
```
kubectl apply -f dc-canary.yaml
```
The operators trigger a rolling restart on the west datacenter. The operators skip the east datacenter because it already has the change.

In a test environment, this deployment step required approximately 12 minutes. Your cluster size and configuration will affect your timing.

Upgrade the server version using a canary datacenter strategy

This example demonstrates upgrading the DSE server version from 6.9.20 to 6.9.21 using a canary deployment strategy.

This example is for demonstration purposes only.

To avoid data loss or corruption, all upgrades must follow the dedicated upgrade guides. For example, for DSE, see About upgrading.

Don’t use these commands to upgrade your clusters without preparing your clusters for upgrade. If your configuration is incompatible with the new version, or required upgrade steps are missed, then outages and data loss can occur. You will need to revert to a previous stable state, losing all interim mutations, and then restart the entire upgrade process.

Always record a backup before upgrading.

Deploy a version upgrade to a canary datacenter

Deploy the server version upgrade to the east datacenter first to validate it before you apply the corresponding cluster-wide update.

Update the MissionControlCluster object to specify the serverVersion and serverImage overrides under the east datacenter:
```
# Note: Specify the east datacenter first in the YAML
# spec.k8ssandra.cassandra.datacenters[0]
serverVersion: 6.9.21
serverImage: REGISTRY_ADDRESS/dse-server:6.9.21
```
Replace REGISTRY_ADDRESS with the address of your container registry.
Apply the changes:
```
kubectl apply -f dc-canary.yaml
```
The operators perform a rolling restart of the east datacenter with the new server version.

Validate the version upgrade in the canary datacenter

Verify that the version upgrade deploys successfully to the canary datacenter before you update the rest of the cluster.

Verify that all pods in the east datacenter are online and ready:
```
kubectl get pods -l cassandra.datastax.com/datacenter=east
```

Monitor the datacenter status:

kubectl get cassandradatacenter east -o yaml

Test the upgraded version in the east datacenter. If you identify issues, continue making changes within the east datacenter until the upgrade is successful.

In a test environment, this validation step required approximately 15 minutes. Your cluster size and configuration will affect your timing.

Deploy the version upgrade cluster-wide

After you validate the upgrade in the canary datacenter, update the cluster-level configuration to apply it to the entire cluster.

Remove the version block from the east datacenter and apply it at the cluster level:

# Remove the following paths:
# spec.k8ssandra.cassandra.datacenters[0].serverVersion
# spec.k8ssandra.cassandra.datacenters[0].serverImage

# spec.k8ssandra.cassandra
serverVersion: 6.9.21
serverImage: REGISTRY_ADDRESS/dse-server:6.9.21

Replace REGISTRY_ADDRESS with the address of your container registry.

Apply the changes:
```
kubectl apply -f dc-canary.yaml
```
The operators trigger a rolling restart on the west datacenter. The operators skip the east datacenter because it already has the change.