Restart the cluster

Restart all nodes in a cluster in a rolling fashion to provide zero downtime to applications. Restart nodes in a DataStax Enterprise (DSE), Hyper-Converged Database (HCD), or Apache Cassandra® cluster after adding or terminating one or more nodes.

Create the CassandraTask custom resource that defines a restart operation in the data plane Kubernetes cluster where the target cassandraDatacenter or MissionControlCluster resource and the datacenter are deployed.

Mission Control detects the restart task custom resource definition (CRD), iterates one rack at a time, and triggers and monitors restart operations one pod at a time. Mission Control reports task progress and status.

Performance impact

Running a restart task has no significant implications on the performance or stability of the cluster as long as the replication factor (RF) is set to 3 or higher. This RF setting tolerates one node being down while maintaining quorum read and write operations.

Prerequisites

A prepared environment on either bare-metal/VM or an existing Kubernetes cluster.

Restart the cluster

Decide to restart one or multiple datacenters, or all datacenters in the cluster. You have granular control over choosing target datacenters and their assigned rack.

Choose UI or CLI steps.

UI - one/multiple DCs
UI - all DCs
CLI

Access Mission Control’s UI.
In the Home Clusters dialog, click the target cluster namespace.
In the Nodes section of the Overview tab:
1. Click the row checkbox for your target datacenter.
2. Click more_vert More Options on your target datacenter.
3. Click Restart.
4. To restart more than one datacenter, repeat substeps a, b, and c for each target datacenter.

The restart activity starts immediately.

To view notifications from the restart operation, see Monitor restart activity status.

Access Mission Control’s UI.
In the Home Clusters dialog, click the target cluster namespace.
In the Nodes section of the Overview tab, click the Name checkbox. This selects all datacenter row checkboxes.
Click Bulk Actions.
In the Bulk Actions dialog:
1. Make sure that the Action type is Restart.
2. Make sure that All is your target datacenter selection.
  
  In the Racks field, All is the default and only selection.
3. Click Run.

The restart activity starts immediately.

To view notifications from the restart operation, see Monitor restart activity status.

Access Mission Control’s CLI.
Create the restart CassandraTask task manifest. This is a YAML configuration file that defines the task and the command in the kind section. The filename for this example is restart-task.cassandratask.yaml.

To perform a rolling restart on an entire cluster, you must create a separate restart CassandraTask per datacenter (DC). Restart nodes one datacenter at a time.

Here is a sample:
```
apiVersion: control.k8ssandra.io/v1alpha1
kind: CassandraTask
metadata:
  name: restart-dc
spec:
  datacenter:
    name: dc1
    namespace: demo
  jobs:
    - name: restart-dc1
      command: restart
```
Key options:
- metadata.name: a unique identifier within the Kubernetes namespace where the task is submitted. While the name can be any value, consider including the cluster and datacenter names to prevent collision with other actions.
- spec.datacenter: a unique namespace and name combination used to determine which datacenter to target with this operation
- spec.jobs[0].command: MUST be restart for this operation
- Optional: spec.jobs[0].args.keyspace_name: restricts this operation to a particular keyspace. Omitting this value results in ALL keyspaces being restarted. By default all keyspaces are rebuilt.
Submit the restart CassandraTask custom resource definition with this kubectl command:
```
kubectl apply -f restart-task.cassandratask.yaml
```
Submit the rebuild CassandraTask object to the Kubernetes cluster where the specified datacenter is deployed.
To view notifications from the restart operation, see Monitor restart activity status.

Monitor restart activity status

UI
CLI

Access Mission Control’s UI.
In the main navigation, click Activities.
See Status notifications regarding the progress of the restart activity.

A status of SUCCESS indicates the restart operation completed without issue. Timestamps are issued for the Start and End of the restart activity.

The Activities pane refreshes often and automatically.

Access Mission Control’s CLI.

Review the restart status with this command:

kubectl get cassandratask restart-dc1 | yq .status

Upon completion of the restart task, Mission Control updates or adds the following fields in the CassandraTask YAML file to indicate the status:

either a status.succeeded or status.failed field with a count of the pods that have been restarted and whether each restart was successful or unsuccessful
status.completionTime field: timestamp
status.condition.status: True

status.condition.type: Complete

Result

kind: CassandraTask
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"control.k8ssandra.io/v1alpha1","kind":"CassandraTask","metadata":{"annotations":{},"name":"restart-dc","namespace":"mission-control"},"spec":{"datacenter":{"name":"dc1","namespace":"mission-control"},"jobs":[{"command":"restart","name":"restart-dc1"}]}}
  creationTimestamp: "2022-10-15T21:24:35Z"
  generation: 2
  labels:
    app.kubernetes.io/created-by: cass-operator
    app.kubernetes.io/instance: cassandra-demo
    app.kubernetes.io/managed-by: cass-operator
    app.kubernetes.io/name: cassandra
    app.kubernetes.io/version: 6.8.26
    cassandra.datastax.com/cluster: demo
    cassandra.datastax.com/datacenter: dc1
    control.k8ssandra.io/status: completed
  name: restart-dc
  namespace: mission-control
  ownerReferences:
  - apiVersion: cassandra.datastax.com/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: CassandraDatacenter
    name: dc1
    uid: 51561d2f-2d20-4a16-b90c-fe9b0655f1ba
  resourceVersion: "780428"
  uid: 2202880f-16f0-4b83-a5ce-ab3057cd6d70
spec:
  concurrencyPolicy: Forbid
  datacenter:
    name: dc1
    namespace: mission-control
  jobs:
  - args: {}
    command: restart
    name: restart-dc1
  restartPolicy: Never
status:
  completionTime: "2022-10-15T21:36:54Z"
  conditions:
  - lastTransitionTime: "2022-10-15T21:36:54Z"
    status: "True"
    type: Complete
  startTime: "2022-10-15T21:24:35Z"

Was this helpful?