Restart the cluster

Restart all nodes in a cluster in a rolling fashion to provide zero downtime to applications. Restart nodes in a DataStax Enterprise (DSE) or Apache Cassandra® cluster after adding or terminating one or more nodes.

Create the CassandraTask custom resource that defines a restart operation in the Data Plane Kubernetes cluster where the target cassandraDatacenter or MissionControlCluster resource and the datacenter are deployed.

Mission Control detects the restart task custom resource definition (CRD), iterates one rack at a time, and triggers and monitors restart operations one pod at a time. Mission Control reports task progress and status.

Performance impact

Running a restart task has no significant implications on the performance or stability of the cluster as long as the replication factor (RF) is set to 3 or higher. This RF setting tolerates one node being down while maintaining quorum read and write operations.

Restart the cluster

Decide to restart one or multiple datacenters (DCs), or all datacenters in the cluster. You have granular control over choosing target datacenters and their assigned rack.

Choose User Interface (UI) or Command Line Interface (CLI) steps.

  • UI - one/multiple DCs

  • UI - all DCs

  • CLI

  1. Access Mission Control’s UI.

  2. In the Home Clusters dialog, click the target cluster namespace.

  3. In the Nodes section of the Overview tab:

    1. Click the row checkbox for your target datacenter.

    2. Click the overflow menu icon (3 dots) on your target datacenter.

    3. Click Restart.

    4. To restart more than one datacenter, repeat substeps a, b, and c for each target datacenter.

The restart activity starts immediately.

To view notifications from the restart operation, see Monitor restart activity status.

  1. Access Mission Control’s UI.

  2. In the Home Clusters dialog, click the target cluster namespace.

  3. In the Nodes section of the Overview tab, click the Name checkbox. This selects all datacenter row checkboxes.

  4. Click Bulk Actions.

  5. In the Bulk Actions dialog:

    1. Make sure that the Action type is Restart.

    2. Make sure that All is your target datacenter selection.

      In the Racks field, All is the default and only selection.

    3. Click Run.

The restart activity starts immediately.

To view notifications from the restart operation, see Monitor restart activity status.

  1. Access Mission Control’s CLI.

  2. Create the restart CassandraTask task manifest. This is a YAML configuration file that defines the task and the command in the kind section. The filename for this example is restart-task.cassandratask.yaml.

    In order to perform a rolling restart on an entire cluster, you must create a separate restart CassandraTask per datacenter (DC). Restart nodes one datacenter at a time.

    Here is a sample:

    apiVersion: control.k8ssandra.io/v1alpha1
    kind: CassandraTask
    metadata:
      name: restart-dc
    spec:
      datacenter:
        name: dc1
        namespace: demo
      jobs:
        - name: restart-dc1
          command: restart

    Key options:

    • metadata.name: a unique identifer within the Kubernetes namespace where the task is submitted. While the name can be any value, consider including the datacenter name to prevent collision with other options.

    • spec.datacenter: a unique namespace and name combination used to determine which datacenter to target with this operation

    • spec.jobs[0].command: MUST be restart for this operation

    • Optional: spec.jobs[0].args.keyspace_name: restricts this operation to a particular keyspace. Omitting this value results in ALL keyspaces being restarted. By default all keyspaces are rebuilt.

  3. Submit the restart CassandraTask custom resource definition with this kubectl command:

    kubectl apply -f restart-task.cassandratask.yaml

    Submit the rebuild CassandraTask object to the Kubernetes cluster where the specified datacenter is deployed.

  4. To view notifications from the restart operation, see Monitor restart activity status.

Monitor restart activity status

  • UI

  • CLI

  1. Access Mission Control’s UI.

  2. In the main navigation, click Activities.

  3. See Status notifications regarding the progress of the restart activity.

    A status of SUCCESS indicates the restart operation completed without issue. Timestamps are issued for the Start and End of the restart activity.

The Activities pane refreshes often and automatically.

  1. Access Mission Control’s CLI.

  2. Review the restart status with this command:

    kubectl get cassandratask restart-dc1 | yq .status

    Upon completion of the restart task, Mission Control updates or adds the following fields in the CassandraTask YAML file to indicate the status:

    • either a status.succeeded or status.failed field with a count of the pods that have been restarted and whether each restart was successful or unsuccessful

    • status.completionTime field: timestamp

    • status.condition.status: True

    • status.condition.type: Complete

      Sample results
      kind: CassandraTask
      metadata:
        annotations:
          kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"control.k8ssandra.io/v1alpha1","kind":"CassandraTask","metadata":{"annotations":{},"name":"restart-dc","namespace":"mission-control"},"spec":{"datacenter":{"name":"dc1","namespace":"mission-control"},"jobs":[{"command":"restart","name":"restart-dc1"}]}}
        creationTimestamp: "2022-10-15T21:24:35Z"
        generation: 2
        labels:
          app.kubernetes.io/created-by: cass-operator
          app.kubernetes.io/instance: cassandra-demo
          app.kubernetes.io/managed-by: cass-operator
          app.kubernetes.io/name: cassandra
          app.kubernetes.io/version: 6.8.26
          cassandra.datastax.com/cluster: demo
          cassandra.datastax.com/datacenter: dc1
          control.k8ssandra.io/status: completed
        name: restart-dc
        namespace: mission-control
        ownerReferences:
        - apiVersion: cassandra.datastax.com/v1beta1
          blockOwnerDeletion: true
          controller: true
          kind: CassandraDatacenter
          name: dc1
          uid: 51561d2f-2d20-4a16-b90c-fe9b0655f1ba
        resourceVersion: "780428"
        uid: 2202880f-16f0-4b83-a5ce-ab3057cd6d70
      spec:
        concurrencyPolicy: Forbid
        datacenter:
          name: dc1
          namespace: mission-control
        jobs:
        - args: {}
          command: restart
          name: restart-dc1
        restartPolicy: Never
      status:
        completionTime: "2022-10-15T21:36:54Z"
        conditions:
        - lastTransitionTime: "2022-10-15T21:36:54Z"
          status: "True"
          type: Complete
        startTime: "2022-10-15T21:24:35Z"

Was this helpful?

Give Feedback

How can we improve the documentation?

© 2024 DataStax | Privacy policy | Terms of use

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: +1 (650) 389-6000, info@datastax.com