Replace a node

Replacing a node destroys it and its data, forcing a replacement node that is clean and empty.

Run this operation when a node is defective and you need to create a new node that is identical to the node being replaced.

Mission Control detects the replacenode CassandraTask custom resource definition (CRD), iterates one rack at a time, and triggers and monitors replacement operations one pod at a time. Mission Control reports task progress and status.

Performance impact

This operation results in the complete replacement of a node with a new and empty node. The new node contains no data, but it retains the same token range as the node it is replacing. In this situation the new node bootstraps rebuilding its data from the remaining replicas within the cluster. This results in some disk pressure while the replacement node bootstraps.

Replace a defective node

Choose User Interface (UI) or Command Line Interface (CLI) steps.

  • UI

  • CLI

  1. Access Mission Control’s UI.

  2. In the Home Clusters dialog, click the target cluster namespace.

  3. In the Nodes section of the Overview tab, click the row checkbox for your target node in its datacenter.

  4. Click the overflow menu icon (3 dots) on your target node.

  5. Click Replace.

The replace activity starts immediately.

To view notifications from the replace operation, see Monitor replace activity status.

  1. Access Mission Control’s CLI.

  2. Modify the replace-node-task.cassandratask.yaml file to define a replacenode CassandraTask.

    Here is a sample:

    apiVersion: control.k8ssandra.io/v1alpha1
    kind: CassandraTask
    metadata:
      name: replace-dc1
    spec:
      datacenter:
        name: dc1
        namespace: demo
      jobs:
        - name: replace-dc1
          command: replacenode
          args:
            keyspace_name: my_keyspace

    Key options:

    • metadata.name: a unique identifer within the Kubernetes namespace where the task is submitted. While the name can be any value, consider including the cluster name to prevent collision with other options.

    • spec.datacenter: a unique namespace and name combination used to determine which datacenter to target with this operation.

    • spec.jobs[0].command: MUST be replacenode for this operation.

    • Optional: spec.jobs[0].args.keyspace_name: restricts this operation to a particular keyspace. Omitting this value results in ALL keyspaces being replaced. By default all keyspaces are rebuilt.

  3. Submit the replacenode CassandraTask custom resource definition to the Data Plane Kubernetes cluster where the target datacenter and its node are deployed:

    kubectl apply -f replace-node-task.cassandratask.yaml

    Mission Control detects and manages the modified CassandraTask custom resource definition (CRD). Mission Control stops the DSE node if it is running and then deletes the Persistent Volume(s) (PV). It then deletes the node (pod) where the DSE or Cassandra database is running. Mission Control deploys a new replacement node, starts it normally, and picks up the same token range as the deleted node.

Monitor replace activity status

Choose User Interface (UI) or Command Line Interface (CLI) steps.

  • UI

  • CLI

  1. Access Mission Control’s UI.

  2. In the main navigation, click Activities.

  3. See Status notifications regarding the progress of the replace activity.

    A status of SUCCESS indicates the replace operation completed without issue. Timestamps are issued for the Start and End of the replace activity.

The Activities pane refreshes often and automatically.

  1. Access Mission Control’s CLI.

  2. Monitor the progress and view the status of the CassandraTask object by issuing this command in the Control Plane cluster:

    kubectl get cassandratask replace-node -o yaml
    Sample results
    ...
    status:
      completionTime: "2022-11-01T03:28:33Z"
      conditions:
      - lastTransitionTime: "2022-11-01T03:28:12Z"
        status: "True"
        type: Running
      - lastTransitionTime: "2022-11-01T03:28:34Z"
        status: "False"
        type: Complete
      startTime: "2022-11-01T03:28:12Z"
      succeeded: 1

    Mission Control sets the startTime field prior to starting the replacenode operation. It updates the completionTime field when the replacenode operation is completed.

    The status field starts as "True" and is set to "False" when the replacenode operation completes. The type field changes from ReplacingNodes to Complete when the replacenode operation completes.

Was this helpful?

Give Feedback

How can we improve the documentation?

© 2024 DataStax | Privacy policy | Terms of use

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: +1 (650) 389-6000, info@datastax.com