Rebuild a datacenter’s replicas

A rebuild operation runs the nodetool rebuild command, rebuilding data on a single node by streaming from another (source) datacenter (DC). Run this operation on each node after defining a source datacenter from which to stream data. This command streams data from another datacenter to rebuild the local DC’s replicas.

Redistribute data on remaining nodes after one or more nodes or datacenters are terminated or added.

Performance impact

Rebuilding nodes ensures high availability of data and avoids single points of failure.

Prerequisites

Example

A single datacenter (dc1) is deployed on the data plane Kubernetes cluster. A control plane Kubernetes cluster exists.

Workflow of user and operators

  1. Perform a backup of the node.

  2. User defines the rebuild CassandraTask. Specify the affected node as offline.

  3. User submits a rebuild CassandraTask to the data plane Kubernetes cluster where the datacenter is deployed.

  4. DC-operator detects new task custom resource definition (CRD).

  5. DC-operator iterates one rack at a time.

  6. DC-operator triggers and monitors rebuild operations one pod at a time.

  7. DC-operator reports task progress and status.

  8. User requests a status report of the rebuild CassandraTask with the kubectl command, and views the status response.

Rebuild a datacenter’s replicas

  1. Modify the rebuild-dc1.cassandratask.yaml file.

    Here is a sample:

    apiVersion: control.k8ssandra.io/v1alpha1
kind: CassandraTask
metadata:
  name: rebuild-dc1
spec:
  datacenter:
    name: dc1
    namespace: demo
  jobs:
    - name: rebuild-dc1
      command: rebuild
      args:
        keyspace_name: KEYSPACE_NAME

    Replace KEYSPACE_NAME with the name of the keyspace to rebuild.

    Key options:

    • metadata.name: a unique identifier within the Kubernetes namespace where the task is submitted. While the name can be any value, consider including the cluster name to prevent collision with other options.

    • spec.datacenter: a unique namespace and name combination used to determine which datacenter is the source for this operation.

    • spec.jobs[0].command: MUST be rebuild for this operation.

    • Optional: spec.jobs[0].args.keyspace_name: restricts this operation to a particular keyspace. Omitting this value results in ALL keyspaces being rebuilt. By default all keyspaces are rebuilt.

  2. Submit the rebuild CassandraTask CRD to data plane Kubernetes cluster where the datacenter is deployed with this command:

    kubectl apply -f rebuild-dc1.cassandratask.yaml

    Submit the rebuild CassandraTask object to the Kubernetes cluster where the specified datacenter is deployed.

    If nodetool rebuild is interrupted before completion, restart it by re-entering the command. The process resumes from the point at which it was interrupted.

Configure parallel rebuilds

In Mission Control version 1.18.0 and later, you can configure the number of nodes to rebuild in parallel to speed up the rebuild process. By default, Mission Control rebuilds nodes sequentially, one node at a time.

Use parallel rebuilds to:

  • Restore data after adding a new datacenter.

  • Minimize the time required for data synchronization.

  • Reduce overall rebuild duration for large datasets.

Parallel rebuilds increase network traffic and disk I/O across the cluster. Monitor cluster performance and adjust the parallelism level based on your infrastructure capacity.

Configure parallel rebuilds using two parameters:

  • jobsCount: Set this parameter at the cluster level (K8ssandraCluster or CassandraDatacenter) to control the number of compaction threads Cassandra uses for the rebuild operation.

  • maxConcurrentPods: Set this parameter at the task level (CassandraTask spec) to control how many nodes the operator processes simultaneously within a rack.

Configure cluster-level compaction threads

Configure the jobsCount parameter in your cluster’s K8ssandraCluster manifest:

apiVersion: k8ssandra.io/v1alpha1
kind: K8ssandraCluster
metadata:
  name: demo
spec:
  cassandra:
    datacenters:
      - metadata:
          name: dc1
        size: 3
        datacenters:
          - metadata:
              name: dc2
            size: 3
            jobsCount: 4

The jobsCount parameter controls the number of compaction threads Cassandra uses for the rebuild operation. By default, Cassandra uses all available compaction threads.

Configure task-level node parallelism

Add the maxConcurrentPods parameter to your CassandraTask spec to control how many nodes the operator processes simultaneously within a rack:

apiVersion: control.k8ssandra.io/v1alpha1
kind: CassandraTask
metadata:
  name: rebuild-dc1
spec:
  datacenter:
    name: dc1
    namespace: demo
  maxConcurrentPods: 3
  jobs:
    - name: rebuild-dc1
      command: rebuild
      args:
        keyspace_name: KEYSPACE_NAME

The maxConcurrentPods parameter specifies the maximum number of nodes that the operator processes simultaneously within a rack. Setting maxConcurrentPods: 3 allows up to three nodes to rebuild in parallel.

  1. Monitor the rebuild operation progress:

    kubectl get cassandratask rebuild-dc1 | yq .status
    Result
    ...
status:
  completionTime: "2022-10-23T23:34:38Z"
  conditions:
  - lastTransitionTime: "2022-10-23T23:34:08Z"
    status: "True"
    type: Running
  - lastTransitionTime: "2022-10-23T23:34:39Z"
    status: "True"
    type: Complete
  startTime: "2022-10-23T23:34:08Z"
  succeeded: 3

    The datacenter-level operators set the startTime field prior to starting the rebuild operation. They update the completionTime field when the rebuild operation is completed.

    The sample output indicates that the task is completed with the type: Complete status condition set to True. The succeeded: 3 field indicates that three nodes completed the requested task successfully. A failed field tracks a running count of pods that failed the rebuild operation.

  2. Monitor the node logs on the data plane cluster to verify that the datacenter is rebuilt with this command:

    kubectl logs -f demo-dc1-rack3-sts-1 -c server-system-logger | grep "finished rebuild"
    Result
    INFO  [pool-18-thread-1] 2022-10-23 23:32:18,088  StorageService.java:1895 - finished rebuild for (All keyspaces), (All tokens), 1 streaming connections, NORMAL,  included DCs: dc1 after 5 seconds receiving 3.52 MiB.

Was this helpful?

Give Feedback

How can we improve the documentation?

© Copyright IBM Corporation 2026 | Privacy policy | Terms of use |  Manage Privacy Choices

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: Contact IBM