Clean up nodes

The cleanup operation runs nodetool cleanup for either all or specific keyspaces on all nodes in the specified datacenter. Create the CassandraTask custom resource that defines a cleanup operation in the Data Plane Kubernetes cluster where the target cassandraDatacenter resource is deployed.

Mission Control detects the cleanup task custom resource definition (CRD), iterates one rack at a time, and triggers and monitors cleanup operations one pod at a time. Mission Control reports task progress and status.

DataStax Enterprise (DSE) does not automatically remove data from nodes that lose part of their partition range to a newly added node. After adding a node, run nodetool cleanup on the source node and on neighboring nodes that shared the same subrange. This prevents the database from including the old data in order to rebalance the load on that node. nodetool cleanup temporarily increases disk space use proportional to the size of the largest SSTable and triggers disk I/O.

Failure to run nodetool cleanup after adding a node may result in data inconsistencies, including resurrection of previously deleted data.

Performance impact

This nodetool cleanup operation forces all SSTables to compact on a node, which removes data that is no longer replicated to this node. As with all compactions this leads to an increase in disk operations and potential for latency. Depending on the amount of data present on the node and the query workload, you may want to schedule this cleanup operation during off-peak hours.

Clean up nodes in a cluster

Decide to clean up one, multiple, or all nodes.

Choose User Interface (UI) or Command Line Interface (CLI) steps.

  • UI - one node

  • UI - all nodes

  • UI - multiple nodes

  • CLI

  1. Access Mission Control’s UI.

  2. In the Home Clusters dialog, click the target cluster namespace.

  3. In the Nodes section of the Overview tab, click the overflow menu icon (3 dots) on the datacenter row of your target node.

  4. Click Cleanup from the list.

To review cleanup operation notifications, see Monitor cleanup activity status.

  1. Access Mission Control’s UI.

  2. In the Home Clusters dialog, click the target cluster namespace.

  3. In the Nodes section of the Overview tab, click the Name checkbox. This selects all node row checkboxes.

  4. Click Bulk Actions.

  5. In the Bulk Actions dialog:

    1. Make sure that the Action type is Cleanup.

    2. Make sure that your target datacenter is selected.

    3. Update the Rack field as needed.

    4. Click Run.

To review cleanup operation notifications, see Monitor cleanup activity status.

  1. Access Mission Control’s UI.

  2. In the Nodes section of the Overview tab, click the row checkbox for each of your target datacenters.

  3. Click Bulk Actions.

    1. In the Bulk Actions dialog:

    2. Make sure that the Action type is Cleanup.

    3. Make sure that your target datacenter is selected.

    4. Update the Rack field as needed.

    5. Click Run.

To review cleanup operation notifications, see Monitor cleanup activity status.

With an existing Kubernetes cluster, this example shows how to clean up one datacenter that has nine (9) nodes (pods) distributed across three (3) racks.

  1. Access Mission Control’s CLI.

  2. Modify the cleanup-dc1.cassandratask.yaml file, using this sample as a guide:

    apiVersion: control.k8ssandra.io/v1alpha1
    kind: CassandraTask
    metadata:
      name: cleanup-dc1
    spec:
      datacenter:
        name: dc1
        namespace: demo
      jobs:
        - name: cleanup-dc1
          command: cleanup
          args:
            keyspace_name: my_keyspace

    Key options:

    • metadata.name: a unique identifer within the Kubernetes namespace where the task is submitted. While the name can be any value, consider including the cluster name to prevent collision with other options.

    • spec.datacenter: a unique namespace and name combination used to determine which datacenter to target with this operation.

    • spec.jobs[0].command: MUST be cleanup for this operation.

    • Optional: spec.jobs[0].args.keyspace_name: restricts this operation to a particular keyspace. Omitting this value results in ALL keyspaces being cleaned up. By default all keyspaces are rebuilt.

      Although the jobs parameter is an array, only one entry is permitted. Specifying more than one job results in the task automatically failing.

  3. Submit the cleanup CassandraTask custom resource definition on the Kubernetes cluster where the specified datacenter is deployed:

    kubectl apply -f cleanup-dc1.cassandratask.yaml

    The DC-level operators perform a rolling cleanup operation, one node at a time. The order is determined lexicographically (also known as dictionary order), starting with rack names and then continuing with node (pod) names.

    As the cleanup operation begins, if it discovers a node in the process of being terminated and recreated, regardless of the reason, then the operation fails. In such an event, the DC-level operators retry the cleanup operation.

To review cleanup operation notifications, see Monitor cleanup activity status.

Monitor cleanup activity status

  • UI

  • CLI

  1. Access Mission Control’s UI.

  2. In the main navigation, click Activities.

  3. See Status notifications regarding the progress of the cleanup activity.

    A status of SUCCESS indicates the cleanup operation completed without issue. Timestamps are issued for the Start and End of the cleanup activity.

  1. Access Mission Control’s CLI.

  2. Monitor the cleanup operation progress with this kubectl command:

    kubectl get cassandratask cleanup-dc1 | yq .status
    Sample results
    ...
    status:
      completionTime: "2022-10-13T21:06:55Z"
      conditions:
      - lastTransitionTime: "2022-10-13T21:05:23Z"
        status: "True"
        type: Running
      - lastTransitionTime: "2022-10-13T21:06:55Z"
        status: "True"
        type: Complete
      startTime: "2022-10-13T21:05:23Z"
      succeeded: 9

    The DC-level operators set the startTime field prior to starting the cleanup operation. They update the completionTime field when the cleanup operation is completed.

    The sample output indicates that the task is completed with the type: Complete status condition set to True. The succeeded: 9 field indicates that nine (9) nodes (or pods) completed the requested task successfully. A failed field tracks a running count of pods that failed the cleanup operation.

Was this helpful?

Give Feedback

How can we improve the documentation?

© 2024 DataStax | Privacy policy | Terms of use

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: +1 (650) 389-6000, info@datastax.com