Pending task metrics for cluster operations

Pending task metrics for cluster operations can indicate a backup of cluster operational processes such as those maintaining node consistency, system schemas, fault detection, and inter-node communications. Pending tasks for resource-intensive operations such as repair, bootstrap, or decommission are normal and expected while that operation is in progress, but should continue decreasing at a steady rate in a healthy cluster.

  • TP: Background IO Stage Pending [pending-background-io-stage]

    Background IO Stage Pending

  • TP: Batch Remove Pending [pending-batch-remove]

    Batch Remove Pending

  • TP: Batch Store Pending [pending-batch-store]

    Batch Store Pending

  • TP: Cache Cleaning Pending [pending-cache-cleanup-stage]

    Tasks pending to clean row caches during a cleanup compaction.

  • TP: Compaction Executor Pending [pending-compaction-executor]

    Pending compactions that are known. This metric could deviate from "pending compactions," which includes an estimate of tasks that these pending tasks might create after completion.

  • TP: Flushes Pending [pending-flushes]

    Number of memtables queued for the flush process. A flush sorts and writes the memtables to disk.

  • TP: Gossip Tasks Pending [pending-gossip-stage]

    Number of gossip messages and acknowledgments queued and waiting to be sent or received.

  • TP: Hint Dispatcher Pending [pending-hint-dispatcher]

    Pending tasks to send the stored hinted handoffs to a host.

  • TP: Index Management Pending [pending-secondary-index-management]

    Any initialization work when a new index instance is created. This may involve costly operations such as (re)building the index.

  • TP: Internal Responses Pending [pending-internal-response-stage]

    Number of pending tasks from internal tasks, such as nodes joining and leaving the cluster.

  • TP: Manual Repair Tasks Pending [pending-anti-entropy-stage]

    Repair tasks pending, such as handling the merkle tree transfer after the validation compaction.

  • TP: Memtable Reclaims Pending [memtable-reclaim-pending]

    Waits for current reads to complete and then frees the memory formerly used by the obsoleted memtables.

  • TP: Migrations Pending [pending-migration-stage]

    Number of pending tasks from system methods that modified the schema.

  • TP: Misc. Tasks Pending [pending-misc-stage]

    Number of pending tasks from infrequently run operations, such as taking a snapshot or processing the notification of a completed replication.

  • TP: Pending Range Calculator Pending [pending-pending-range-calculator]

    Pending tasks to calculate the ranges according to bootstrapping and leaving nodes.

  • TP: Pending Range Calculator Active [active-pending-range-calculator]

    Active tasks to calculate the ranges according to bootstrapping and leaving nodes.

  • TP: Post Flushes Pending [pending-memtable-post-flush]

    Tasks related to the last step in flushing memtables to disk as SSTables. Includes removing unnecessary commitlog files and committing Solr-based secondary indexes.

  • TP: Read Local Pending [pending-read-local]

    Read Local Pending

  • TP: Read Range Remote Pending [pending-read-range-remote]

    Read Range Remote Pending

  • TP: Read Remote Pending [pending-read-remote]

    Read Remote Pending

  • TP: Request Responses Pending [pending-request-response-stage]

    Number of pending callbacks to execute after a task on a remote node completes.

  • TP: Validation Executor Pending [pending-validation-executor]

    Pending task to read data from sstables and generate a merkle tree for a repair.

  • TP: View Build Executor Pending [pending-view-build-executor]

    View Build Executor Pending

  • TP: Write Defragment Pending [pending-write-defragment]

    Write Defragment Pending

  • TP: Write Local Pending [pending-write-local]

    Write Local Pending