Pending task metrics for cluster operations

Pending task metrics for cluster operations can indicate a backup of cluster operational processes such as those maintaining node consistency, system schemas, fault detection, and inter-node communications. Pending tasks for resource-intensive operations such as repair, bootstrap, or decommission are normal and expected while that operation is in progress, but should continue decreasing at a steady rate in a healthy cluster.

TP: Manual Repair Tasks Pending [pending-anti-entropy-stage]

Repair tasks pending, such as handling the merkle tree transfer after the validation compaction.

TP: Gossip Tasks Pending [pending-gossip-stage]

Number of gossip messages and acknowledgments queued and waiting to be sent or received.

Hinted handoff pending

While a node is offline, other nodes in the cluster save hints about rows that were updated during the time the node was unavailable. When a node comes back online, its corresponding replicas begin streaming the missed writes to the node to catch it up. The hinted handoff pending metric tracks the number of hints that are queued and waiting to be delivered after a failed node is back online again. High numbers of pending hints are commonly seen when a node is brought back online after some downtime. Viewing this metric can help you determine when the recovering node has been made consistent again. Hinted handoff is an optional feature of Cassandra. Hints are saved for a configurable period of time (an hour by default) before they are dropped. This prevents a large accumulation of hints caused by extended node outages.

TP: Migrations Pending [pending-migration-stage]

Number of pending tasks from system methods that modified the schema.

TP: Misc. Tasks Pending [pending-misc-stage]

Number of pending tasks from infrequently run operations, such as taking a snapshot or processing the notification of a completed replication.

Streams Pending

The progress of rows of data being streamed from the sending node. Streaming of data between nodes happens during operations such as bootstrap and decommission when one node sends large numbers of rows to another node.