Inspect thread pool metrics
You can inspect and monitor thread pool (TP) statistics at the node and cluster level. However, TP statistics are generally intended for tracking node activity rather than cluster activity.
View TP statistics in Node Details
View thread pool statistics for active, pending, completed, blocked, and total blocked processes in Node Details:
-
Click cluster name, and then click Nodes.
-
In the Ring or List view, click the node that you want to inspect.
-
In Node Details, inspect metrics in the Thread Pool Stats pane.
The number of threads for each metric are reported in the Active, Pending, Completed, Blocked, and Total Blocked columns. N/A means that a particular thread stage isn’t applicable to that metric.
View TP statistics with OpsCenter services
-
Use OpsCenter Monitoring to create alerts and dashboards for thread pool metrics
Get TP statistics programmatically
-
Use the
nodetool tpstatscommand. -
Use the metrics retrieval endpoints.
TP metrics reference
The following sections list the thread pool metrics collected by OpsCenter.
Thread Pool metrics are prefixed with TP.
For equivalent keys used by the OpsCenter API, see TP metrics keys.
Authentication and authorization
-
TP: Authentication Active
Authentication Active
-
TP: Authentication Completed
Authentication Completed
-
TP: Auth Stage Active
Auth Stage Active
-
TP: Auth Stage Pending
Auth Stage Pending
-
TP: Auth Stage Completed
Auth Stage Completed
Background IO Stages
-
TP: Background IO Stage Active
Background IO Stage Active
-
TP: Background IO Stage Pending
Background IO Stage Pending
-
TP: Background IO Stage Completed
Background IO Stage Completed
Backpressure (Network)
See also Enable network backpressure.
-
TP: Network Backpressure Active
Network Backpressure Active
-
TP: Network Backpressure Pending
Network Backpressure Pending
-
TP: Network Backpressure Completed
Network Backpressure Completed
-
TP: Network Backpressure Total Blocked
Total Network Backpressure Blocked
Batches
Metrics for batch remove, batch replay, batch store, and batch store response.
-
TP: Batch Remove Active
Batch Remove Active
-
TP: Batch Remove Pending
Batch Remove Pending
-
TP: Batch Remove Completed
Batch Remove Completed
-
TP: Batch Remove Total Blocked
Total Batch Remove Blocked
-
TP: Batch Replay Active
Batch Replay Active
-
TP: Batch Replay Completed
Batch Replay Completed
-
TP: Batch Store Active
Batch Store Active
-
TP: Batch Store Pending
Batch Store Pending
-
TP: Batch Store Completed
Batch Store Completed
-
TP: Batch Store Total Blocked
Total Batch Store Blocked
-
TP: Batch Store Response Active
Batch Store Response Active
-
TP: Batch Store Response Completed
Batch Store Response Completed
Cache Cleaning
-
TP: Cache Cleaning Pending
Tasks pending to clean row caches during a cleanup compaction.
-
TP: Cache Cleaning Active
Tasks to clean row caches during a cleanup compaction.
-
TP: Cache Cleaning Completed
Tasks to clean row caches during a cleanup compaction.
Commit Log (Write Await)
-
TP: Write Await Commitlog Segment Active
Write Await Commitlog Segment Active
-
TP: Write Await Commitlog Sync Active
Write Await Commitlog Sync Active
-
TP: Write Await Commitlog Segment Completed
Write Await Commitlog Segment Completed
-
TP: Write Await Commitlog Sync Completed
Write Await Commitlog Sync Completed
Compaction Executor
-
TP: Compaction Executor Pending
Pending compactions that are known. This metric could deviate from "pending compactions," which includes an estimate of tasks that these pending tasks might create after completion.
-
TP: Compaction Executor Active
Active compactions that are known.
-
TP: Compaction Executor Completed
Completed compactions.
Counter Acquire Lock
-
TP: Counter Acquire Lock Active
Counter Acquire Lock Active
-
TP: Counter Acquire Lock Completed
Counter Acquire Lock Completed
Execute Statements
-
TP: Execute Statement Active
Execute Statement Active
-
TP: Execute Statement Completed
Execute Statement Completed
Flushes (memtables)
-
TP: Flushes Pending
Number of memtables queued for the flush process. A flush sorts and writes the memtables to disk.
-
TP: Post Flushes Pending
Tasks related to the last step in flushing memtables to disk as SSTables. Includes removing unnecessary commit log files and committing Solr-based secondary indexes.
-
TP: Flushes Active
Up to memtable_flush_writers concurrent tasks to flush and write the memtables to disk.
-
TP: Post Flushes Active
Tasks related to the last step in flushing memtables to disk as SSTables. Includes removing unnecessary commit log files and committing Solr-based secondary indexes.
-
TP: Flushes Completed
Number of memtables flushed to disk since the nodes start.
-
TP: Post Flushes Completed
Tasks related to the last step in flushing memtables to disk as SSTables. Includes removing unnecessary commit log files and committing Solr-based secondary indexes.
Frame Decode
-
TP: Frame Decode Active
Frame Decode Active
-
TP: Frame Decode Completed
Frame Decode Completed
Gossip
-
TP: Gossip Tasks Pending
Number of gossip messages and acknowledgments queued and waiting to be sent or received.
-
TP: Gossip Tasks Active
Number of gossip messages and acknowledgments actively being sent or received.
-
TP: Gossip Tasks Completed
Number of gossip messages and acknowledgments recently sent or received.
Hint Dispatch (TPC), Hint Dispatcher (regular TP), and Hint Response
-
TP: Hint Dispatch Active
Hint Dispatch Active
-
TP: Hint Dispatch Pending
Hint Dispatch Pending
-
TP: Hint Dispatch Completed
Hint Dispatch Completed
-
TP: Hint Dispatch Total Blocked
Total Hint Dispatch Blocked
-
TP: Hint Dispatcher Active
Up to max_hints_delivery_threads tasks, each dispatching all hinted handoffs to a host.
-
TP: Hint Dispatcher Pending
Pending tasks to send the stored hinted handoffs to a host.
-
TP: Hint Dispatcher Completed
Number of tasks to transfer hints to a host that have completed.
-
TP: Hint Response Active
Hint Response Active
-
TP: Hint Response Completed
Hint Response Completed
LWT (lightweight transactions)
LWT metrics for propose, prepare, and commit. For more information, see Use lightweight transactions (LWTs).
-
TP: Lwt Propose Pending
Lwt Propose Pending
-
TP: Lwt Propose Active
Lwt Propose Active
-
TP: Lwt Propose Completed
Lwt Propose Completed
-
TP: Lwt Prepare Active
Lwt Prepare Active
-
TP: Lwt Prepare Pending
Lwt Prepare Pending
-
TP: Lwt Prepare Completed
Lwt Prepare Completed
-
TP: Lwt Commit Pending
Lwt Commit Pending
-
TP: Lwt Commit Active
Lwt Commit Active
-
TP: Lwt Commit Completed
Lwt Commit Completed
-
TP: Lwt Commit Total Blocked
Total Lwt Commit Blocked
Memtable memory reclaims
The Memtable-reclaim-memory stats are available within the Thread Pool Stats pane of the Node details dialog.
-
TP: Memtable Reclaims Active
Waits for current reads to complete and then frees the memory formerly used by the obsoleted memtables.
-
TP: Memtable Reclaims Pending
Waits for current reads to complete and then frees the memory formerly used by the obsoleted memtables.
-
TP: Memtable Reclaims Completed
Waits for current reads to complete and then frees the memory formerly used by the obsoleted memtables.
Migrations
-
TP: Migrations Pending
Number of pending tasks from system methods that modified the schema.
-
TP: Migrations Active
Number of active tasks from system methods that modified the schema.
-
TP: Migrations Completed
Number of completed tasks from system methods that modified the schema.
NodeSync Validation
-
TP: NodeSync Validation Active
NodeSync Validation Active
-
TP: NodeSync Validation Completed
NodeSync Validation Completed
Reads
-
TP: Read Disk Async Active
Read Disk Async Active
-
TP: Read Disk Async Completed
Read Disk Async Completed
-
TP: Read Secondary Index Active
Read Secondary Index Active
-
TP: Read Secondary Index Completed
Read Secondary Index Completed
Read Range NodeSync
-
TP: Read Range NodeSync Active
Read Range NodeSync Active
-
TP: Read Range NodeSync Pending
Read Range NodeSync Pending
-
TP: Read Range NodeSync Completed
Read Range NodeSync Completed
-
TP: Read Range NodeSync Total Blocked
Total Read Range NodeSync Blocked
Writes
-
TP: Write Defragment Active
Write Defragment Active
-
TP: Write Defragment Pending
Write Defragment Pending
-
TP: Write Defragment Completed
Write Defragment Completed
-
TP: Write Defragment Total Blocked
Total Write Defragment Blocked
-
TP: Write Internal Active
Write Internal Active
-
TP: Write Internal Completed
Write Internal Completed
-
TP: Write Local Active
Write Local Active
-
TP: Write Local Pending
Write Local Pending
-
TP: Write Local Completed
Write Local Completed
-
TP: Write Local Total Blocked
Total Write Local Blocked
-
TP: Write Remote Active
Write Remote Active
-
TP: Write Remote Pending
Write Remote Pending
-
TP: Write Remote Completed
Write Remote Completed
-
TP: Write Remote Total Blocked
Total Write Remote Blocked
-
TP: Write Switch For Memtable Active
Write Switch For Memtable Active
-
TP: Write Switch For Memtable Completed
Write Switch For Memtable Completed
-
TP: Write Switch For Response Active
Write Switch For Response Active
-
TP: Write Switch For Response Completed
Write Switch For Response Completed
Unknown and Timeout
Unknown thread pool metrics monitor threads for undetermined processes.
-
TP: Timed Timeout Active
Timed Timeout Active
-
TP: Timed Timeout Completed
Timed Timeout Completed
-
TP: Timed Unknown Active
Timed Unknown Active
-
TP: Timed Unknown Completed
Timed Unknown Completed
-
TP: Unknown Active
Unknown Active
-
TP: Unknown Completed
Unknown Completed
View Build Executor
-
TP: View Build Executor Active
View Build Executor Active
-
TP: View Build Executor Pending
View Build Executor Pending
-
TP: View Build Executor Completed
View Build Executor Completed
