nodetool tpstats
Returns usage statistics of thread pools.
The database separates different tasks into stages connected by a messaging service. Each stage has a queue and a thread pool. Some stages skip the messaging service and queue tasks immediately on a different stage when it exists on the same node. If the next stage is too busy, the database can back up a queue and lead to performance bottlenecks, as described in Monitor an HCD cluster.
Reports are updated after SSTables change through compaction or flushing.
|
HCD is built on Apache Cassandra® and includes additional capabilities for vector search. Vector search operations can impact thread pool usage patterns compared to standard Apache Cassandra® deployments. |
Report columns
The nodetool tpstats command report includes the following columns:
- Active
-
The number of
Activethreads.
- Pending
-
The number of
Pendingrequests waiting to be executed by this thread pool.
- Completed
-
The number of tasks
Completedby this thread pool.
- Blocked
-
The number of requests that are currently
Blockedbecause the thread pool for the next step in the service is full.
- All-Time Blocked
-
The total number of
All-Time Blockedrequests, which are all requests blocked in this thread pool up to now.
Report rows
The follow list describes the task or property associated with the task reported in the nodetool tpstats output.
General metrics
The following report aggregated statistics for tasks on the local node:
- BackgroundIoStage
-
Completes background tasks like submitting hints and deserializing the row cache.
- CompactionExecutor
-
Running compaction.
- GossipStage
-
Distributing node information via Gossip. Out of sync schemas can cause issues. You might have to sync using
`nodetool resetlocalschema. - HintsDispatcher
-
Dispatches a single hints file to a specified node in a batched manner.
- InternalResponseStage
-
Responding to non-client initiated messages, including bootstrapping and schema checking.
- MemtableFlushWriter
-
Writing memtable contents to disk. Might back up if the queue is overruns the disk I/O, or because of sorting processes.
nodetool tpstatsno longer reports blocked threads in the MemtableFlushWriter pool. Check the Pending Flushes metric reported bynodetool tblestats. - MemtablePostFlush
-
Cleaning up after flushing the memtable (discarding commit logs and secondary indexes as needed).
- MemtableReclaimMemory
-
Making unused memory available.
- PendingRangeCalculator
-
Calculating pending ranges per bootstraps and departed nodes. Reporting by this tool is not useful — see Developer notes.
- PerDiskMemtableFlushWriter_N
-
Activity for the memtable flush writer of each disk.
- ReadRepairStage
-
Performing read repairs. Usually fast, if there is good connectivity between replicas.
The main thread pool stages include:
- ReadStage
-
Handles read requests and range queries. When a read request comes in, it’s queued on the ReadStage and processed by one of its threads. Vector search operations and SAI queries are also processed through this stage. See the Pythian guide to Cassandra thread pools for detailed information about read request flow.
- WriteStage
-
Processes write operations including mutations, counter mutations, and read repairs. This stage handles all write-related tasks that need to be persisted to the database. Vector data writes and SAI index updates are processed through this stage.
- RequestResponseStage
-
Manages the response handling for completed requests. This stage processes the final responses before sending them back to clients.
- ReadRepairStage
-
Handles asynchronous read repair operations. This stage processes read repair tasks that are not in the client feedback loop, so they can queue up without affecting client response times.
- MutationStage
-
Processes mutation operations. This stage handles the actual data mutations and writes to the database.
- GossipStage
-
Manages cluster membership and gossip communication between nodes.
- AntiEntropyStage
-
Handles anti-entropy operations for data consistency across the cluster.
- MigrationStage
-
Manages schema changes and data migration operations.
- CompactionExecutor
-
Handles SSTable compaction operations to optimize storage and performance.
For HCD deployments with vector search and SAI indexing, the CompactionExecutor can experience increased load due to the additional indexing overhead required for vector data and SAI indexes.
For a comprehensive understanding of how these stages work together in the Staged Event Driven Architecture (SEDA), see the Pythian guide to Cassandra thread pools.
Droppable messages
The database generates the messages listed below, but discards them after a timeout.
The nodetool tpstats command reports the number of messages of each type that have been dropped.
You can view the messages themselves using a JMX client.
| Message Type | Stage | Notes |
|---|---|---|
BINARY |
n/a |
Deprecated. |
_TRACE |
n/a (special) |
Used for recording traces ( |
MUTATION |
If a write message is processed after its timeout ( |
|
COUNTER_MUTATION |
If a write message is processed after its timeout ( |
|
READ_REPAIR |
Times out after |
|
READ |
Times out after |
|
RANGE_SLICE |
Times out after |
|
PAGED_RANGE |
Times out after |
|
REQUEST_RESPONSE |
Times out after |
Synopsis
nodetool [<connection_options>] tpstats
[-C] [-F json | yaml]Definition
If a parameter has multiple forms, the following reference provides both the short form and long form, separated by a comma.
Connection options
- -h, --host hostname
-
The hostname or IP address of a remote node or nodes. When omitted, the default is the local machine.
- -p, --port jmx_port
-
The JMX port number.
- -pw, --password jmxpassword
-
The JMX password for authenticating with secure JMX. If a password is not provided, you are prompted to enter one.
- -pwf, --password-file jmx_password_filepath
-
The filepath to the file that stores JMX authentication credentials.
- -u, --username jmx_username
-
The username for authenticating with secure JMX.
Command arguments
- -C, --cores
-
Include data for each core.
- -F, --format json | yaml
-
The format for the output. The default is plain text. The following wait latencies (in ms) are included in the following order: 50%, 75%, 95%, 98%, 99%, Min, and Max.
Examples
Get usage statistics for thread pools
nodetool tpstatsResults:
Pool Name Active Pending (w/Backpressure) Delayed Completed Blocked All time blocked
CompactionExecutor 0 0 (N/A) N/A 427073 0 0
GossipStage 0 0 (N/A) N/A 1565077 0 0
InternalResponseStage 0 0 (N/A) N/A 2 0 0
MemtableFlushWriter 0 0 (N/A) N/A 183 0 0
MemtablePostFlush 0 0 (N/A) N/A 700 0 0
MemtableReclaimMemory 0 0 (N/A) N/A 183 0 0
MigrationStage 0 0 (N/A) N/A 2 0 0
PendingRangeCalculator 0 0 (N/A) N/A 2 0 0
PerDiskMemtableFlushWriter_0 0 0 (N/A) N/A 181 0 0
ReadRepairStage 0 0 (N/A) N/A 1779 0 0
TPC/all/BATCH_REMOVE 0 0 (N/A) N/A 24989 N/A 0
TPC/all/EVENTLOOP_PROCESSED_TASKS 0 N/A (N/A) N/A 15246816 N/A N/A
TPC/all/EVENTLOOP_SCHEDULED_TASKS 0 N/A (N/A) N/A 26137492 N/A N/A
TPC/all/EVENTLOOP_SELECTOR_EVENTS 0 N/A (N/A) N/A 6134785 N/A N/A
TPC/all/EVENTLOOP_SELECT_CALLS 1 N/A (N/A) N/A 8991407 N/A N/A
TPC/all/EVENTLOOP_SELECT_NOW_CALLS 0 N/A (N/A) N/A 9002688 N/A N/A
TPC/all/EVENTLOOP_SPIN 0 N/A (N/A) N/A 80922912 N/A N/A
TPC/all/EXECUTE_STATEMENT 0 N/A (N/A) N/A 56 N/A N/A
TPC/all/NODESYNC_VALIDATION 0 N/A (N/A) N/A 5194 N/A N/A
TPC/all/READ_DISK_ASYNC 0 N/A (N/A) N/A 129 N/A N/A
TPC/all/READ_INTERNAL 0 N/A (N/A) N/A 1010946 N/A N/A
TPC/all/READ_RANGE_INTERNAL 0 N/A (N/A) N/A 6984 N/A N/A
TPC/all/READ_RANGE_LOCAL 0 0 (N/A) N/A 1 N/A 0
TPC/all/READ_RANGE_RESPONSE 0 N/A (N/A) N/A 210 N/A N/A
TPC/all/READ_SPECULATE 0 N/A (N/A) N/A 6 N/A N/A
TPC/all/READ_RESPONSE 0 N/A (N/A) N/A 1062638 N/A N/A
TPC/all/TIMED_TIMEOUT 0 N/A (N/A) N/A 5218530 N/A N/A
TPC/all/UNKNOWN 0 N/A (N/A) N/A 1 N/A N/A
TPC/all/WRITE_INTERNAL 0 N/A (N/A) N/A 23958 N/A N/A
TPC/all/WRITE_LOCAL 0 0 (N/A) N/A 262963 N/A 0
TPC/all/WRITE_REMOTE 0 0 (N/A) N/A 25052 N/A 0
TPC/all/WRITE_MEMTABLE 0 N/A (N/A) N/A 87315 N/A N/A
TPC/all/WRITE_RESPONSE 0 N/A (N/A) N/A 536269 N/A N/A
ViewBuildExecutor 0 0 (N/A) N/A 32 0 0
Message type Dropped Latency waiting in queue (micros)
50% 95% 99% Max
RANGE_SLICE 0 3670.02 3670.02 3670.02 4194.30
SNAPSHOT 0 N/A N/A N/A N/A
HINT 0 N/A N/A N/A N/A
COUNTER_MUTATION 0 N/A N/A N/A N/A
LWT 0 N/A N/A N/A N/A
BATCH_STORE 0 N/A N/A N/A N/A
VIEW_MUTATION 0 0.00 2621.44 8388.61 25165.82
READ 0 0.00 0.00 0.00 0.00
OTHER 0 1835.01 3670.02 3670.02 4194.30
REPAIR 0 N/A N/A N/A N/A
SCHEMA 0 0.00 50331.65 50331.65 58720.26
MUTATION 0 0.00 5242.88 5242.88 6291.46
NODESYNC 0 1835.01 7340.03 7340.03 8388.61
READ_REPAIR 0 N/A N/A N/A N/A
TRUNCATE 0 N/A N/A N/A N/AGet usage statistics for thread pools with information about each core
nodetool tpstats -CResults:
Pool Name Active Pending (w/Backpressure) Delayed Completed Blocked All time blocked
CompactionExecutor 0 0 (N/A) N/A 427751 0 0
GossipStage 0 0 (N/A) N/A 1567102 0 0
InternalResponseStage 0 0 (N/A) N/A 2 0 0
MemtableFlushWriter 0 0 (N/A) N/A 189 0 0
MemtablePostFlush 0 0 (N/A) N/A 712 0 0
MemtableReclaimMemory 0 0 (N/A) N/A 189 0 0
MigrationStage 0 0 (N/A) N/A 2 0 0
PendingRangeCalculator 0 0 (N/A) N/A 2 0 0
PerDiskMemtableFlushWriter_0 0 0 (N/A) N/A 187 0 0
ReadRepairStage 0 0 (N/A) N/A 1782 0 0
TPC/0 0 0 (0) 0 3044649 N/A 0
TPC/0/BATCH_REMOVE 0 0 (N/A) N/A 24989 N/A 0
TPC/0/EVENTLOOP_PROCESSED_TASKS 0 N/A (N/A) N/A 15260947 N/A N/A
TPC/0/EVENTLOOP_SCHEDULED_TASKS 0 N/A (N/A) N/A 26158321 N/A N/A
TPC/0/EVENTLOOP_SELECTOR_EVENTS 0 N/A (N/A) N/A 6141833 N/A N/A
TPC/0/EVENTLOOP_SELECT_CALLS 1 N/A (N/A) N/A 8998452 N/A N/A
TPC/0/EVENTLOOP_SELECT_NOW_CALLS 0 N/A (N/A) N/A 9009734 N/A N/A
TPC/0/EVENTLOOP_SPIN 0 N/A (N/A) N/A 80986317 N/A N/A
TPC/0/READ_DISK_ASYNC 0 N/A (N/A) N/A 131 N/A N/A
TPC/0/READ_INTERNAL 0 N/A (N/A) N/A 1012395 N/A N/A
TPC/0/READ_RANGE_INTERNAL 0 N/A (N/A) N/A 6996 N/A N/A
TPC/0/READ_RANGE_LOCAL 0 0 (N/A) N/A 1 N/A 0
TPC/0/READ_RANGE_RESPONSE ccccccccccc 0 N/A (N/A) N/A 210 N/A N/A
TPC/0/READ_SPECULATE 0 N/A (N/A) N/A 6 N/A N/A
TPC/0/READ_RESPONSE ccccccccccc 0 N/A (N/A) N/A 1064206 N/A N/A
TPC/0/TIMED_TIMEOUT 0 N/A (N/A) N/A 5225278 N/A N/A
TPC/0/UNKNOWN 0 N/A (N/A) N/A 1 N/A N/A
TPC/0/WRITE_INTERNAL 0 N/A (N/A) N/A 23998 N/A N/A
TPC/0/WRITE_LOCAL 0 0 (N/A) N/A 262963 N/A 0
TPC/0/WRITE_REMOTE 0 0 (N/A) N/A 25052 N/A 0
TPC/0/WRITE_MEMTABLE 0 N/A (N/A) N/A 87400 N/A N/A
TPC/0/WRITE_RESPONSE 0 N/A (N/A) N/A 536301 N/A N/A
TPC/all/BATCH_REMOVE 0 0 (N/A) N/A 24989 N/A 0
TPC/all/EVENTLOOP_PROCESSED_TASKS 0 N/A (N/A) N/A 15260957 N/A N/A
TPC/all/EVENTLOOP_SCHEDULED_TASKS 0 N/A (N/A) N/A 26158333 N/A N/A
TPC/all/EVENTLOOP_SELECTOR_EVENTS 0 N/A (N/A) N/A 6141828 N/A N/A
TPC/all/EVENTLOOP_SELECT_CALLS 1 N/A (N/A) N/A 8998452 N/A N/A
TPC/all/EVENTLOOP_SELECT_NOW_CALLS 0 N/A (N/A) N/A 9009735 N/A N/A
TPC/all/EVENTLOOP_SPIN 0 N/A (N/A) N/A 80986335 N/A N/A
TPC/all/EXECUTE_STATEMENT 0 N/A (N/A) N/A 56 N/A N/A
TPC/all/NODESYNC_VALIDATION 0 N/A (N/A) N/A 5202 N/A N/A
TPC/all/READ_DISK_ASYNC 0 N/A (N/A) N/A 131 N/A N/A
TPC/all/READ_INTERNAL 0 N/A (N/A) N/A 1012395 N/A N/A
TPC/all/READ_RANGE_INTERNAL 0 N/A (N/A) N/A 6996 N/A N/A
TPC/all/READ_RANGE_LOCAL 0 0 (N/A) N/A 1 N/A 0
TPC/all/READ_RANGE_RESPONSE 0 N/A (N/A) N/A 210 N/A N/A
TPC/all/READ_SPECULATE 0 N/A (N/A) N/A 6 N/A N/A
TPC/all/READ_RESPONSE 0 N/A (N/A) N/A 1064206 N/A N/A
TPC/all/TIMED_TIMEOUT 0 N/A (N/A) N/A 5225277 N/A N/A
TPC/all/UNKNOWN 0 N/A (N/A) N/A 1 N/A N/A
TPC/all/WRITE_INTERNAL 0 N/A (N/A) N/A 23998 N/A N/A
TPC/all/WRITE_LOCAL 0 0 (N/A) N/A 262963 N/A 0
TPC/all/WRITE_REMOTE 0 0 (N/A) N/A 25052 N/A 0
TPC/all/WRITE_MEMTABLE 0 N/A (N/A) N/A 87400 N/A N/A
TPC/all/WRITE_RESPONSE 0 N/A (N/A) N/A 536301 N/A N/A
TPC/other 0 0 (0) 0 5258 N/A 0
TPC/other/EXECUTE_STATEMENT 0 N/A (N/A) N/A 56 N/A N/A
TPC/other/NODESYNC_VALIDATION 0 N/A (N/A) N/A 5202 N/A N/A
ViewBuildExecutor 0 0 (N/A) N/A 32 0 0
Meters One Minute Rate Five Minute Rate Fifteen Minute Rate Mean Rate Count Connections
TPC/0/CLIENT_REQUEST 0.00 0.00 0.00 0.00 0 0
TPC/0/INTERNODE_MESSAGE 0.00 0.00 0.00 0.00 0 0
TPC/1/CLIENT_REQUEST 0.00 0.00 0.00 0.00 0 0
TPC/1/INTERNODE_MESSAGE 0.00 0.00 0.00 0.00 0 0
TPC/10/CLIENT_REQUEST 0.00 0.00 0.00 0.00 0 0
TPC/10/INTERNODE_MESSAGE 0.00 0.00 0.00 0.00 0 0
TPC/2/CLIENT_REQUEST 0.00 0.00 0.00 0.00 0 0
TPC/2/INTERNODE_MESSAGE 0.00 0.00 0.00 0.00 0 0
TPC/3/CLIENT_REQUEST 0.00 0.00 0.00 0.00 0 0
TPC/3/INTERNODE_MESSAGE 0.00 0.00 0.00 0.00 0 0
TPC/4/CLIENT_REQUEST 0.00 0.00 0.00 0.00 0 0
TPC/4/INTERNODE_MESSAGE 0.00 0.00 0.00 0.00 0 0
TPC/5/CLIENT_REQUEST 0.00 0.00 0.00 0.00 0 0
TPC/5/INTERNODE_MESSAGE 0.00 0.00 0.00 0.00 0 0
Message type Dropped Latency waiting in queue (micros)
50% 95% 99% Max
RANGE_SLICE 0 3670.02 3670.02 3670.02 4194.30
SNAPSHOT 0 N/A N/A N/A N/A
HINT 0 N/A N/A N/A N/A
COUNTER_MUTATION 0 N/A N/A N/A N/A
LWT 0 N/A N/A N/A N/A
BATCH_STORE 0 N/A N/A N/A N/A
VIEW_MUTATION 0 0.00 2621.44 8388.61 25165.82
READ 0 0.00 0.00 0.00 0.00
OTHER 0 1835.01 3670.02 3670.02 4194.30
REPAIR 0 N/A N/A N/A N/A
SCHEMA 0 0.00 50331.65 50331.65 58720.26
MUTATION 0 0.00 3670.02 3670.02 4194.30
NODESYNC 0 0.00 4194.30 4194.30 5242.88
READ_REPAIR 0 N/A N/A N/A N/A
TRUNCATE 0 N/A N/A N/A N/A