nodetool tpstats
Displays usage statistics of thread pools.
The database separates different tasks into stages connected by a messaging service. Each stage has a queue and a thread pool. Some stages skip the messaging service and queue tasks immediately on a different stage when it exists on the same node. If the next stage is too busy, the database can back up a queue and lead to performance bottlenecks, as described in Monitoring a DataStax Enterprise cluster.
Reports are updated after SSTables change through compaction or flushing.
Report columns
The nodetool tpstats command report includes the following columns:
- Active
-
The number of
Activethreads.
- Pending
-
The number of
Pendingrequests waiting to be served by the thread pool.
- Backpressure
-
The number of requests that are
Pendingbecause the thread pool is full. HighBackpressurevalues indicate that the node is struggling to keep up with the incoming workload and needs to signal others nodes to slow down the rate of incoming requests.
- Delayed
-
The number of requests that are
Delayedbecause the thread pool is full.
- Stolen
-
The number of tasks that this thread pool has taken from other thread pools.
- Completed
-
The number of tasks
Completedby this thread pool.
- Blocked
-
The number of requests that are currently
Blockedbecause the thread pool for the next step in the service is full.
- All-Time Blocked
-
The total number of
All-Time Blockedrequests, which are all requests blocked in this thread pool up to now.
Report rows
The follow list describes the task or property associated with the task reported in the nodetool tpstats output.
General metrics
The following report aggregated statistics for tasks on the local node:
- BackgroundIoStage
-
Completes background tasks like submitting hints and deserializing the row cache.
- CompactionExecutor
-
Running compaction.
- GossipStage
-
Distributing node information via Gossip. Out of sync schemas can cause issues. You may have to sync using
nodetool resetlocalschema. - HintsDispatcher
-
Dispatches a single hints file to a specified node in a batched manner.
- InternalResponseStage
-
Responding to non-client initiated messages, including bootstrapping and schema checking.
- MemtableFlushWriter
-
Writing memtable contents to disk. May back up if the queue is overruns the disk I/O, or because of sorting processes.
nodetool tpstatsno longer reports blocked threads in theMemtableFlushWriterpool. Check the Pending Flushes metric reported bynodetool tblestats. - MemtablePostFlush
-
Cleaning up after flushing the memtable (discard commit logs and secondary indexes as needed).
- MemtableReclaimMemory
-
Making unused memory available.
- PendingRangeCalculator
-
Calculating pending ranges per bootstraps and departed nodes. Reporting by this tool isn’t considered useful.
- PerDiskMemtableFlushWriter_N
-
Activity for the memtable flush writer of each disk.
- ReadRepairStage
-
Performing read repairs. Usually fast if there is good connectivity between replicas.
Thread per core (TPC) task metrics
All actions in the TPC loop are labeled and observable.
Tasks marked Pendable are throttled, limited to the value set for tpc_concurrent_requests_limit in the cassandra.yaml.
Default: 128.
Thread per core messages are prepended with TPC/<type>, where:
-
TPC/<N> are metrics for the core number when
--coresis specified. -
TPC/other are metrics for tasks executed that are not on TPC threads.
-
TPC/all are the aggregate task metrics for all cores.
- UNKNOWN
-
Unknown task.
- FRAME_DECODE
-
Asynchronous frame decoding.
- READ_LOCAL
-
Single-partition read request from a local node generated directly from clients.
- READ_REMOTE
-
Single-partition read request from a remote replica.
- READ_TIMEOUT
-
Signals read timeout errors.
- READ_DEFERRED
-
Single-partition read request that will be first scheduled on an event loop (Pendable)
- READ_RESPONSE
-
Single-partition read response.
- READ_RANGE_LOCAL
-
Partition range read request from a local node generated directly from clients.
- READ_RANGE_REMOTE
-
Partition range read request from a remote replica.
- READ_RANGE_NODESYNC
-
Partition range read originating from NodeSync.
- READ_RANGE_INTERNAL
-
Range reads to internal tables.
- READ_RANGE_RESPONSE
-
Partition range read response.
- READ_FROM_ITERATOR
-
Switching thread to read from an iterator.
- READ_SECONDARY_INDEX
-
Switching thread to read from secondary index.
- READ_DISK_ASYNC
-
Waiting for data from disk.
- WRITE_LOCAL
-
Write request from a local node generated directly from clients.
- WRITE_REMOTE
-
Write request from a remote replica
- WRITE_INTERNAL
-
Writes to internal tables.
- WRITE_RESPONSE
-
Write response
- WRITE_DEFRAGMENT
-
Write issued to defragment data that required too many sstables to read (Pendable)
- WRITE_MEMTABLE
-
Switching thread to write in memtable when not already on the correct thread
- WRITE_POST_COMMITLOG_SEGMENT
-
Write request is waiting for the commit log segment to be allocated
- WRITE_POST_COMMITLOG_SYNC
-
Write request is waiting for commit log to sync to disk
- WRITE_POST_MEMTABLE_FULL
-
Write request is waiting for space in memtable
- BATCH_REPLAY
-
Replaying a batch mutation
- BATCH_STORE
-
Store a batchlog entry request (Pendable)
- BATCH_STORE_RESPONSE
-
Store a batchlog entry response
- BATCH_REMOVE
-
Remove a batchlog entry (Pendable)
- COUNTER_ACQUIRE_LOCK
-
Acquiring counter lock.
- EXECUTE_STATEMENT
-
Executing a statement.
- CAS
-
Executing compare-and-set (LWT).
- LWT_PREPARE
-
Preparation phase of lightweight transaction (Pendable).
- LWT_PROPOSE
-
Proposal phase of lightweight transaction (Pendable).
- LWT_COMMIT
-
Commit phase of lightn weight transaction (Pendable).
- TRUNCATE
-
Truncate request (Pendable).
- NODESYNC_VALIDATION
-
NodeSync validation of a partition.
- AUTHENTICATION
-
Authentication request.
- AUTHORIZATION
-
Authorization request.
- TIMED_UNKNOWN
-
Unknown timed task.
- TIMED_TIMEOUT
-
Scheduled timeout task.
- EVENTLOOP_SPIN
-
Number of busy spin cycles done by this TPC thread when it has no tasks to perform.
- EVENTLOOP_YIELD
-
Number of Thread.yield() calls done by this TPC thread when it has no tasks to perform.
- EVENTLOOP_PARK
-
Number of LockSupport.park() calls done by this TPC thread when it has no tasks to perform.
- HINT_DISPATCH
-
Hint dispatch request (Pendable).
- HINT_RESPONSE
-
Hint dispatch response.
- NETWORK_BACKPRESSURE
-
Scheduled network backpressure.
Client and internode requests
ClientRequest and InternodeMessage meters track the rate of client requests per TPC core and internode requests per TPC core respectively:
- Meter
-
Either CLIENT_REQUEST or INTERNODE_MESSAGE where TPC/<N> is the core number.
- One Minute Rate
-
Requests or messaging rate over a one minute period.
- Five Minute Rate
-
Requests or messaging rate over a five minute period.
- Fifteen Minute Rate
-
Requests or messaging rate over a fifteen minute period.
- Mean Rate
-
Mean rate of total requests or messages.
- Connections
-
Number of driver connections or inbound internode connections per thread.
|
Traffic for INTERNODE_MESSAGE meters will typically be quite low as opposed to CLIENT_REQUEST messages that indicate end user query execution. |
Droppable messages
The database generates the messages listed below, but discards them after a timeout.
The nodetool tpstats command reports the number of messages of each type that have been dropped.
You can view the messages themselves using a JMX client.
| Message Type | Stage | Notes |
|---|---|---|
BINARY |
n/a |
Deprecated. |
_TRACE |
n/a (special) |
Used for recording traces (nodetool settraceprobability). Has a special executor (1 thread, 1000 queue depth) that throws away messages on insertion instead of within the execute. |
MUTATION |
If a write message is processed after its timeout ( |
|
COUNTER_MUTATION |
If a write message is processed after its timeout ( |
|
READ_REPAIR |
Times out after |
|
READ |
Times out after |
|
RANGE_SLICE |
Times out after |
|
PAGED_RANGE |
Times out after |
|
REQUEST_RESPONSE |
Times out after |
Synopsis
nodetool [<connection_options>] tpstats
[-C] [-F json | yaml]
Syntax legend
| Syntax conventions | Description |
|---|---|
Italic, bold, or |
Syntax diagrams and code samples use one or more of these styles to mark placeholders for variable values. Replace placeholders with a valid option or your own user-defined value. In CQL statements, angle brackets are required to enclose data types in a set, list, map, or tuple.
Separate the data types with a comma.
For example: In Search CQL statements, angle brackets are used to identify the entity and literal value to overwrite the XML element in the schema and |
|
Square brackets surround optional command arguments. Do not type the square brackets. |
|
Parentheses identify a group to choose from. Do not type the parentheses. |
|
A pipe separates alternative elements. Type any one of the elements. Do not type the pipe. |
|
Indicates that you can repeat the syntax element as often as required. |
|
Single quotation marks must surround literal strings in CQL statements.
Use single quotation marks to preserve upper case.
+
For Search CQL only: Single quotation marks surround an entire XML schema declaration, such as |
|
Map collection.
Curly braces enclose maps ( |
|
Ends a CQL statement. |
|
Separate command line options from command arguments with two hyphens. This syntax is useful when arguments might be mistaken for command line options. |
Options
If an option has a short and long form, both forms are given, separated by a comma.
- -h, --host hostname
-
The hostname or IP address of a remote node or nodes. When omitted, the default is the local machine.
- -p, --port jmx_port
-
The JMX port number.
- -pw, --password jmxpassword
-
The JMX password for authenticating with secure JMX. If a password is not provided, you are prompted to enter one.
- -pwf, --password-file jmx_password_filepath
-
The filepath to the file that stores JMX authentication credentials.
- -u, --username jmx_username
-
The username for authenticating with secure JMX.
- -C, --cores
-
Include data for each core.
- -F, --format json | yaml
-
The format for the output. The default is plain text. The following wait latencies (in ms) are included in the following order: 50%, 75%, 95%, 98%, 99%, Min, and Max.
Examples
Get usage statistics for thread pools
nodetool tpstats
Example output:
Pools Active Pending Backpressure Delayed Shared Stolen Completed Blocked All time blocked
BackgroundIoStage 0 0 N/A N/A N/A N/A 45127 0 0
CompactionExecutor 0 0 N/A N/A N/A N/A 159 0 0
GossipStage 0 0 N/A N/A N/A N/A 1035 0 0
HintsDispatcher 0 0 N/A N/A N/A N/A 2 0 0
InternalResponseStage 0 0 N/A N/A N/A N/A 486 0 0
MemtableFlushWriter 0 0 N/A N/A N/A N/A 156 0 0
MemtablePostFlush 0 0 N/A N/A N/A N/A 251 0 0
MemtableReclaimMemory 0 0 N/A N/A N/A N/A 166 0 0
MigrationStage 0 0 N/A N/A N/A N/A 6 0 0
MiscStage 0 0 N/A N/A N/A N/A 552 0 0
PendingRangeCalculator 0 0 N/A N/A N/A N/A 7 0 0
PerDiskMemtableFlushWriter_0 0 0 N/A N/A N/A N/A 21 0 0
TPC/all/AUTHENTICATION 0 N/A N/A N/A 0 0 102 N/A N/A
TPC/all/AUTHORIZATION 0 N/A N/A N/A 0 0 7392 N/A N/A
TPC/all/EXECUTE_STATEMENT 0 N/A N/A N/A 125 125 329 N/A N/A
TPC/all/HINT_DISPATCH 0 0 N/A N/A 0 0 54711 N/A 0
TPC/all/HINT_RESPONSE 0 N/A N/A N/A 0 0 51825 N/A N/A
TPC/all/INCREMENTAL_NODESYNC_MAINTENANCE 0 N/A N/A N/A 0 0 1621 N/A N/A
TPC/all/NODESYNC_VALIDATION 0 N/A N/A N/A 0 0 780 N/A N/A
TPC/all/OUTBOUND_FLUSH 0 N/A N/A N/A 0 0 358222 N/A N/A
TPC/all/PARALLEL_MESSAGE_DECODE 0 N/A N/A N/A 152218 152218 333997 N/A N/A
TPC/all/READ_DISK_ASYNC 0 N/A N/A N/A 0 0 199 N/A N/A
TPC/all/READ_INTERNAL 0 N/A N/A N/A 48 48 3292 N/A N/A
TPC/all/READ_LOCAL 0 0 N/A N/A 88 88 126 N/A 0
TPC/all/READ_RANGE_INTERNAL 0 N/A N/A N/A 11 11 26 N/A N/A
TPC/all/READ_RANGE_NODESYNC 0 0 N/A N/A 0 0 1261 N/A 0
TPC/all/READ_RANGE_RESPONSE 0 N/A N/A N/A 11 11 25 N/A N/A
TPC/all/READ_REMOTE 0 0 N/A N/A 7 7 53 N/A 0
TPC/all/READ_RESPONSE 0 N/A N/A N/A 1403 1403 3500 N/A N/A
TPC/all/READ_SPECULATE 0 N/A N/A N/A 0 0 11 N/A N/A
TPC/all/SAVE_LOCAL_NODE 0 N/A N/A N/A 0 0 210 N/A N/A
TPC/all/SAVE_PEER_NODES 0 N/A N/A N/A 0 0 2 N/A N/A
TPC/all/TIMED_TIMEOUT 0 N/A N/A N/A 0 0 2388 N/A N/A
TPC/all/TIMED_UNKNOWN 0 N/A N/A N/A 0 0 7 N/A N/A
TPC/all/UNKNOWN 0 N/A N/A N/A 0 0 45127 N/A N/A
TPC/all/WRITE_AWAIT_COMMITLOG_SEGMENT 0 N/A N/A N/A 0 0 1 N/A N/A
TPC/all/WRITE_INTERNAL 0 N/A N/A N/A 0 0 1654 N/A N/A
TPC/all/WRITE_LOCAL 0 0 N/A N/A 0 0 124963 N/A 0
TPC/all/WRITE_REMOTE 0 0 N/A N/A 0 0 89510 N/A 0
TPC/all/WRITE_RESPONSE 0 N/A N/A N/A 59388 59388 455408 N/A N/A
TPC/all/WRITE_SWITCH_FOR_MEMTABLE 0 N/A N/A N/A 0 0 3892 N/A N/A
Meters One Minute Rate Five Minute Rate Fifteen Minute Rate Mean Rate Count Connections
TPC/all/CLIENT_REQUEST 500.09 286.90 120.74 524.70 125547 42
TPC/all/INTERNODE_MESSAGE 2088.58 1210.39 511.00 2222.53 531792 16
Messages Dropped Latency waiting in queue (micros)
50% 95% 99% Max
SCHEMA 0 0.00 81.92 81.92 98.30
SNAPSHOT 0 N/A N/A N/A N/A
RANGE_SLICE 0 0.00 0.00 0.00 0.00
READ 0 0.00 88.47 262.14 327.68
LWT 0 N/A N/A N/A N/A
OTHER 0 0.00 95.57 152.92 7340.03
BATCH_STORE 0 N/A N/A N/A N/A
VIEW_MUTATION 0 N/A N/A N/A N/A
REPAIR 0 N/A N/A N/A N/A
MULTI_RANGE_READ 0 N/A N/A N/A N/A
COUNTER_MUTATION 0 N/A N/A N/A N/A
READ_REPAIR 0 N/A N/A N/A N/A
NODESYNC 0 0.00 1164.41 1502.96 3145.73
MUTATION 0 67.57 187.31 559.95 234881.02
TRUNCATE 0 N/A N/A N/A N/A
HINT 0 70.92 97.38 503.59 234881.02
Get usage statistics for thread pools with information about each core
nodetool tpstats -C
Example output:
Pool Name Active Pending (w/Backpressure) Delayed Completed Blocked All time blocked
CompactionExecutor 0 0 (N/A) N/A 427751 0 0
GossipStage 0 0 (N/A) N/A 1567102 0 0
InternalResponseStage 0 0 (N/A) N/A 2 0 0
MemtableFlushWriter 0 0 (N/A) N/A 189 0 0
MemtablePostFlush 0 0 (N/A) N/A 712 0 0
MemtableReclaimMemory 0 0 (N/A) N/A 189 0 0
MigrationStage 0 0 (N/A) N/A 2 0 0
PendingRangeCalculator 0 0 (N/A) N/A 2 0 0
PerDiskMemtableFlushWriter_0 0 0 (N/A) N/A 187 0 0
ReadRepairStage 0 0 (N/A) N/A 1782 0 0
TPC/0 0 0 (0) 0 3044649 N/A 0
TPC/0/BATCH_REMOVE 0 0 (N/A) N/A 24989 N/A 0
TPC/0/EVENTLOOP_PROCESSED_TASKS 0 N/A (N/A) N/A 15260947 N/A N/A
TPC/0/EVENTLOOP_SCHEDULED_TASKS 0 N/A (N/A) N/A 26158321 N/A N/A
TPC/0/EVENTLOOP_SELECTOR_EVENTS 0 N/A (N/A) N/A 6141833 N/A N/A
TPC/0/EVENTLOOP_SELECT_CALLS 1 N/A (N/A) N/A 8998452 N/A N/A
TPC/0/EVENTLOOP_SELECT_NOW_CALLS 0 N/A (N/A) N/A 9009734 N/A N/A
TPC/0/EVENTLOOP_SPIN 0 N/A (N/A) N/A 80986317 N/A N/A
TPC/0/READ_DISK_ASYNC 0 N/A (N/A) N/A 131 N/A N/A
TPC/0/READ_INTERNAL 0 N/A (N/A) N/A 1012395 N/A N/A
TPC/0/READ_RANGE_INTERNAL 0 N/A (N/A) N/A 6996 N/A N/A
TPC/0/READ_RANGE_LOCAL 0 0 (N/A) N/A 1 N/A 0
TPC/0/READ_RANGE_RESPONSE ccccccccccc 0 N/A (N/A) N/A 210 N/A N/A
TPC/0/READ_SPECULATE 0 N/A (N/A) N/A 6 N/A N/A
TPC/0/READ_RESPONSE ccccccccccc 0 N/A (N/A) N/A 1064206 N/A N/A
TPC/0/TIMED_TIMEOUT 0 N/A (N/A) N/A 5225278 N/A N/A
TPC/0/UNKNOWN 0 N/A (N/A) N/A 1 N/A N/A
TPC/0/WRITE_INTERNAL 0 N/A (N/A) N/A 23998 N/A N/A
TPC/0/WRITE_LOCAL 0 0 (N/A) N/A 262963 N/A 0
TPC/0/WRITE_REMOTE 0 0 (N/A) N/A 25052 N/A 0
TPC/0/WRITE_MEMTABLE 0 N/A (N/A) N/A 87400 N/A N/A
TPC/0/WRITE_RESPONSE 0 N/A (N/A) N/A 536301 N/A N/A
TPC/all/BATCH_REMOVE 0 0 (N/A) N/A 24989 N/A 0
TPC/all/EVENTLOOP_PROCESSED_TASKS 0 N/A (N/A) N/A 15260957 N/A N/A
TPC/all/EVENTLOOP_SCHEDULED_TASKS 0 N/A (N/A) N/A 26158333 N/A N/A
TPC/all/EVENTLOOP_SELECTOR_EVENTS 0 N/A (N/A) N/A 6141828 N/A N/A
TPC/all/EVENTLOOP_SELECT_CALLS 1 N/A (N/A) N/A 8998452 N/A N/A
TPC/all/EVENTLOOP_SELECT_NOW_CALLS 0 N/A (N/A) N/A 9009735 N/A N/A
TPC/all/EVENTLOOP_SPIN 0 N/A (N/A) N/A 80986335 N/A N/A
TPC/all/EXECUTE_STATEMENT 0 N/A (N/A) N/A 56 N/A N/A
TPC/all/NODESYNC_VALIDATION 0 N/A (N/A) N/A 5202 N/A N/A
TPC/all/READ_DISK_ASYNC 0 N/A (N/A) N/A 131 N/A N/A
TPC/all/READ_INTERNAL 0 N/A (N/A) N/A 1012395 N/A N/A
TPC/all/READ_RANGE_INTERNAL 0 N/A (N/A) N/A 6996 N/A N/A
TPC/all/READ_RANGE_LOCAL 0 0 (N/A) N/A 1 N/A 0
TPC/all/READ_RANGE_RESPONSE 0 N/A (N/A) N/A 210 N/A N/A
TPC/all/READ_SPECULATE 0 N/A (N/A) N/A 6 N/A N/A
TPC/all/READ_RESPONSE 0 N/A (N/A) N/A 1064206 N/A N/A
TPC/all/TIMED_TIMEOUT 0 N/A (N/A) N/A 5225277 N/A N/A
TPC/all/UNKNOWN 0 N/A (N/A) N/A 1 N/A N/A
TPC/all/WRITE_INTERNAL 0 N/A (N/A) N/A 23998 N/A N/A
TPC/all/WRITE_LOCAL 0 0 (N/A) N/A 262963 N/A 0
TPC/all/WRITE_REMOTE 0 0 (N/A) N/A 25052 N/A 0
TPC/all/WRITE_MEMTABLE 0 N/A (N/A) N/A 87400 N/A N/A
TPC/all/WRITE_RESPONSE 0 N/A (N/A) N/A 536301 N/A N/A
TPC/other 0 0 (0) 0 5258 N/A 0
TPC/other/EXECUTE_STATEMENT 0 N/A (N/A) N/A 56 N/A N/A
TPC/other/NODESYNC_VALIDATION 0 N/A (N/A) N/A 5202 N/A N/A
ViewBuildExecutor 0 0 (N/A) N/A 32 0 0
Meters One Minute Rate Five Minute Rate Fifteen Minute Rate Mean Rate Count Connections
TPC/0/CLIENT_REQUEST 0.00 0.00 0.00 0.00 0 0
TPC/0/INTERNODE_MESSAGE 0.00 0.00 0.00 0.00 0 0
TPC/1/CLIENT_REQUEST 0.00 0.00 0.00 0.00 0 0
TPC/1/INTERNODE_MESSAGE 0.00 0.00 0.00 0.00 0 0
TPC/10/CLIENT_REQUEST 0.00 0.00 0.00 0.00 0 0
TPC/10/INTERNODE_MESSAGE 0.00 0.00 0.00 0.00 0 0
TPC/2/CLIENT_REQUEST 0.00 0.00 0.00 0.00 0 0
TPC/2/INTERNODE_MESSAGE 0.00 0.00 0.00 0.00 0 0
TPC/3/CLIENT_REQUEST 0.00 0.00 0.00 0.00 0 0
TPC/3/INTERNODE_MESSAGE 0.00 0.00 0.00 0.00 0 0
TPC/4/CLIENT_REQUEST 0.00 0.00 0.00 0.00 0 0
TPC/4/INTERNODE_MESSAGE 0.00 0.00 0.00 0.00 0 0
TPC/5/CLIENT_REQUEST 0.00 0.00 0.00 0.00 0 0
TPC/5/INTERNODE_MESSAGE 0.00 0.00 0.00 0.00 0 0
Message type Dropped Latency waiting in queue (micros)
50% 95% 99% Max
RANGE_SLICE 0 3670.02 3670.02 3670.02 4194.30
SNAPSHOT 0 N/A N/A N/A N/A
HINT 0 N/A N/A N/A N/A
COUNTER_MUTATION 0 N/A N/A N/A N/A
LWT 0 N/A N/A N/A N/A
BATCH_STORE 0 N/A N/A N/A N/A
VIEW_MUTATION 0 0.00 2621.44 8388.61 25165.82
READ 0 0.00 0.00 0.00 0.00
OTHER 0 1835.01 3670.02 3670.02 4194.30
REPAIR 0 N/A N/A N/A N/A
SCHEMA 0 0.00 50331.65 50331.65 58720.26
MUTATION 0 0.00 3670.02 3670.02 4194.30
NODESYNC 0 0.00 4194.30 4194.30 5242.88
READ_REPAIR 0 N/A N/A N/A N/A
TRUNCATE 0 N/A N/A N/A N/A