nodetool tpstats

Prints usage statistics of thread pools.

cassandra.yaml

The location of the cassandra.yaml file depends on the type of installation:
Package installations /etc/dse/cassandra/cassandra.yaml
Tarball installations installation_location/resources/cassandra/conf/cassandra.yaml

Prints usage statistics of thread pools. The DataStax Enterprise (DSE) database is based on a staged event-driven architecture (SEDA).

The database separates different tasks into stages connected by a messaging service. Each stage has a queue and a thread pool. Some stages skip the messaging service and queue tasks immediately on a different stage when it exists on the same node. The database can back up a queue if the next stage is too busy and lead to performance bottlenecks, as described in Monitoring a DataStax Enterprise cluster.

Reports are updated after SSTables change through compaction or flushing.

Report columns

The nodetool tpstats command report includes the following columns:
Active
The number of Active threads.
Pending
The number of Pending requests waiting to be executed by this thread pool.
Completed
The number of tasks Completed by this thread pool.
Blocked
The number of requests that are currently Blocked because the thread pool for the next step in the service is full.
All-Time Blocked
The total number of All-Time Blocked requests, which are all requests blocked in this thread pool up to now.

Report rows

The follow list describes the task or property associated with the task reported in the nodetool tpstats output.

General metrics

The following report aggregated statistics for tasks on the local node:
BackgroundIoStage
Completes background tasks like submitting hints and deserializing the row cache.
CompactionExecutor
Running compaction.
GossipStage
Distributing node information via Gossip. Out of sync schemas can cause issues. You may have to sync using nodetool resetlocalschema.
HintsDispatcher
Dispatches a single hints file to a specified node in a batched manner.
InternalResponseStage
Responding to non-client initiated messages, including bootstrapping and schema checking.
MemtableFlushWriter
Writing memtable contents to disk. May back up if the queue is overruns the disk I/O, or because of sorting processes.
Warning: nodetool tpstats no longer reports blocked threads in the MemtableFlushWriter pool. Check the Pending Flushes metric reported by nodetool tblestats.
MemtablePostFlush
Cleaning up after flushing the memtable (discarding commit logs and secondary indexes as needed).
MemtableReclaimMemory
Making unused memory available.
PendingRangeCalculator
Calculating pending ranges per bootstraps and departed nodes. Reporting by this tool is not useful — see Developer notes.
PerDiskMemtableFlushWriter_N
Activity for the memtable flush writer of each disk.
ReadRepairStage
Performing read repairs. Usually fast, if there is good connectivity between replicas.

Thread per core (TPC) task metrics

All actions in the TPC loop are labeled and therefore observable. Tasks marked Pendable are throttled, limited to the value set for tpc_concurrent_requests_limit in the cassandra.yaml (by default, 128). Thread per core messages are prepended with TPC/type, where:
  • TPC/N are metrics for the core number (when --cores is specified).
  • TPC/other are metrics for tasks executed that are not on TPC threads.
  • TPC/all are the aggregate task metrics for all cores.
UNKNOWN
Unknown task.
FRAME_DECODE
Asynchronous frame decoding.
READ_LOCAL
Single-partition read request from a local node generated directly from clients.
READ_REMOTE
Single-partition read request from a remote replica.
READ_TIMEOUT
Signals read timeout errors.
READ_DEFERRED
Single-partition read request that will be first scheduled on an event loop (Pendable)
READ_RESPONSE
Single-partition read response.
READ_RANGE_LOCAL
Partition range read request from a local node generated directly from clients.
READ_RANGE_REMOTE
Partition range read request from a remote replica.
READ_RANGE_NODESYNC
Partition range read originating from NodeSync.
READ_RANGE_INTERNAL
Range reads to internal tables.
READ_RANGE_RESPONSE
Partition range read response.
READ_FROM_ITERATOR
Switching thread to read from an iterator.
READ_SECONDARY_INDEX
Switching thread to read from secondary index.
READ_DISK_ASYNC
Waiting for data from disk.
WRITE_LOCAL
Write request from a local node generated directly from clients.
WRITE_REMOTE
Write request from a remote replica
WRITE_INTERNAL
Writes to internal tables.
WRITE_RESPONSE
Write response
WRITE_DEFRAGMENT
Write issued to defragment data that required too many sstables to read (Pendable)
WRITE_MEMTABLE
Switching thread to write in memtable when not already on the correct thread
WRITE_POST_COMMITLOG_SEGMENT
Write request is waiting for the commit log segment to be allocated
WRITE_POST_COMMITLOG_SYNC
Write request is waiting for commit log to sync to disk
WRITE_POST_MEMTABLE_FULL
Write request is waiting for space in memtable
BATCH_REPLAY
Replaying a batch mutation
BATCH_STORE
Store a batchlog entry request (Pendable)
BATCH_STORE_RESPONSE
Store a batchlog entry response
BATCH_REMOVE
Remove a batchlog entry (Pendable)
COUNTER_ACQUIRE_LOCK
Acquiring counter lock.
EXECUTE_STATEMENT
Executing a statement.
CAS
Executing compare-and-set (LWT).
LWT_PREPARE
Preparation phase of light-weight transaction (Pendable).
LWT_PROPOSE
Proposal phase of light-weight transaction (Pendable).
LWT_COMMIT
Commit phase of light-weight transaction (Pendable).
TRUNCATE
Truncate request (Pendable).
NODESYNC_VALIDATION
NodeSync validation of a partition.
AUTHENTICATION
Authentication request.
AUTHORIZATION
Authorization request.
TIMED_UNKNOWN
Unknown timed task.
TIMED_TIMEOUT
Scheduled timeout task.
EVENTLOOP_SPIN
Number of busy spin cycles done by this TPC thread when it has no tasks to perform.
EVENTLOOP_YIELD
Number of Thread.yield() calls done by this TPC thread when it has no tasks to perform.
EVENTLOOP_PARK
Number of LockSupport.park() calls done by this TPC thread when it has no tasks to perform.
HINT_DISPATCH
Hint dispatch request (Pendable).
HINT_RESPONSE
Hint dispatch response.
NETWORK_BACKPRESSURE
Scheduled network backpressure.

Droppable messages

The database generates the messages listed below, but discards them after a timeout. The nodetool tpstats command reports the number of messages of each type that have been dropped. You can view the messages themselves using a JMX client.

Message Type Stage Notes
BINARY n/a Deprecated
_TRACE n/a (special) Used for recording traces (nodetool settraceprobability) Has a special executor (1 thread, 1000 queue depth) that throws away messages on insertion instead of within the execute
MUTATION MutationStage If a write message is processed after its timeout (write_request_timeout_in_ms) it either sent a failure to the client or it met its requested consistency level and will relay on hinted handoff and read repairs to do the mutation if it succeeded.
COUNTER_MUTATION MutationStage If a write message is processed after its timeout (write_request_timeout_in_ms) it either sent a failure to the client or it met its requested consistency level and will relay on hinted handoff and read repairs to do the mutation if it succeeded.
READ_REPAIR MutationStage Times out after write_request_timeout_in_ms
READ ReadStage Times out after read_request_timeout_in_ms. No point in servicing reads after that point since it would of returned error to client
RANGE_SLICE ReadStage Times out after range_request_timeout_in_ms.
PAGED_RANGE ReadStage Times out after request_timeout_in_ms.
REQUEST_RESPONSE RequestResponseStage Times out after request_timeout_in_ms. Response was completed and sent back but not before the timeout

Synopsis

nodetool [connection_options] tpstats
[-C] [-F json | yaml]
Table 1. Legend
Syntax conventions Description
UPPERCASE Literal keyword.
Lowercase Not literal.
Italics Variable value. Replace with a valid option or user-defined value.
[ ] Optional. Square brackets ( [ ] ) surround optional command arguments. Do not type the square brackets.
( ) Group. Parentheses ( ( ) ) identify a group to choose from. Do not type the parentheses.
| Or. A vertical bar ( | ) separates alternative elements. Type any one of the elements. Do not type the vertical bar.
... Repeatable. An ellipsis ( ... ) indicates that you can repeat the syntax element as often as required.
'Literal string' Single quotation ( ' ) marks must surround literal strings in CQL statements. Use single quotation marks to preserve upper case.
{ key:value } Map collection. Braces ( { } ) enclose map collections or key value pairs. A colon separates the key and the value.
<datatype1,datatype2> Set, list, map, or tuple. Angle brackets ( < > ) enclose data types in a set, list, map, or tuple. Separate the data types with a comma.
cql_statement; End CQL statement. A semicolon ( ; ) terminates all CQL statements.
[ -- ] Separate the command line options from the command arguments with two hyphens ( -- ). This syntax is useful when arguments might be mistaken for command line options.
' <schema> ... </schema> ' Search CQL only: Single quotation marks ( ' ) surround an entire XML schema declaration.
@xml_entity='xml_entity_type' Search CQL only: Identify the entity and literal value to overwrite the XML element in the schema and solrconfig files.

Definition

The short form and long form parameters are comma-separated.

Connection options

-h, --host hostname
The hostname or IP address of a remote node or nodes. When omitted, the default is the local machine.
-p, --port jmx_port
The JMX port number.
-pw, --password jmxpassword
The JMX password for authenticating with secure JMX. If a password is not provided, you are prompted to enter one.
-pwf, --password-file jmx_password_filepath
The filepath to the file that stores JMX authentication credentials.
-u, --username jmx_username
The user name for authenticating with secure JMX.

Command arguments

-C, --cores
Include data for each core. The number of cores is determined by the tpc_cores.
-F, --format json | yaml
The format for the output. The default is plain text. The following wait latencies (in ms) are included in the following order: 50%, 75%, 95%, 98%, 99%, Min, and Max.

Examples

Run nodetool tpstats on the host labcluster

nodetool tpstats -C

Command output is:

Pool Name                                     Active      Pending (w/Backpressure)   Delayed      Completed   Blocked  All time blocked
CompactionExecutor                                 0                       0 (N/A)       N/A            196         0                 0
GossipStage                                        0                       0 (N/A)       N/A           2177         0                 0
InternalResponseStage                              0                       0 (N/A)       N/A              8         0                 0
MemtableFlushWriter                                0                       0 (N/A)       N/A             25         0                 0
MemtablePostFlush                                  0                       0 (N/A)       N/A            143         0                 0
MemtableReclaimMemory                              0                       0 (N/A)       N/A             25         0                 0
MigrationStage                                     0                       0 (N/A)       N/A              7         0                 0
PendingRangeCalculator                             0                       0 (N/A)       N/A              4         0                 0
PerDiskMemtableFlushWriter_0                       0                       0 (N/A)       N/A             25         0                 0
ReadRepairStage                                    0                       0 (N/A)       N/A              2         0                 0
TPC/0                                              0                         0 (0)         0           3470       N/A                 0
TPC/0/EVENTLOOP_SPIN                               0                     N/A (N/A)       N/A          49289       N/A               N/A
TPC/0/READ_DISK_ASYNC                              0                     N/A (N/A)       N/A             21       N/A               N/A
TPC/0/READ_INTERNAL                                0                     N/A (N/A)       N/A           1565       N/A               N/A
TPC/0/READ_RANGE_INTERNAL                          0                     N/A (N/A)       N/A             14       N/A               N/A
TPC/0/READ_SWITCH_FOR_RESPONSE                     0                     N/A (N/A)       N/A           1572       N/A               N/A
TPC/0/TIMED_TIMEOUT                                0                     N/A (N/A)       N/A           5005       N/A               N/A
TPC/0/UNKNOWN                                      0                     N/A (N/A)       N/A              1       N/A               N/A
TPC/0/WRITE_INTERNAL                               0                     N/A (N/A)       N/A             33       N/A               N/A
TPC/0/WRITE_SWITCH_FOR_MEMTABLE                    0                     N/A (N/A)       N/A            251       N/A               N/A
TPC/0/WRITE_SWITCH_FOR_RESPONSE                    0                     N/A (N/A)       N/A             13       N/A               N/A
TPC/all/EVENTLOOP_SPIN                             0                     N/A (N/A)       N/A          49307       N/A               N/A
TPC/all/NODESYNC_VALIDATION                        0                     N/A (N/A)       N/A              2       N/A               N/A
TPC/all/READ_DISK_ASYNC                            0                     N/A (N/A)       N/A             21       N/A               N/A
TPC/all/READ_INTERNAL                              0                     N/A (N/A)       N/A           1565       N/A               N/A
TPC/all/READ_RANGE_INTERNAL                        0                     N/A (N/A)       N/A             14       N/A               N/A
TPC/all/READ_SWITCH_FOR_RESPONSE                   0                     N/A (N/A)       N/A           1572       N/A               N/A
TPC/all/TIMED_TIMEOUT                              0                     N/A (N/A)       N/A           5003       N/A               N/A
TPC/all/UNKNOWN                                    0                     N/A (N/A)       N/A              1       N/A               N/A
TPC/all/WRITE_INTERNAL                             0                     N/A (N/A)       N/A             33       N/A               N/A
TPC/all/WRITE_SWITCH_FOR_MEMTABLE                  0                     N/A (N/A)       N/A            251       N/A               N/A
TPC/all/WRITE_SWITCH_FOR_RESPONSE                  0                     N/A (N/A)       N/A             13       N/A               N/A
TPC/other                                          0                         0 (0)         0              2       N/A                 0
TPC/other/NODESYNC_VALIDATION                      0                     N/A (N/A)       N/A              2       N/A               N/A

Message type            Dropped                  Latency waiting in queue (micros)
                                              50%               95%               99%               Max
RANGE_SLICE                   0               N/A               N/A               N/A               N/A
SNAPSHOT                      0               N/A               N/A               N/A               N/A
HINT                          0               N/A               N/A               N/A               N/A
COUNTER_MUTATION              0               N/A               N/A               N/A               N/A
LWT                           0               N/A               N/A               N/A               N/A
BATCH_STORE                   0               N/A               N/A               N/A               N/A
VIEW_MUTATION                 0               N/A               N/A               N/A               N/A
READ                          0              0.00            917.50            917.50           1048.58
OTHER                         0              0.00           5242.88           5242.88           6291.46
REPAIR                        0               N/A               N/A               N/A               N/A
SCHEMA                        0            917.50           1835.01           1835.01           2097.15
MUTATION                      0              0.00          14680.06          14680.06          16777.22
NODESYNC                      0            917.50          58720.26          58720.26          67108.86
READ_REPAIR                   0               N/A               N/A               N/A               N/A
TRUNCATE                      0               N/A               N/A               N/A               N/A