nodetool tpstats

Returns usage statistics of thread pools.

The database separates different tasks into stages connected by a messaging service. Each stage has a queue and a thread pool. Some stages skip the messaging service and queue tasks immediately on a different stage when it exists on the same node. If the next stage is too busy, the database can back up a queue and lead to performance bottlenecks, as described in Monitoring a DataStax Enterprise cluster.

Reports are updated after SSTables change through compaction or flushing.

Report columns

The nodetool tpstats command report includes the following columns:

Active

The number of Active threads.

Pending

The number of Pending requests waiting to be executed by this thread pool.

Completed

The number of tasks Completed by this thread pool.

Blocked

The number of requests that are currently Blocked because the thread pool for the next step in the service is full.

All-Time Blocked

The total number of All-Time Blocked requests, which are all requests blocked in this thread pool up to now.

Report rows

The follow list describes the task or property associated with the task reported in the nodetool tpstats output.

General metrics

The following report aggregated statistics for tasks on the local node:

BackgroundIoStage

Completes background tasks like submitting hints and deserializing the row cache.

CompactionExecutor

Running compaction.

GossipStage

Distributing node information via Gossip. Out of sync schemas can cause issues. You may have to sync using nodetool resetlocalschema.

HintsDispatcher

Dispatches a single hints file to a specified node in a batched manner.

InternalResponseStage

Responding to non-client initiated messages, including bootstrapping and schema checking.

MemtableFlushWriter

Writing memtable contents to disk. May back up if the queue is overruns the disk I/O, or because of sorting processes.

nodetool tpstats no longer reports blocked threads in the MemtableFlushWriter pool. Check the Pending Flushes metric reported by nodetool tblestats.

MemtablePostFlush

Cleaning up after flushing the memtable (discarding commit logs and secondary indexes as needed).

MemtableReclaimMemory

Making unused memory available.

PendingRangeCalculator

Calculating pending ranges per bootstraps and departed nodes. Reporting by this tool is not useful — see Developer notes.

PerDiskMemtableFlushWriter_N

Activity for the memtable flush writer of each disk.

ReadRepairStage

Performing read repairs. Usually fast, if there is good connectivity between replicas.

Thread per core (TPC) task metrics

All actions in the TPC loop are labeled and observable. Tasks marked Pendable are throttled, limited to the value set for tpc_concurrent_requests_limit in the cassandra.yaml. Default: 128.

Thread per core messages are prepended with TPC/<type>, where:

  • TPC/<N> are metrics for the core number when --cores is specified.

  • TPC/other are metrics for tasks executed that are not on TPC threads.

  • TPC/all are the aggregate task metrics for all cores.

UNKNOWN

Unknown task.

FRAME_DECODE

Asynchronous frame decoding.

READ_LOCAL

Single-partition read request from a local node generated directly from clients.

READ_REMOTE

Single-partition read request from a remote replica.

READ_TIMEOUT

Signals read timeout errors.

READ_DEFERRED

Single-partition read request that will be first scheduled on an event loop (Pendable)

READ_RESPONSE

Single-partition read response.

READ_RANGE_LOCAL

Partition range read request from a local node generated directly from clients.

READ_RANGE_REMOTE

Partition range read request from a remote replica.

READ_RANGE_NODESYNC

Partition range read originating from NodeSync.

READ_RANGE_INTERNAL

Range reads to internal tables.

READ_RANGE_RESPONSE

Partition range read response.

READ_FROM_ITERATOR

Switching thread to read from an iterator.

READ_SECONDARY_INDEX

Switching thread to read from secondary index.

READ_DISK_ASYNC

Waiting for data from disk.

WRITE_LOCAL

Write request from a local node generated directly from clients.

WRITE_REMOTE

Write request from a remote replica

WRITE_INTERNAL

Writes to internal tables.

WRITE_RESPONSE

Write response

WRITE_DEFRAGMENT

Write issued to defragment data that required too many sstables to read (Pendable)

WRITE_MEMTABLE

Switching thread to write in memtable when not already on the correct thread

WRITE_POST_COMMITLOG_SEGMENT

Write request is waiting for the commit log segment to be allocated

WRITE_POST_COMMITLOG_SYNC

Write request is waiting for commit log to sync to disk

WRITE_POST_MEMTABLE_FULL

Write request is waiting for space in memtable

BATCH_REPLAY

Replaying a batch mutation

BATCH_STORE

Store a batchlog entry request (Pendable)

BATCH_STORE_RESPONSE

Store a batchlog entry response

BATCH_REMOVE

Remove a batchlog entry (Pendable)

COUNTER_ACQUIRE_LOCK

Acquiring counter lock.

EXECUTE_STATEMENT

Executing a statement.

CAS

Executing compare-and-set (LWT).

LWT_PREPARE

Preparation phase of lightweight transaction (Pendable).

LWT_PROPOSE

Proposal phase of lightweight transaction (Pendable).

LWT_COMMIT

Commit phase of lightn weight transaction (Pendable).

TRUNCATE

Truncate request (Pendable).

NODESYNC_VALIDATION

NodeSync validation of a partition.

AUTHENTICATION

Authentication request.

AUTHORIZATION

Authorization request.

TIMED_UNKNOWN

Unknown timed task.

TIMED_TIMEOUT

Scheduled timeout task.

EVENTLOOP_SPIN

Number of busy spin cycles done by this TPC thread when it has no tasks to perform.

EVENTLOOP_YIELD

Number of Thread.yield() calls done by this TPC thread when it has no tasks to perform.

EVENTLOOP_PARK

Number of LockSupport.park() calls done by this TPC thread when it has no tasks to perform.

HINT_DISPATCH

Hint dispatch request (Pendable).

HINT_RESPONSE

Hint dispatch response.

NETWORK_BACKPRESSURE

Scheduled network backpressure.

Client and internode requests

ClientRequest and InternodeMessage meters track the rate of client requests per TPC core and internode requests per TPC core respectively:

Meter

Either CLIENT_REQUEST or INTERNODE_MESSAGE where TPC/<N> is the core number.

One Minute Rate

Requests or messaging rate over a one minute period.

Five Minute Rate

Requests or messaging rate over a five minute period.

Fifteen Minute Rate

Requests or messaging rate over a fifteen minute period.

Mean Rate

Mean rate of total requests or messages.

Connections

Number of driver connections or inbound internode connections per thread.

Traffic for INTERNODE_MESSAGE meters will typically be quite low as opposed to CLIENT_REQUEST messages that indicate end user query execution.

Droppable messages

The database generates the messages listed below, but discards them after a timeout. The nodetool tpstats command reports the number of messages of each type that have been dropped. You can view the messages themselves using a JMX client.

Message Type Stage Notes

BINARY

n/a

Deprecated.

_TRACE

n/a (special)

Used for recording traces (nodetool settraceprobability). Has a special executor (1 thread, 1000 queue depth) that throws away messages on insertion instead of within the execute.

MUTATION

MutationStage

If a write message is processed after its timeout (write_request_timeout_in_ms), it either sent a failure to the client or it met its requested consistency level and will relay on hinted handoff and read repairs to do the mutation if it succeeded.

COUNTER_MUTATION

MutationStage

If a write message is processed after its timeout (write_request_timeout_in_ms), it either sent a failure to the client or it met its requested consistency level and will relay on hinted handoff and read repairs to do the mutation.

READ_REPAIR

MutationStage

Times out after write_request_timeout_in_ms.

READ

ReadStage

Times out after read_request_timeout_in_ms. No point in servicing reads after that point because it would have returned error to client.

RANGE_SLICE

ReadStage

Times out after range_request_timeout_in_ms.

PAGED_RANGE

ReadStage

Times out after request_timeout_in_ms.

REQUEST_RESPONSE

RequestResponseStage

Times out after request_timeout_in_ms. Response was completed and sent back but not before the timeout.

Synopsis

nodetool [<connection_options>] tpstats
[-C] [-F json | yaml]
Syntax conventions Description

UPPERCASE

Literal keyword.

Lowercase

Not literal.

<`Italics>`

Variable value. Replace with a valid option or user-defined value.

[ ]

Optional. Square brackets ( [ ] ) surround optional command arguments. Do not type the square brackets.

( )

Group. Parentheses ( ( ) ) identify a group to choose from. Do not type the parentheses.

|

Or. A vertical bar ( | ) separates alternative elements. Type any one of the elements. Do not type the vertical bar.

...

Repeatable. An ellipsis ( ... ) indicates that you can repeat the syntax element as often as required.

'<Literal string>'

Single quotation ( ' ) marks must surround literal strings in CQL statements. Use single quotation marks to preserve upper case.

{ <key>:<value> }

Map collection. Braces ( { } ) enclose map collections or key value pairs. A colon separates the key and the value.

<<datatype1>,<datatype2>>

Set, list, map, or tuple. Angle brackets ( < > ) enclose data types in a set, list, map, or tuple. Separate the data types with a comma.

cql_statement;

End CQL statement. A semicolon ( ; ) terminates all CQL statements.

[ -- ]

Separate the command line options from the command arguments with two hyphens ( -- ). This syntax is useful when arguments might be mistaken for command line options.

' <<schema> ... </schema> >'

Search CQL only: Single quotation marks ( ' ) surround an entire XML schema declaration.

@<xml_entity>='<xml_entity_type>'

Search CQL only: Identify the entity and literal value to overwrite the XML element in the schema and solrconfig files.

Definition

include:ROOT:partial$cql/CqlWHlistShortLongFormNodetoolShort.adoc[]

include:ROOT:partial$cql/CqlWHconnOptsPara.adoc[]

include:ROOT:partial$cql/CqlWHnodetoolConnectOpts.adoc[]

include:ROOT:partial$cql/CqlWHcmdOptsPara.adoc[]

include:ROOT:partial$cql/CqlWHC_cores.adoc[]

include:ROOT:partial$cql/CqlWHF_format.adoc[]

Examples

Get usage statistics for thread pools

nodetool tpstats

Results:

Pool Name                                     Active      Pending (w/Backpressure)   Delayed      Completed   Blocked  All time blocked
CompactionExecutor                                 0                       0 (N/A)       N/A         427073         0                 0
GossipStage                                        0                       0 (N/A)       N/A        1565077         0                 0
InternalResponseStage                              0                       0 (N/A)       N/A              2         0                 0
MemtableFlushWriter                                0                       0 (N/A)       N/A            183         0                 0
MemtablePostFlush                                  0                       0 (N/A)       N/A            700         0                 0
MemtableReclaimMemory                              0                       0 (N/A)       N/A            183         0                 0
MigrationStage                                     0                       0 (N/A)       N/A              2         0                 0
PendingRangeCalculator                             0                       0 (N/A)       N/A              2         0                 0
PerDiskMemtableFlushWriter_0                       0                       0 (N/A)       N/A            181         0                 0
ReadRepairStage                                    0                       0 (N/A)       N/A           1779         0                 0
TPC/all/BATCH_REMOVE                               0                       0 (N/A)       N/A          24989       N/A                 0
TPC/all/EVENTLOOP_PROCESSED_TASKS                  0                     N/A (N/A)       N/A       15246816       N/A               N/A
TPC/all/EVENTLOOP_SCHEDULED_TASKS                  0                     N/A (N/A)       N/A       26137492       N/A               N/A
TPC/all/EVENTLOOP_SELECTOR_EVENTS                  0                     N/A (N/A)       N/A        6134785       N/A               N/A
TPC/all/EVENTLOOP_SELECT_CALLS                     1                     N/A (N/A)       N/A        8991407       N/A               N/A
TPC/all/EVENTLOOP_SELECT_NOW_CALLS                 0                     N/A (N/A)       N/A        9002688       N/A               N/A
TPC/all/EVENTLOOP_SPIN                             0                     N/A (N/A)       N/A       80922912       N/A               N/A
TPC/all/EXECUTE_STATEMENT                          0                     N/A (N/A)       N/A             56       N/A               N/A
TPC/all/NODESYNC_VALIDATION                        0                     N/A (N/A)       N/A           5194       N/A               N/A
TPC/all/READ_DISK_ASYNC                            0                     N/A (N/A)       N/A            129       N/A               N/A
TPC/all/READ_INTERNAL                              0                     N/A (N/A)       N/A        1010946       N/A               N/A
TPC/all/READ_RANGE_INTERNAL                        0                     N/A (N/A)       N/A           6984       N/A               N/A
TPC/all/READ_RANGE_LOCAL                           0                       0 (N/A)       N/A              1       N/A                 0
TPC/all/READ_RANGE_RESPONSE                        0                     N/A (N/A)       N/A            210       N/A               N/A
TPC/all/READ_SPECULATE                             0                     N/A (N/A)       N/A              6       N/A               N/A
TPC/all/READ_RESPONSE                              0                     N/A (N/A)       N/A        1062638       N/A               N/A
TPC/all/TIMED_TIMEOUT                              0                     N/A (N/A)       N/A        5218530       N/A               N/A
TPC/all/UNKNOWN                                    0                     N/A (N/A)       N/A              1       N/A               N/A
TPC/all/WRITE_INTERNAL                             0                     N/A (N/A)       N/A          23958       N/A               N/A
TPC/all/WRITE_LOCAL                                0                       0 (N/A)       N/A         262963       N/A                 0
TPC/all/WRITE_REMOTE                               0                       0 (N/A)       N/A          25052       N/A                 0
TPC/all/WRITE_MEMTABLE                             0                     N/A (N/A)       N/A          87315       N/A               N/A
TPC/all/WRITE_RESPONSE                             0                     N/A (N/A)       N/A         536269       N/A               N/A
ViewBuildExecutor                                  0                       0 (N/A)       N/A             32         0                 0

Message type            Dropped                  Latency waiting in queue (micros)
                                              50%               95%               99%               Max
RANGE_SLICE                   0           3670.02           3670.02           3670.02           4194.30
SNAPSHOT                      0               N/A               N/A               N/A               N/A
HINT                          0               N/A               N/A               N/A               N/A
COUNTER_MUTATION              0               N/A               N/A               N/A               N/A
LWT                           0               N/A               N/A               N/A               N/A
BATCH_STORE                   0               N/A               N/A               N/A               N/A
VIEW_MUTATION                 0              0.00           2621.44           8388.61          25165.82
READ                          0              0.00              0.00              0.00              0.00
OTHER                         0           1835.01           3670.02           3670.02           4194.30
REPAIR                        0               N/A               N/A               N/A               N/A
SCHEMA                        0              0.00          50331.65          50331.65          58720.26
MUTATION                      0              0.00           5242.88           5242.88           6291.46
NODESYNC                      0           1835.01           7340.03           7340.03           8388.61
READ_REPAIR                   0               N/A               N/A               N/A               N/A
TRUNCATE                      0               N/A               N/A               N/A               N/A

Get usage statistics for thread pools with information about each core

nodetool tpstats -C

Results:

Pool Name                                     Active      Pending (w/Backpressure)   Delayed      Completed   Blocked  All time blocked
CompactionExecutor                                 0                       0 (N/A)       N/A         427751         0                 0
GossipStage                                        0                       0 (N/A)       N/A        1567102         0                 0
InternalResponseStage                              0                       0 (N/A)       N/A              2         0                 0
MemtableFlushWriter                                0                       0 (N/A)       N/A            189         0                 0
MemtablePostFlush                                  0                       0 (N/A)       N/A            712         0                 0
MemtableReclaimMemory                              0                       0 (N/A)       N/A            189         0                 0
MigrationStage                                     0                       0 (N/A)       N/A              2         0                 0
PendingRangeCalculator                             0                       0 (N/A)       N/A              2         0                 0
PerDiskMemtableFlushWriter_0                       0                       0 (N/A)       N/A            187         0                 0
ReadRepairStage                                    0                       0 (N/A)       N/A           1782         0                 0
TPC/0                                              0                         0 (0)         0        3044649       N/A                 0
TPC/0/BATCH_REMOVE                                 0                       0 (N/A)       N/A          24989       N/A                 0
TPC/0/EVENTLOOP_PROCESSED_TASKS                    0                     N/A (N/A)       N/A       15260947       N/A               N/A
TPC/0/EVENTLOOP_SCHEDULED_TASKS                    0                     N/A (N/A)       N/A       26158321       N/A               N/A
TPC/0/EVENTLOOP_SELECTOR_EVENTS                    0                     N/A (N/A)       N/A        6141833       N/A               N/A
TPC/0/EVENTLOOP_SELECT_CALLS                       1                     N/A (N/A)       N/A        8998452       N/A               N/A
TPC/0/EVENTLOOP_SELECT_NOW_CALLS                   0                     N/A (N/A)       N/A        9009734       N/A               N/A
TPC/0/EVENTLOOP_SPIN                               0                     N/A (N/A)       N/A       80986317       N/A               N/A
TPC/0/READ_DISK_ASYNC                              0                     N/A (N/A)       N/A            131       N/A               N/A
TPC/0/READ_INTERNAL                                0                     N/A (N/A)       N/A        1012395       N/A               N/A
TPC/0/READ_RANGE_INTERNAL                          0                     N/A (N/A)       N/A           6996       N/A               N/A
TPC/0/READ_RANGE_LOCAL                             0                       0 (N/A)       N/A              1       N/A                 0
TPC/0/READ_RANGE_RESPONSE  ccccccccccc             0                     N/A (N/A)       N/A            210       N/A               N/A
TPC/0/READ_SPECULATE                               0                     N/A (N/A)       N/A              6       N/A               N/A
TPC/0/READ_RESPONSE    ccccccccccc                 0                     N/A (N/A)       N/A        1064206       N/A               N/A
TPC/0/TIMED_TIMEOUT                                0                     N/A (N/A)       N/A        5225278       N/A               N/A
TPC/0/UNKNOWN                                      0                     N/A (N/A)       N/A              1       N/A               N/A
TPC/0/WRITE_INTERNAL                               0                     N/A (N/A)       N/A          23998       N/A               N/A
TPC/0/WRITE_LOCAL                                  0                       0 (N/A)       N/A         262963       N/A                 0
TPC/0/WRITE_REMOTE                                 0                       0 (N/A)       N/A          25052       N/A                 0
TPC/0/WRITE_MEMTABLE                               0                     N/A (N/A)       N/A          87400       N/A               N/A
TPC/0/WRITE_RESPONSE                               0                     N/A (N/A)       N/A         536301       N/A               N/A
TPC/all/BATCH_REMOVE                               0                       0 (N/A)       N/A          24989       N/A                 0
TPC/all/EVENTLOOP_PROCESSED_TASKS                  0                     N/A (N/A)       N/A       15260957       N/A               N/A
TPC/all/EVENTLOOP_SCHEDULED_TASKS                  0                     N/A (N/A)       N/A       26158333       N/A               N/A
TPC/all/EVENTLOOP_SELECTOR_EVENTS                  0                     N/A (N/A)       N/A        6141828       N/A               N/A
TPC/all/EVENTLOOP_SELECT_CALLS                     1                     N/A (N/A)       N/A        8998452       N/A               N/A
TPC/all/EVENTLOOP_SELECT_NOW_CALLS                 0                     N/A (N/A)       N/A        9009735       N/A               N/A
TPC/all/EVENTLOOP_SPIN                             0                     N/A (N/A)       N/A       80986335       N/A               N/A
TPC/all/EXECUTE_STATEMENT                          0                     N/A (N/A)       N/A             56       N/A               N/A
TPC/all/NODESYNC_VALIDATION                        0                     N/A (N/A)       N/A           5202       N/A               N/A
TPC/all/READ_DISK_ASYNC                            0                     N/A (N/A)       N/A            131       N/A               N/A
TPC/all/READ_INTERNAL                              0                     N/A (N/A)       N/A        1012395       N/A               N/A
TPC/all/READ_RANGE_INTERNAL                        0                     N/A (N/A)       N/A           6996       N/A               N/A
TPC/all/READ_RANGE_LOCAL                           0                       0 (N/A)       N/A              1       N/A                 0
TPC/all/READ_RANGE_RESPONSE                        0                     N/A (N/A)       N/A            210       N/A               N/A
TPC/all/READ_SPECULATE                             0                     N/A (N/A)       N/A              6       N/A               N/A
TPC/all/READ_RESPONSE                              0                     N/A (N/A)       N/A        1064206       N/A               N/A
TPC/all/TIMED_TIMEOUT                              0                     N/A (N/A)       N/A        5225277       N/A               N/A
TPC/all/UNKNOWN                                    0                     N/A (N/A)       N/A              1       N/A               N/A
TPC/all/WRITE_INTERNAL                             0                     N/A (N/A)       N/A          23998       N/A               N/A
TPC/all/WRITE_LOCAL                                0                       0 (N/A)       N/A         262963       N/A                 0
TPC/all/WRITE_REMOTE                               0                       0 (N/A)       N/A          25052       N/A                 0
TPC/all/WRITE_MEMTABLE                             0                     N/A (N/A)       N/A          87400       N/A               N/A
TPC/all/WRITE_RESPONSE                             0                     N/A (N/A)       N/A         536301       N/A               N/A
TPC/other                                          0                         0 (0)         0           5258       N/A                 0
TPC/other/EXECUTE_STATEMENT                        0                     N/A (N/A)       N/A             56       N/A               N/A
TPC/other/NODESYNC_VALIDATION                      0                     N/A (N/A)       N/A           5202       N/A               N/A
ViewBuildExecutor                                  0                       0 (N/A)       N/A             32         0                 0

Meters                                   One Minute Rate    Five Minute Rate     Fifteen Minute Rate      Mean Rate          Count    Connections
TPC/0/CLIENT_REQUEST                                0.00                0.00                    0.00           0.00              0              0
TPC/0/INTERNODE_MESSAGE                             0.00                0.00                    0.00           0.00              0              0
TPC/1/CLIENT_REQUEST                                0.00                0.00                    0.00           0.00              0              0
TPC/1/INTERNODE_MESSAGE                             0.00                0.00                    0.00           0.00              0              0
TPC/10/CLIENT_REQUEST                               0.00                0.00                    0.00           0.00              0              0
TPC/10/INTERNODE_MESSAGE                            0.00                0.00                    0.00           0.00              0              0
TPC/2/CLIENT_REQUEST                                0.00                0.00                    0.00           0.00              0              0
TPC/2/INTERNODE_MESSAGE                             0.00                0.00                    0.00           0.00              0              0
TPC/3/CLIENT_REQUEST                                0.00                0.00                    0.00           0.00              0              0
TPC/3/INTERNODE_MESSAGE                             0.00                0.00                    0.00           0.00              0              0
TPC/4/CLIENT_REQUEST                                0.00                0.00                    0.00           0.00              0              0
TPC/4/INTERNODE_MESSAGE                             0.00                0.00                    0.00           0.00              0              0
TPC/5/CLIENT_REQUEST                                0.00                0.00                    0.00           0.00              0              0
TPC/5/INTERNODE_MESSAGE                             0.00                0.00                    0.00           0.00              0              0

Message type            Dropped                  Latency waiting in queue (micros)
                                              50%               95%               99%               Max
RANGE_SLICE                   0           3670.02           3670.02           3670.02           4194.30
SNAPSHOT                      0               N/A               N/A               N/A               N/A
HINT                          0               N/A               N/A               N/A               N/A
COUNTER_MUTATION              0               N/A               N/A               N/A               N/A
LWT                           0               N/A               N/A               N/A               N/A
BATCH_STORE                   0               N/A               N/A               N/A               N/A
VIEW_MUTATION                 0              0.00           2621.44           8388.61          25165.82
READ                          0              0.00              0.00              0.00              0.00
OTHER                         0           1835.01           3670.02           3670.02           4194.30
REPAIR                        0               N/A               N/A               N/A               N/A
SCHEMA                        0              0.00          50331.65          50331.65          58720.26
MUTATION                      0              0.00           3670.02           3670.02           4194.30
NODESYNC                      0              0.00           4194.30           4194.30           5242.88
READ_REPAIR                   0               N/A               N/A               N/A               N/A
TRUNCATE                      0               N/A               N/A               N/A               N/A

Was this helpful?

Give Feedback

How can we improve the documentation?

© 2024 DataStax | Privacy policy | Terms of use

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: +1 (650) 389-6000, info@datastax.com