Untitled | Hyper-Converged Database

nodetool tpstats

Returns usage statistics of thread pools.

The database separates different tasks into stages connected by a messaging service. Each stage has a queue and a thread pool. Some stages skip the messaging service and queue tasks immediately on a different stage when it exists on the same node. If the next stage is too busy, the database can back up a queue and lead to performance bottlenecks, as described in Monitor an HCD cluster.

Reports are updated after SSTables change through compaction or flushing.

Report columns

The nodetool tpstats command report includes the following columns:

Active: The number of Active threads.

Pending: The number of Pending requests waiting to be executed by this thread pool.

Completed: The number of tasks Completed by this thread pool.

Blocked: The number of requests that are currently Blocked because the thread pool for the next step in the service is full.

All-Time Blocked: The total number of All-Time Blocked requests, which are all requests blocked in this thread pool up to now.

Report rows

The follow list describes the task or property associated with the task reported in the nodetool tpstats output.

General metrics

The following report aggregated statistics for tasks on the local node:

BackgroundIoStage

Completes background tasks like submitting hints and deserializing the row cache.

CompactionExecutor

Running compaction.

GossipStage

Distributing node information via Gossip. Out of sync schemas can cause issues. You may have to sync using nodetool resetlocalschema.

HintsDispatcher

Dispatches a single hints file to a specified node in a batched manner.

InternalResponseStage

Responding to non-client initiated messages, including bootstrapping and schema checking.

MemtableFlushWriter

Writing memtable contents to disk. May back up if the queue is overruns the disk I/O, or because of sorting processes.

nodetool tpstats no longer reports blocked threads in the MemtableFlushWriter pool. Check the Pending Flushes metric reported by nodetool tblestats.

MemtablePostFlush

Cleaning up after flushing the memtable (discarding commit logs and secondary indexes as needed).

MemtableReclaimMemory

Making unused memory available.

PendingRangeCalculator

Calculating pending ranges per bootstraps and departed nodes. Reporting by this tool is not useful — see Developer notes.

PerDiskMemtableFlushWriter_N

Activity for the memtable flush writer of each disk.

ReadRepairStage

Performing read repairs. Usually fast, if there is good connectivity between replicas.

Droppable messages

The database generates the messages listed below, but discards them after a timeout. The nodetool tpstats command reports the number of messages of each type that have been dropped. You can view the messages themselves using a JMX client.

Message Type Stage Notes

Message Type	Stage	Notes
BINARY	n/a	Deprecated.
_TRACE	n/a (special)	Used for recording traces (nodetool settraceprobability). Has a special executor (1 thread, 1000 queue depth) that throws away messages on insertion instead of within the execute.
MUTATION	MutationStage	If a write message is processed after its timeout (`write_request_timeout_in_ms`), it either sent a failure to the client or it met its requested consistency level and will relay on hinted handoff and read repairs to do the mutation if it succeeded.
COUNTER_MUTATION	MutationStage	If a write message is processed after its timeout (`write_request_timeout_in_ms`), it either sent a failure to the client or it met its requested consistency level and will relay on hinted handoff and read repairs to do the mutation.
READ_REPAIR	MutationStage	Times out after `write_request_timeout_in_ms`.
READ	ReadStage	Times out after `read_request_timeout_in_ms`. No point in servicing reads after that point because it would have returned error to client.
RANGE_SLICE	ReadStage	Times out after `range_request_timeout_in_ms`.
PAGED_RANGE	ReadStage	Times out after `request_timeout_in_ms`.
REQUEST_RESPONSE	RequestResponseStage	Times out after `request_timeout_in_ms`. Response was completed and sent back but not before the timeout.

BINARY

n/a

Deprecated.

_TRACE

n/a (special)

Used for recording traces (nodetool settraceprobability). Has a special executor (1 thread, 1000 queue depth) that throws away messages on insertion instead of within the execute.

MUTATION

MutationStage

If a write message is processed after its timeout (write_request_timeout_in_ms), it either sent a failure to the client or it met its requested consistency level and will relay on hinted handoff and read repairs to do the mutation if it succeeded.

COUNTER_MUTATION

MutationStage

READ_REPAIR

MutationStage

Times out after write_request_timeout_in_ms.

READ

ReadStage

Times out after read_request_timeout_in_ms. No point in servicing reads after that point because it would have returned error to client.

RANGE_SLICE

ReadStage

Times out after range_request_timeout_in_ms.

PAGED_RANGE

ReadStage

Times out after request_timeout_in_ms.

REQUEST_RESPONSE

RequestResponseStage

Times out after request_timeout_in_ms. Response was completed and sent back but not before the timeout.

Synopsis

nodetool [<connection_options>] tpstats
[-C] [-F json | yaml]

Definition

include:ROOT:partial$cql/CqlWHlistShortLongFormNodetoolShort.adoc[]

include:ROOT:partial$cql/CqlWHconnOptsPara.adoc[]

include:ROOT:partial$cql/CqlWHnodetoolConnectOpts.adoc[]

include:ROOT:partial$cql/CqlWHcmdOptsPara.adoc[]

include:ROOT:partial$cql/CqlWHC_cores.adoc[]

include:ROOT:partial$cql/CqlWHF_format.adoc[]

Examples

Get usage statistics for thread pools

nodetool tpstats

Results:

Pool Name                                     Active      Pending (w/Backpressure)   Delayed      Completed   Blocked  All time blocked
CompactionExecutor                                 0                       0 (N/A)       N/A         427073         0                 0
GossipStage                                        0                       0 (N/A)       N/A        1565077         0                 0
InternalResponseStage                              0                       0 (N/A)       N/A              2         0                 0
MemtableFlushWriter                                0                       0 (N/A)       N/A            183         0                 0
MemtablePostFlush                                  0                       0 (N/A)       N/A            700         0                 0
MemtableReclaimMemory                              0                       0 (N/A)       N/A            183         0                 0
MigrationStage                                     0                       0 (N/A)       N/A              2         0                 0
PendingRangeCalculator                             0                       0 (N/A)       N/A              2         0                 0
PerDiskMemtableFlushWriter_0                       0                       0 (N/A)       N/A            181         0                 0
ReadRepairStage                                    0                       0 (N/A)       N/A           1779         0                 0
TPC/all/BATCH_REMOVE                               0                       0 (N/A)       N/A          24989       N/A                 0
TPC/all/EVENTLOOP_PROCESSED_TASKS                  0                     N/A (N/A)       N/A       15246816       N/A               N/A
TPC/all/EVENTLOOP_SCHEDULED_TASKS                  0                     N/A (N/A)       N/A       26137492       N/A               N/A
TPC/all/EVENTLOOP_SELECTOR_EVENTS                  0                     N/A (N/A)       N/A        6134785       N/A               N/A
TPC/all/EVENTLOOP_SELECT_CALLS                     1                     N/A (N/A)       N/A        8991407       N/A               N/A
TPC/all/EVENTLOOP_SELECT_NOW_CALLS                 0                     N/A (N/A)       N/A        9002688       N/A               N/A
TPC/all/EVENTLOOP_SPIN                             0                     N/A (N/A)       N/A       80922912       N/A               N/A
TPC/all/EXECUTE_STATEMENT                          0                     N/A (N/A)       N/A             56       N/A               N/A
TPC/all/NODESYNC_VALIDATION                        0                     N/A (N/A)       N/A           5194       N/A               N/A
TPC/all/READ_DISK_ASYNC                            0                     N/A (N/A)       N/A            129       N/A               N/A
TPC/all/READ_INTERNAL                              0                     N/A (N/A)       N/A        1010946       N/A               N/A
TPC/all/READ_RANGE_INTERNAL                        0                     N/A (N/A)       N/A           6984       N/A               N/A
TPC/all/READ_RANGE_LOCAL                           0                       0 (N/A)       N/A              1       N/A                 0
TPC/all/READ_RANGE_RESPONSE                        0                     N/A (N/A)       N/A            210       N/A               N/A
TPC/all/READ_SPECULATE                             0                     N/A (N/A)       N/A              6       N/A               N/A
TPC/all/READ_RESPONSE                              0                     N/A (N/A)       N/A        1062638       N/A               N/A
TPC/all/TIMED_TIMEOUT                              0                     N/A (N/A)       N/A        5218530       N/A               N/A
TPC/all/UNKNOWN                                    0                     N/A (N/A)       N/A              1       N/A               N/A
TPC/all/WRITE_INTERNAL                             0                     N/A (N/A)       N/A          23958       N/A               N/A
TPC/all/WRITE_LOCAL                                0                       0 (N/A)       N/A         262963       N/A                 0
TPC/all/WRITE_REMOTE                               0                       0 (N/A)       N/A          25052       N/A                 0
TPC/all/WRITE_MEMTABLE                             0                     N/A (N/A)       N/A          87315       N/A               N/A
TPC/all/WRITE_RESPONSE                             0                     N/A (N/A)       N/A         536269       N/A               N/A
ViewBuildExecutor                                  0                       0 (N/A)       N/A             32         0                 0

Message type            Dropped                  Latency waiting in queue (micros)
                                              50%               95%               99%               Max
RANGE_SLICE                   0           3670.02           3670.02           3670.02           4194.30
SNAPSHOT                      0               N/A               N/A               N/A               N/A
HINT                          0               N/A               N/A               N/A               N/A
COUNTER_MUTATION              0               N/A               N/A               N/A               N/A
LWT                           0               N/A               N/A               N/A               N/A
BATCH_STORE                   0               N/A               N/A               N/A               N/A
VIEW_MUTATION                 0              0.00           2621.44           8388.61          25165.82
READ                          0              0.00              0.00              0.00              0.00
OTHER                         0           1835.01           3670.02           3670.02           4194.30
REPAIR                        0               N/A               N/A               N/A               N/A
SCHEMA                        0              0.00          50331.65          50331.65          58720.26
MUTATION                      0              0.00           5242.88           5242.88           6291.46
NODESYNC                      0           1835.01           7340.03           7340.03           8388.61
READ_REPAIR                   0               N/A               N/A               N/A               N/A
TRUNCATE                      0               N/A               N/A               N/A               N/A

Get usage statistics for thread pools with information about each core

nodetool tpstats -C

Results:

Pool Name                                     Active      Pending (w/Backpressure)   Delayed      Completed   Blocked  All time blocked
CompactionExecutor                                 0                       0 (N/A)       N/A         427751         0                 0
GossipStage                                        0                       0 (N/A)       N/A        1567102         0                 0
InternalResponseStage                              0                       0 (N/A)       N/A              2         0                 0
MemtableFlushWriter                                0                       0 (N/A)       N/A            189         0                 0
MemtablePostFlush                                  0                       0 (N/A)       N/A            712         0                 0
MemtableReclaimMemory                              0                       0 (N/A)       N/A            189         0                 0
MigrationStage                                     0                       0 (N/A)       N/A              2         0                 0
PendingRangeCalculator                             0                       0 (N/A)       N/A              2         0                 0
PerDiskMemtableFlushWriter_0                       0                       0 (N/A)       N/A            187         0                 0
ReadRepairStage                                    0                       0 (N/A)       N/A           1782         0                 0
TPC/0                                              0                         0 (0)         0        3044649       N/A                 0
TPC/0/BATCH_REMOVE                                 0                       0 (N/A)       N/A          24989       N/A                 0
TPC/0/EVENTLOOP_PROCESSED_TASKS                    0                     N/A (N/A)       N/A       15260947       N/A               N/A
TPC/0/EVENTLOOP_SCHEDULED_TASKS                    0                     N/A (N/A)       N/A       26158321       N/A               N/A
TPC/0/EVENTLOOP_SELECTOR_EVENTS                    0                     N/A (N/A)       N/A        6141833       N/A               N/A
TPC/0/EVENTLOOP_SELECT_CALLS                       1                     N/A (N/A)       N/A        8998452       N/A               N/A
TPC/0/EVENTLOOP_SELECT_NOW_CALLS                   0                     N/A (N/A)       N/A        9009734       N/A               N/A
TPC/0/EVENTLOOP_SPIN                               0                     N/A (N/A)       N/A       80986317       N/A               N/A
TPC/0/READ_DISK_ASYNC                              0                     N/A (N/A)       N/A            131       N/A               N/A
TPC/0/READ_INTERNAL                                0                     N/A (N/A)       N/A        1012395       N/A               N/A
TPC/0/READ_RANGE_INTERNAL                          0                     N/A (N/A)       N/A           6996       N/A               N/A
TPC/0/READ_RANGE_LOCAL                             0                       0 (N/A)       N/A              1       N/A                 0
TPC/0/READ_RANGE_RESPONSE  ccccccccccc             0                     N/A (N/A)       N/A            210       N/A               N/A
TPC/0/READ_SPECULATE                               0                     N/A (N/A)       N/A              6       N/A               N/A
TPC/0/READ_RESPONSE    ccccccccccc                 0                     N/A (N/A)       N/A        1064206       N/A               N/A
TPC/0/TIMED_TIMEOUT                                0                     N/A (N/A)       N/A        5225278       N/A               N/A
TPC/0/UNKNOWN                                      0                     N/A (N/A)       N/A              1       N/A               N/A
TPC/0/WRITE_INTERNAL                               0                     N/A (N/A)       N/A          23998       N/A               N/A
TPC/0/WRITE_LOCAL                                  0                       0 (N/A)       N/A         262963       N/A                 0
TPC/0/WRITE_REMOTE                                 0                       0 (N/A)       N/A          25052       N/A                 0
TPC/0/WRITE_MEMTABLE                               0                     N/A (N/A)       N/A          87400       N/A               N/A
TPC/0/WRITE_RESPONSE                               0                     N/A (N/A)       N/A         536301       N/A               N/A
TPC/all/BATCH_REMOVE                               0                       0 (N/A)       N/A          24989       N/A                 0
TPC/all/EVENTLOOP_PROCESSED_TASKS                  0                     N/A (N/A)       N/A       15260957       N/A               N/A
TPC/all/EVENTLOOP_SCHEDULED_TASKS                  0                     N/A (N/A)       N/A       26158333       N/A               N/A
TPC/all/EVENTLOOP_SELECTOR_EVENTS                  0                     N/A (N/A)       N/A        6141828       N/A               N/A
TPC/all/EVENTLOOP_SELECT_CALLS                     1                     N/A (N/A)       N/A        8998452       N/A               N/A
TPC/all/EVENTLOOP_SELECT_NOW_CALLS                 0                     N/A (N/A)       N/A        9009735       N/A               N/A
TPC/all/EVENTLOOP_SPIN                             0                     N/A (N/A)       N/A       80986335       N/A               N/A
TPC/all/EXECUTE_STATEMENT                          0                     N/A (N/A)       N/A             56       N/A               N/A
TPC/all/NODESYNC_VALIDATION                        0                     N/A (N/A)       N/A           5202       N/A               N/A
TPC/all/READ_DISK_ASYNC                            0                     N/A (N/A)       N/A            131       N/A               N/A
TPC/all/READ_INTERNAL                              0                     N/A (N/A)       N/A        1012395       N/A               N/A
TPC/all/READ_RANGE_INTERNAL                        0                     N/A (N/A)       N/A           6996       N/A               N/A
TPC/all/READ_RANGE_LOCAL                           0                       0 (N/A)       N/A              1       N/A                 0
TPC/all/READ_RANGE_RESPONSE                        0                     N/A (N/A)       N/A            210       N/A               N/A
TPC/all/READ_SPECULATE                             0                     N/A (N/A)       N/A              6       N/A               N/A
TPC/all/READ_RESPONSE                              0                     N/A (N/A)       N/A        1064206       N/A               N/A
TPC/all/TIMED_TIMEOUT                              0                     N/A (N/A)       N/A        5225277       N/A               N/A
TPC/all/UNKNOWN                                    0                     N/A (N/A)       N/A              1       N/A               N/A
TPC/all/WRITE_INTERNAL                             0                     N/A (N/A)       N/A          23998       N/A               N/A
TPC/all/WRITE_LOCAL                                0                       0 (N/A)       N/A         262963       N/A                 0
TPC/all/WRITE_REMOTE                               0                       0 (N/A)       N/A          25052       N/A                 0
TPC/all/WRITE_MEMTABLE                             0                     N/A (N/A)       N/A          87400       N/A               N/A
TPC/all/WRITE_RESPONSE                             0                     N/A (N/A)       N/A         536301       N/A               N/A
TPC/other                                          0                         0 (0)         0           5258       N/A                 0
TPC/other/EXECUTE_STATEMENT                        0                     N/A (N/A)       N/A             56       N/A               N/A
TPC/other/NODESYNC_VALIDATION                      0                     N/A (N/A)       N/A           5202       N/A               N/A
ViewBuildExecutor                                  0                       0 (N/A)       N/A             32         0                 0

Meters                                   One Minute Rate    Five Minute Rate     Fifteen Minute Rate      Mean Rate          Count    Connections
TPC/0/CLIENT_REQUEST                                0.00                0.00                    0.00           0.00              0              0
TPC/0/INTERNODE_MESSAGE                             0.00                0.00                    0.00           0.00              0              0
TPC/1/CLIENT_REQUEST                                0.00                0.00                    0.00           0.00              0              0
TPC/1/INTERNODE_MESSAGE                             0.00                0.00                    0.00           0.00              0              0
TPC/10/CLIENT_REQUEST                               0.00                0.00                    0.00           0.00              0              0
TPC/10/INTERNODE_MESSAGE                            0.00                0.00                    0.00           0.00              0              0
TPC/2/CLIENT_REQUEST                                0.00                0.00                    0.00           0.00              0              0
TPC/2/INTERNODE_MESSAGE                             0.00                0.00                    0.00           0.00              0              0
TPC/3/CLIENT_REQUEST                                0.00                0.00                    0.00           0.00              0              0
TPC/3/INTERNODE_MESSAGE                             0.00                0.00                    0.00           0.00              0              0
TPC/4/CLIENT_REQUEST                                0.00                0.00                    0.00           0.00              0              0
TPC/4/INTERNODE_MESSAGE                             0.00                0.00                    0.00           0.00              0              0
TPC/5/CLIENT_REQUEST                                0.00                0.00                    0.00           0.00              0              0
TPC/5/INTERNODE_MESSAGE                             0.00                0.00                    0.00           0.00              0              0

Message type            Dropped                  Latency waiting in queue (micros)
                                              50%               95%               99%               Max
RANGE_SLICE                   0           3670.02           3670.02           3670.02           4194.30
SNAPSHOT                      0               N/A               N/A               N/A               N/A
HINT                          0               N/A               N/A               N/A               N/A
COUNTER_MUTATION              0               N/A               N/A               N/A               N/A
LWT                           0               N/A               N/A               N/A               N/A
BATCH_STORE                   0               N/A               N/A               N/A               N/A
VIEW_MUTATION                 0              0.00           2621.44           8388.61          25165.82
READ                          0              0.00              0.00              0.00              0.00
OTHER                         0           1835.01           3670.02           3670.02           4194.30
REPAIR                        0               N/A               N/A               N/A               N/A
SCHEMA                        0              0.00          50331.65          50331.65          58720.26
MUTATION                      0              0.00           3670.02           3670.02           4194.30
NODESYNC                      0              0.00           4194.30           4194.30           5242.88
READ_REPAIR                   0               N/A               N/A               N/A               N/A
TRUNCATE                      0               N/A               N/A               N/A               N/A

nodetool tpstats

Report columns

Report rows

General metrics

Droppable messages

Synopsis

Definition

Examples

Get usage statistics for thread pools

Get usage statistics for thread pools with information about each core

Was this helpful?

Give Feedback