nodetool tpstats
Prints usage statistics of thread pools.
cassandra.yaml
The location of the cassandra.yaml file depends on the type of installation:Package installations | /etc/dse/cassandra/cassandra.yaml |
Tarball installations | installation_location/resources/cassandra/conf/cassandra.yaml |
Prints usage statistics of thread pools. The DataStax Enterprise (DSE) database is based on a staged event-driven architecture (SEDA).
The database separates different tasks into stages connected by a messaging service. Each stage has a queue and a thread pool. Some stages skip the messaging service and queue tasks immediately on a different stage when it exists on the same node. The database can back up a queue if the next stage is too busy and lead to performance bottlenecks, as described in Monitoring a DataStax Enterprise cluster.
Reports are updated after SSTables change through compaction or flushing.
Report columns
nodetool tpstats
command report includes the following columns:- Active
- The number of
Active
threads. - Pending
- The number of
Pending
requests waiting to be executed by this thread pool. - Completed
- The number of tasks
Completed
by this thread pool. - Blocked
- The number of requests that are currently
Blocked
because the thread pool for the next step in the service is full. - All-Time Blocked
- The total number of
All-Time Blocked
requests, which are all requests blocked in this thread pool up to now.
Report rows
The follow list describes the task or property associated with the task reported in the
nodetool tpstats
output.
General metrics
- BackgroundIoStage
- Completes background tasks like submitting hints and deserializing the row cache.
- CompactionExecutor
- Running compaction.
- GossipStage
- Distributing node information via Gossip. Out of sync schemas can cause
issues. You may have to sync using
nodetool resetlocalschema.
- HintsDispatcher
- Dispatches a single hints file to a specified node in a batched manner.
- InternalResponseStage
- Responding to non-client initiated messages, including bootstrapping and schema checking.
- MemtableFlushWriter
- Writing memtable contents to disk. May back up if the queue is overruns the
disk I/O, or because of sorting processes.Warning:
nodetool tpstats
no longer reports blocked threads in the MemtableFlushWriter pool. Check the Pending Flushes metric reported bynodetool tblestats
. - MemtablePostFlush
- Cleaning up after flushing the memtable (discarding commit logs and secondary indexes as needed).
- MemtableReclaimMemory
- Making unused memory available.
- PendingRangeCalculator
- Calculating pending ranges per bootstraps and departed nodes. Reporting by this tool is not useful — see Developer notes.
- PerDiskMemtableFlushWriter_N
- Activity for the memtable flush writer of each disk.
- ReadRepairStage
- Performing read repairs. Usually fast, if there is good connectivity between replicas.
Thread per core (TPC) task metrics
tpc_concurrent_requests_limit
in the
cassandra.yaml (by default, 128). Thread per
core messages are prepended with TPC/type
,
where: - TPC/N are metrics for the core
number (when
--cores
is specified). - TPC/other are metrics for tasks executed that are not on TPC threads.
- TPC/all are the aggregate task metrics for all cores.
- UNKNOWN
- Unknown task.
- FRAME_DECODE
- Asynchronous frame decoding.
- READ_LOCAL
- Single-partition read request from a local node generated directly from clients.
- READ_REMOTE
- Single-partition read request from a remote replica.
- READ_TIMEOUT
- Signals read timeout errors.
- READ_DEFERRED
- Single-partition read request that will be first scheduled on an event loop (Pendable)
- READ_RESPONSE
- Single-partition read response.
- READ_RANGE_LOCAL
- Partition range read request from a local node generated directly from clients.
- READ_RANGE_REMOTE
- Partition range read request from a remote replica.
- READ_RANGE_NODESYNC
- Partition range read originating from NodeSync.
- READ_RANGE_INTERNAL
- Range reads to internal tables.
- READ_RANGE_RESPONSE
- Partition range read response.
- READ_FROM_ITERATOR
- Switching thread to read from an iterator.
- READ_SECONDARY_INDEX
- Switching thread to read from secondary index.
- READ_DISK_ASYNC
- Waiting for data from disk.
- WRITE_LOCAL
- Write request from a local node generated directly from clients.
- WRITE_REMOTE
- Write request from a remote replica
- WRITE_INTERNAL
- Writes to internal tables.
- WRITE_RESPONSE
- Write response
- WRITE_DEFRAGMENT
- Write issued to defragment data that required too many sstables to read (Pendable)
- WRITE_MEMTABLE
- Switching thread to write in memtable when not already on the correct thread
- WRITE_POST_COMMITLOG_SEGMENT
- Write request is waiting for the commit log segment to be allocated
- WRITE_POST_COMMITLOG_SYNC
- Write request is waiting for commit log to sync to disk
- WRITE_POST_MEMTABLE_FULL
- Write request is waiting for space in memtable
- BATCH_REPLAY
- Replaying a batch mutation
- BATCH_STORE
- Store a batchlog entry request (Pendable)
- BATCH_STORE_RESPONSE
- Store a batchlog entry response
- BATCH_REMOVE
- Remove a batchlog entry (Pendable)
- COUNTER_ACQUIRE_LOCK
- Acquiring counter lock.
- EXECUTE_STATEMENT
- Executing a statement.
- CAS
- Executing compare-and-set (LWT).
- LWT_PREPARE
- Preparation phase of light-weight transaction (Pendable).
- LWT_PROPOSE
- Proposal phase of light-weight transaction (Pendable).
- LWT_COMMIT
- Commit phase of light-weight transaction (Pendable).
- TRUNCATE
- Truncate request (Pendable).
- NODESYNC_VALIDATION
- NodeSync validation of a partition.
- AUTHENTICATION
- Authentication request.
- AUTHORIZATION
- Authorization request.
- TIMED_UNKNOWN
- Unknown timed task.
- TIMED_TIMEOUT
- Scheduled timeout task.
- EVENTLOOP_SPIN
- Number of busy spin cycles done by this TPC thread when it has no tasks to perform.
- EVENTLOOP_YIELD
- Number of Thread.yield() calls done by this TPC thread when it has no tasks to perform.
- EVENTLOOP_PARK
- Number of LockSupport.park() calls done by this TPC thread when it has no tasks to perform.
- HINT_DISPATCH
- Hint dispatch request (Pendable).
- HINT_RESPONSE
- Hint dispatch response.
- NETWORK_BACKPRESSURE
- Scheduled network backpressure.
Droppable messages
The database generates the messages listed below, but discards them after a timeout. The
nodetool tpstats
command reports the number of messages of each type
that have been dropped. You can view the messages themselves using a JMX client.
Message Type | Stage | Notes |
---|---|---|
BINARY | n/a | Deprecated |
_TRACE | n/a (special) | Used for recording traces (nodetool settraceprobability) Has a special executor (1 thread, 1000 queue depth) that throws away messages on insertion instead of within the execute |
MUTATION | MutationStage | If a write message is processed after its timeout (write_request_timeout_in_ms) it either sent a failure to the client or it met its requested consistency level and will relay on hinted handoff and read repairs to do the mutation if it succeeded. |
COUNTER_MUTATION | MutationStage | If a write message is processed after its timeout (write_request_timeout_in_ms) it either sent a failure to the client or it met its requested consistency level and will relay on hinted handoff and read repairs to do the mutation if it succeeded. |
READ_REPAIR | MutationStage | Times out after write_request_timeout_in_ms |
READ | ReadStage | Times out after read_request_timeout_in_ms. No point in servicing reads after that point since it would of returned error to client |
RANGE_SLICE | ReadStage | Times out after range_request_timeout_in_ms. |
PAGED_RANGE | ReadStage | Times out after request_timeout_in_ms. |
REQUEST_RESPONSE | RequestResponseStage | Times out after request_timeout_in_ms. Response was completed and sent back but not before the timeout |
Synopsis
nodetool [connection_options] tpstats [-C] [-F json | yaml]
Syntax conventions | Description |
---|---|
UPPERCASE | Literal keyword. |
Lowercase | Not literal. |
Italics |
Variable value. Replace with a valid option or user-defined value. |
[ ] |
Optional. Square brackets ( [ ] ) surround optional command
arguments. Do not type the square brackets. |
( ) |
Group. Parentheses ( ( ) ) identify a group to choose from. Do
not type the parentheses. |
| |
Or. A vertical bar ( | ) separates alternative elements. Type
any one of the elements. Do not type the vertical bar. |
... |
Repeatable. An ellipsis ( ... ) indicates that you can repeat
the syntax element as often as required. |
'Literal string' |
Single quotation ( ' ) marks must surround literal strings in
CQL statements. Use single quotation marks to preserve upper case. |
{ key:value } |
Map collection. Braces ( { } ) enclose map collections or key
value pairs. A colon separates the key and the value. |
<datatype1,datatype2> |
Set, list, map, or tuple. Angle brackets ( < > ) enclose
data types in a set, list, map, or tuple. Separate the data types with a comma.
|
cql_statement; |
End CQL statement. A semicolon ( ; ) terminates all CQL
statements. |
[ -- ] |
Separate the command line options from the command arguments with two hyphens (
-- ). This syntax is useful when arguments might be mistaken for
command line options. |
' <schema> ... </schema>
' |
Search CQL only: Single quotation marks ( ' ) surround an entire
XML schema declaration. |
@xml_entity='xml_entity_type' |
Search CQL only: Identify the entity and literal value to overwrite the XML element in the schema and solrconfig files. |
Definition
The short form and long form parameters are comma-separated.
Connection options
- -h, --host hostname
- The hostname or IP address of a remote node or nodes. When omitted, the default is the local machine.
- -p, --port jmx_port
- The JMX port number.
- -pw, --password jmxpassword
- The JMX password for authenticating with secure JMX. If a password is not provided, you are prompted to enter one.
- -pwf, --password-file jmx_password_filepath
- The filepath to the file that stores JMX authentication credentials.
- -u, --username jmx_username
- The user name for authenticating with secure JMX.
Command arguments
- -C, --cores
- Include data for each core. The number of cores is determined by the tpc_cores.
- -F, --format json | yaml
- The format for the output. The default is plain text. The following wait latencies (in ms) are included in the following order: 50%, 75%, 95%, 98%, 99%, Min, and Max.
Examples
Run nodetool tpstats on the host labcluster
nodetool tpstats -C
Command output is:
Pool Name Active Pending (w/Backpressure) Delayed Completed Blocked All time blocked
CompactionExecutor 0 0 (N/A) N/A 196 0 0
GossipStage 0 0 (N/A) N/A 2177 0 0
InternalResponseStage 0 0 (N/A) N/A 8 0 0
MemtableFlushWriter 0 0 (N/A) N/A 25 0 0
MemtablePostFlush 0 0 (N/A) N/A 143 0 0
MemtableReclaimMemory 0 0 (N/A) N/A 25 0 0
MigrationStage 0 0 (N/A) N/A 7 0 0
PendingRangeCalculator 0 0 (N/A) N/A 4 0 0
PerDiskMemtableFlushWriter_0 0 0 (N/A) N/A 25 0 0
ReadRepairStage 0 0 (N/A) N/A 2 0 0
TPC/0 0 0 (0) 0 3470 N/A 0
TPC/0/EVENTLOOP_SPIN 0 N/A (N/A) N/A 49289 N/A N/A
TPC/0/READ_DISK_ASYNC 0 N/A (N/A) N/A 21 N/A N/A
TPC/0/READ_INTERNAL 0 N/A (N/A) N/A 1565 N/A N/A
TPC/0/READ_RANGE_INTERNAL 0 N/A (N/A) N/A 14 N/A N/A
TPC/0/READ_SWITCH_FOR_RESPONSE 0 N/A (N/A) N/A 1572 N/A N/A
TPC/0/TIMED_TIMEOUT 0 N/A (N/A) N/A 5005 N/A N/A
TPC/0/UNKNOWN 0 N/A (N/A) N/A 1 N/A N/A
TPC/0/WRITE_INTERNAL 0 N/A (N/A) N/A 33 N/A N/A
TPC/0/WRITE_SWITCH_FOR_MEMTABLE 0 N/A (N/A) N/A 251 N/A N/A
TPC/0/WRITE_SWITCH_FOR_RESPONSE 0 N/A (N/A) N/A 13 N/A N/A
TPC/all/EVENTLOOP_SPIN 0 N/A (N/A) N/A 49307 N/A N/A
TPC/all/NODESYNC_VALIDATION 0 N/A (N/A) N/A 2 N/A N/A
TPC/all/READ_DISK_ASYNC 0 N/A (N/A) N/A 21 N/A N/A
TPC/all/READ_INTERNAL 0 N/A (N/A) N/A 1565 N/A N/A
TPC/all/READ_RANGE_INTERNAL 0 N/A (N/A) N/A 14 N/A N/A
TPC/all/READ_SWITCH_FOR_RESPONSE 0 N/A (N/A) N/A 1572 N/A N/A
TPC/all/TIMED_TIMEOUT 0 N/A (N/A) N/A 5003 N/A N/A
TPC/all/UNKNOWN 0 N/A (N/A) N/A 1 N/A N/A
TPC/all/WRITE_INTERNAL 0 N/A (N/A) N/A 33 N/A N/A
TPC/all/WRITE_SWITCH_FOR_MEMTABLE 0 N/A (N/A) N/A 251 N/A N/A
TPC/all/WRITE_SWITCH_FOR_RESPONSE 0 N/A (N/A) N/A 13 N/A N/A
TPC/other 0 0 (0) 0 2 N/A 0
TPC/other/NODESYNC_VALIDATION 0 N/A (N/A) N/A 2 N/A N/A
Message type Dropped Latency waiting in queue (micros)
50% 95% 99% Max
RANGE_SLICE 0 N/A N/A N/A N/A
SNAPSHOT 0 N/A N/A N/A N/A
HINT 0 N/A N/A N/A N/A
COUNTER_MUTATION 0 N/A N/A N/A N/A
LWT 0 N/A N/A N/A N/A
BATCH_STORE 0 N/A N/A N/A N/A
VIEW_MUTATION 0 N/A N/A N/A N/A
READ 0 0.00 917.50 917.50 1048.58
OTHER 0 0.00 5242.88 5242.88 6291.46
REPAIR 0 N/A N/A N/A N/A
SCHEMA 0 917.50 1835.01 1835.01 2097.15
MUTATION 0 0.00 14680.06 14680.06 16777.22
NODESYNC 0 917.50 58720.26 58720.26 67108.86
READ_REPAIR 0 N/A N/A N/A N/A
TRUNCATE 0 N/A N/A N/A N/A