nodetool leaksdetection

Enables and configures memory leak tracking.

cassandra.yaml

The location of the cassandra.yaml file depends on the type of installation:
Package installations /etc/dse/cassandra/cassandra.yaml
Tarball installations installation_location/resources/cassandra/conf/cassandra.yaml

system.log

The location of the system.log file is:
  • /var/log/cassandra/system.log

debug.log

  • The location of the debug.log file is: /var/log/cassandra/debug.log

Enables and configures memory leak tracking. When a leak is detected, tracking information is provided along with a stack trace in debug.log and system.log.

The following resources are tracked:
CachedReadsBufferPool
The non-blocking I/O (NIO) byte buffers used by file chunks stored in the chunk cache.
DirectReadsBufferPool
The NIO byte buffers used for transient, short-term operations, such as scattered file reads.
ChunkCache
The file chunks in the chunk cache. Also referred to as the file cache.
Memory
Native memory accessed directly with malloc calls and therefore not managed by the JVM. Used for compression metadata, bloom filters, and the row cache.
Note: The row cache should be disabled in DSE 6.x and later.
If memtables are using off-heap objects, the following resource can also be tracked:
NativeAllocator
The memory used for memtables when the memtable allocation type is offheap objects.
Tip: The nodetool leaksdetection parameters can also be set in cassandra.yaml. See Memory leak detection settings.

Synopsis

nodetool [connection_options] leaksdetection
[--set_max_stack_depth number] 
[--set_max_stacks_cache_size number]
[--set_num_access_records number] 
[--set_sampling_probability number]
[resource]
Table 1. Legend
Syntax conventions Description
UPPERCASE Literal keyword.
Lowercase Not literal.
Italics Variable value. Replace with a valid option or user-defined value.
[ ] Optional. Square brackets ( [ ] ) surround optional command arguments. Do not type the square brackets.
( ) Group. Parentheses ( ( ) ) identify a group to choose from. Do not type the parentheses.
| Or. A vertical bar ( | ) separates alternative elements. Type any one of the elements. Do not type the vertical bar.
... Repeatable. An ellipsis ( ... ) indicates that you can repeat the syntax element as often as required.
'Literal string' Single quotation ( ' ) marks must surround literal strings in CQL statements. Use single quotation marks to preserve upper case.
{ key:value } Map collection. Braces ( { } ) enclose map collections or key value pairs. A colon separates the key and the value.
<datatype1,datatype2> Set, list, map, or tuple. Angle brackets ( < > ) enclose data types in a set, list, map, or tuple. Separate the data types with a comma.
cql_statement; End CQL statement. A semicolon ( ; ) terminates all CQL statements.
[ -- ] Separate the command line options from the command arguments with two hyphens ( -- ). This syntax is useful when arguments might be mistaken for command line options.
' <schema> ... </schema> ' Search CQL only: Single quotation marks ( ' ) surround an entire XML schema declaration.
@xml_entity='xml_entity_type' Search CQL only: Identify the entity and literal value to overwrite the XML element in the schema and solrconfig files.

Definition

The short form and long form parameters are comma-separated.

Connection options

-h, --host hostname
The hostname or IP address of a remote node or nodes. When omitted, the default is the local machine.
-p, --port jmx_port
The JMX port number.
-pw, --password jmxpassword
The JMX password for authenticating with secure JMX. If a password is not provided, you are prompted to enter one.
-pwf, --password-file jmx_password_filepath
The filepath to the file that stores JMX authentication credentials.
-u, --username jmx_username
The username for authenticating with secure JMX.

Command arguments

--set_max_stack_depth number
The depth of the stack traces collected. Changes only the depth of the stack traces that will be collected from the time the parameter is set. Deeper stacks are more unique, so increasing the depth may require increasing stacks_cache_size_mb.
Default: 30
--set_max_stacks_cache_size_mb number
The maximum size of the cache for call stack traces. Stack traces are used to debug leaked resources, and use heap memory. Set the amount of heap memory dedicated to each resource by setting the max stacks cache size in MB.
Default: 32
--set_num_access_records number
The average number of stack traces kept when a resource is accessed. Only supported for chunks in the cache.
Default: 0
--set_sampling_probability number
The sampling probability to track for the specified resource.
  • 0 - disable tracking.
  • 1 - enable tracking all the time.
  • A number between 0 and 1 - the percentage of time to randomly track a resource. For example, 0.1 tracks resources 10% of the time.
CAUTION: Tracking incurs a significant stack trace collection cost for every access and consumes heap space. Enable tracking only when directed by DataStax Support.
Default: 0
resource
The resource to which the parameters should be applied. If not specified, the parameters affect all resources.

Examples

Get the current memory leak detection status

nodetool leaksdetection
Results:
Current Status:
NativeAllocator/Region           - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
CachedReadsBufferPool/ByteBuffer - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
DirectReadsBufferPool/ByteBuffer - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
ChunkCache/Chunk                 - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
Memory/Memory                    - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
RandomAccessReader/RandomAccessReader - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
AsyncPartitionReader/FlowSource  - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
AsyncSSTableScanner/FlowSource   - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30

Set the sampling probability to 25 percent on the Memory resource

nodetool leaksdetection --set_sampling_probability .25 Memory
CAUTION: Tracking incurs a significant stack trace collection cost for every access and consumes heap space. Enable tracking only when directed by DataStax Support.
Results:
Current Status:
NativeAllocator/Region           - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
CachedReadsBufferPool/ByteBuffer - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
DirectReadsBufferPool/ByteBuffer - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
ChunkCache/Chunk                 - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
Memory/Memory                    - Sampling probability: 0.250000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
RandomAccessReader/RandomAccessReader - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
AsyncPartitionReader/FlowSource  - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
AsyncSSTableScanner/FlowSource   - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30