nodetool leaksdetection

Enables and configures memory leak tracking. When a leak is detected, tracking information is provided along with a stack trace in debug.log and system.log.

The following resources are tracked:

CachedReadsBufferPool

The non-blocking I/O (NIO) byte buffers used by file chunks stored in the chunk cache.

DirectReadsBufferPool

The NIO byte buffers used for transient, short-term operations, such as scattered file reads.

ChunkCache

The file chunks in the chunk cache. Also referred to as the file cache.

Memory

Native memory accessed directly with malloc calls and therefore not managed by the JVM. Used for compression metadata, bloom filters, and the row cache.

The row cache should be disabled in DataStax Enterprise (DSE) 6.x and later.

If memtables are using off-heap objects, the following resource can also be tracked:

NativeAllocator: The memory used for memtables when the memtable allocation type is offheap objects.

The nodetool leaksdetection parameters can also be set in cassandra.yaml. See Memory leak detection settings.

Synopsis

nodetool [<connection_options>] leaksdetection
[--set_max_stack_depth <number>]
[--set_max_stacks_cache_size <number>]
[--set_num_access_records <number>]
[--set_sampling_probability <number>]
[resource]

Syntax legend

Syntax conventions Description

Syntax conventions	Description
Italic, bold, or `< >`	Syntax diagrams and code samples use one or more of these styles to mark placeholders for variable values. Replace placeholders with a valid option or your own user-defined value. In CQL statements, angle brackets are required to enclose data types in a set, list, map, or tuple. Separate the data types with a comma. For example: `<datatype2` In Search CQL statements, angle brackets are used to identify the entity and literal value to overwrite the XML element in the schema and `solrconfig` files, such as `@<xml_entity>='<xml_entity_type>'`.
`[ ]`	Square brackets surround optional command arguments. Do not type the square brackets.
`( )`	Parentheses identify a group to choose from. Do not type the parentheses.
`\|`	A pipe separates alternative elements. Type any one of the elements. Do not type the pipe.
`...`	Indicates that you can repeat the syntax element as often as required.
`'`	Single quotation marks must surround literal strings in CQL statements. Use single quotation marks to preserve upper case. + For Search CQL only: Single quotation marks surround an entire XML schema declaration, such as `'<<schema> ... </schema>>'`
`{ }`	Map collection. Curly braces enclose maps (`{ <key_datatype>:<value_datatype> }`) or key value pairs (`{ <key>:<value> }`). A colon separates the key and the value.
`;`	Ends a CQL statement.
`--`	Separate command line options from command arguments with two hyphens. This syntax is useful when arguments might be mistaken for command line options.

Italic, bold, or < >

Syntax diagrams and code samples use one or more of these styles to mark placeholders for variable values. Replace placeholders with a valid option or your own user-defined value.

In CQL statements, angle brackets are required to enclose data types in a set, list, map, or tuple. Separate the data types with a comma. For example: <datatype2

In Search CQL statements, angle brackets are used to identify the entity and literal value to overwrite the XML element in the schema and solrconfig files, such as @<xml_entity>='<xml_entity_type>'.

[ ]

Square brackets surround optional command arguments. Do not type the square brackets.

( )

Parentheses identify a group to choose from. Do not type the parentheses.

|

A pipe separates alternative elements. Type any one of the elements. Do not type the pipe.

...

Indicates that you can repeat the syntax element as often as required.

'

Single quotation marks must surround literal strings in CQL statements. Use single quotation marks to preserve upper case. + For Search CQL only: Single quotation marks surround an entire XML schema declaration, such as '<<schema> ... </schema>>'

{ }

Map collection. Curly braces enclose maps ({ <key_datatype>:<value_datatype> }) or key value pairs ({ <key>:<value> }). A colon separates the key and the value.

;

Ends a CQL statement.

--

Separate command line options from command arguments with two hyphens. This syntax is useful when arguments might be mistaken for command line options.

Options

If an option has a short and long form, both forms are given, separated by a comma.

-h, --host hostname

The hostname or IP address of a remote node or nodes. When omitted, the default is the local machine.

-p, --port jmx_port

The JMX port number.

-pw, --password jmxpassword

The JMX password for authenticating with secure JMX. If a password is not provided, you are prompted to enter one.

-pwf, --password-file jmx_password_filepath

The filepath to the file that stores JMX authentication credentials.

-u, --username jmx_username

The username for authenticating with secure JMX.

--set_max_stack_depth <number>

The depth of the stack traces collected. Changes only the depth of the stack traces that will be collected from the time the parameter is set. Deeper stacks are more unique, so increasing the depth may require increasing stacks_cache_size_mb.

Default: 30

--set_max_stacks_cache_size_mb <number>

The maximum size of the cache for call stack traces. Stack traces are used to debug leaked resources, and use heap memory. Set the amount of heap memory dedicated to each resource by setting the max stacks cache size in MB.

Default: 32

--set_num_access_records <number>

The average number of stack traces kept when a resource is accessed. Only supported for chunks in the cache.

Default: 0

--set_sampling_probability <number>

The sampling probability to track for the specified resource:

0 - disable tracking.
1 - enable tracking all the time.
A number between 0 and 1 - the percentage of time to randomly track a resource. For example, 0.1 tracks resources 10% of the time.

The default is .01.

Tracking incurs a significant stack trace collection cost for every access and consumes heap space. Enable tracking only when directed by DataStax Support.

resource

The resource to which the parameters should be applied. If not specified, the parameters affect all resources.

Examples

Get the current memory leak detection status

nodetool leaksdetection

Results:

Current Status:
NativeAllocator/Region           - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
CachedReadsBufferPool/ByteBuffer - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
DirectReadsBufferPool/ByteBuffer - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
ChunkCache/Chunk                 - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
Memory/Memory                    - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
RandomAccessReader/RandomAccessReader - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
AsyncPartitionReader/FlowSource  - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
AsyncSSTableScanner/FlowSource   - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30

Set the sampling probability to 25 percent on the Memory resource

nodetool leaksdetection --set_sampling_probability .25 Memory

Tracking incurs a significant stack trace collection cost for every access and consumes heap space. Enable tracking only when directed by DataStax Support.

Results:

Current Status:
NativeAllocator/Region           - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
CachedReadsBufferPool/ByteBuffer - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
DirectReadsBufferPool/ByteBuffer - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
ChunkCache/Chunk                 - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
Memory/Memory                    - Sampling probability: 0.250000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
RandomAccessReader/RandomAccessReader - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
AsyncPartitionReader/FlowSource  - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
AsyncSSTableScanner/FlowSource   - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30

nodetool leaksdetection

Synopsis

Options

Examples

Get the current memory leak detection status

Set the sampling probability to 25 percent on the Memory resource

Was this helpful?

Give Feedback