nodetool leaksdetection

Enables and configures memory leak tracking. When a leak is detected, tracking information is provided along with a stack trace in debug.log and system.log.

The following resources are tracked:

CachedReadsBufferPool

The non-blocking I/O (NIO) byte buffers used by file chunks stored in the chunk cache.

DirectReadsBufferPool

The NIO byte buffers used for transient, short-term operations, such as scattered file reads.

ChunkCache

The file chunks in the chunk cache. Also referred to as the file cache.

Memory

Native memory accessed directly with malloc calls and therefore not managed by the JVM. Used for compression metadata, bloom filters, and the row cache.

The row cache should be disabled in DSE 6.x and later.

If memtables are using off-heap objects, the following resource can also be tracked:

NativeAllocator: The memory used for memtables when the memtable allocation type is offheap objects.

The nodetool leaksdetection parameters can also be set in cassandra.yaml. See Memory leak detection settings.

Synopsis

nodetool [<connection_options>] leaksdetection
[--set_max_stack_depth <number>]
[--set_max_stacks_cache_size <number>]
[--set_num_access_records <number>]
[--set_sampling_probability <number>]
[resource]

Syntax conventions Description

Syntax conventions	Description
UPPERCASE	Literal keyword.
Lowercase	Not literal.
<`Italics>`	Variable value. Replace with a valid option or user-defined value.
`[ ]`	Optional. Square brackets ( `[ ]` ) surround optional command arguments. Do not type the square brackets.
`( )`	Group. Parentheses ( `( )` ) identify a group to choose from. Do not type the parentheses.
`\|`	Or. A vertical bar ( `\|` ) separates alternative elements. Type any one of the elements. Do not type the vertical bar.
`...`	Repeatable. An ellipsis ( `...` ) indicates that you can repeat the syntax element as often as required.
`'<Literal string>'`	Single quotation ( `'` ) marks must surround literal strings in CQL statements. Use single quotation marks to preserve upper case.
`{ <key>:<value> }`	Map collection. Braces ( `{ }` ) enclose map collections or key value pairs. A colon separates the key and the value.
`<<datatype1>,<datatype2>>`	Set, list, map, or tuple. Angle brackets ( `< >` ) enclose data types in a set, list, map, or tuple. Separate the data types with a comma.
`cql_statement;`	End CQL statement. A semicolon ( `;` ) terminates all CQL statements.
`[ -- ]`	Separate the command line options from the command arguments with two hyphens ( `--` ). This syntax is useful when arguments might be mistaken for command line options.
`' <<schema> ... </schema> >'`	Search CQL only: Single quotation marks ( `'` ) surround an entire XML schema declaration.
`@<xml_entity>='<xml_entity_type>'`	Search CQL only: Identify the entity and literal value to overwrite the XML element in the schema and solrconfig files.

UPPERCASE

Literal keyword.

Lowercase

Not literal.

<`Italics>`

Variable value. Replace with a valid option or user-defined value.

[ ]

Optional. Square brackets ( [ ] ) surround optional command arguments. Do not type the square brackets.

( )

Group. Parentheses ( ( ) ) identify a group to choose from. Do not type the parentheses.

|

Or. A vertical bar ( | ) separates alternative elements. Type any one of the elements. Do not type the vertical bar.

...

Repeatable. An ellipsis ( ... ) indicates that you can repeat the syntax element as often as required.

'<Literal string>'

Single quotation ( ' ) marks must surround literal strings in CQL statements. Use single quotation marks to preserve upper case.

{ <key>:<value> }

Map collection. Braces ( { } ) enclose map collections or key value pairs. A colon separates the key and the value.

<<datatype1>,<datatype2>>

Set, list, map, or tuple. Angle brackets ( < > ) enclose data types in a set, list, map, or tuple. Separate the data types with a comma.

cql_statement;

End CQL statement. A semicolon ( ; ) terminates all CQL statements.

[ -- ]

Separate the command line options from the command arguments with two hyphens ( -- ). This syntax is useful when arguments might be mistaken for command line options.

' <<schema> ... </schema> >'

Search CQL only: Single quotation marks ( ' ) surround an entire XML schema declaration.

@<xml_entity>='<xml_entity_type>'

Search CQL only: Identify the entity and literal value to overwrite the XML element in the schema and solrconfig files.

Definition

The short- and long-form options are comma-separated.

Connection options

-h, --host hostname: The hostname or IP address of a remote node or nodes. When omitted, the default is the local machine.
-p, --port jmx_port: The JMX port number.
-pw, --password jmxpassword: The JMX password for authenticating with secure JMX. If a password is not provided, you are prompted to enter one.
-pwf, --password-file jmx_password_filepath: The filepath to the file that stores JMX authentication credentials.
-u, --username jmx_username: The username for authenticating with secure JMX.

Command arguments

--set_max_stack_depth <number>: The depth of the stack traces collected. Changes only the depth of the stack traces that will be collected from the time the parameter is set. Deeper stacks are more unique, so increasing the depth may require increasing stacks_cache_size_mb.

Default: 30

--set_max_stacks_cache_size_mb <number>: The maximum size of the cache for call stack traces. Stack traces are used to debug leaked resources, and use heap memory. Set the amount of heap memory dedicated to each resource by setting the max stacks cache size in MB.

Default: 32

--set_num_access_records <number>: The average number of stack traces kept when a resource is accessed. Only supported for chunks in the cache.

Default: 0

--set_sampling_probability <number>

The sampling probability to track for the specified resource.

0 - disable tracking.
1 - enable tracking all the time.
A number between 0 and 1 - the percentage of time to randomly track a resource. For example, 0.1 tracks resources 10% of the time.

Tracking incurs a significant stack trace collection cost for every access and consumes heap space. Enable tracking only when directed by DataStax Support.

+ Default: .01

resource: The resource to which the parameters should be applied. If not specified, the parameters affect all resources.

Examples

Get the current memory leak detection status

nodetool leaksdetection

Results:

Current Status:
NativeAllocator/Region           - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
CachedReadsBufferPool/ByteBuffer - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
DirectReadsBufferPool/ByteBuffer - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
ChunkCache/Chunk                 - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
Memory/Memory                    - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
RandomAccessReader/RandomAccessReader - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
AsyncPartitionReader/FlowSource  - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
AsyncSSTableScanner/FlowSource   - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30

Set the sampling probability to 25 percent on the Memory resource

nodetool leaksdetection --set_sampling_probability .25 Memory

Tracking incurs a significant stack trace collection cost for every access and consumes heap space. Enable tracking only when directed by DataStax Support.

Results:

Current Status:
NativeAllocator/Region           - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
CachedReadsBufferPool/ByteBuffer - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
DirectReadsBufferPool/ByteBuffer - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
ChunkCache/Chunk                 - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
Memory/Memory                    - Sampling probability: 0.250000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
RandomAccessReader/RandomAccessReader - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
AsyncPartitionReader/FlowSource  - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
AsyncSSTableScanner/FlowSource   - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30

nodetool leaksdetection

Synopsis

Definition

Connection options

Command arguments

Examples

Get the current memory leak detection status

Set the sampling probability to 25 percent on the Memory resource

Was this helpful?

Give Feedback