nodetool leaksdetection

Enables and configures memory leak tracking.

debug.log

The location of the debug.log file is: /var/log/cassandra/debug.log

cassandra.yaml

The location of the cassandra.yaml file depends on the type of installation:

Package installations	/etc/dse/cassandra/cassandra.yaml
Tarball installations	`installation_location`/resources/cassandra/conf/cassandra.yaml

system.log

The location of the system.log file is:

/var/log/cassandra/system.log

Enables and configures memory leak tracking. Tracking information is provided along with a stack trace in debug.log and system.log when a leak is detected.

The resources currently tracked are:

CachedReadsBufferPool: The non-blocking i/o (NIO) byte buffers that are used by file chunks stored in the chunk cache. The chunk cache is also referred to as the file cache.
DirectReadsBufferPool: The NIO byte buffers that are used for transient, short-term operations, such as some scattered file reads.
ChunkCache: The file chunks in the chunk cache. The chunk cache is also referred to as the file cache.
Memory: Native memory accessed directly with malloc calls and therefore not managed by the JVM. Currently used for compression metadata, bloom filters and the row cache.
Note: The row cache should be disabled in DSE 6.x and later.

If memtables are using off-heap objects, the following resource can also be tracked:

NativeAllocator: The memory used for memtables when the memtable allocation type is offheap objects.

Tip: The leaksdetection parameters can also be set in . See Memory leak detection settings.

Synopsis

nodetool [connection_options] leaksdetection
         [--set_max_stack_depth number] 
         [--set_max_stacks_cache_size number]
         [--set_num_access_records number] 
         [--set_sampling_probability number]
         [resource]

Table 1. Legend
Syntax conventions	Description
UPPERCASE	Literal keyword.
Lowercase	Not literal.
`Italics`	Variable value. Replace with a valid option or user-defined value.
`[ ]`	Optional. Square brackets ( `[ ]` ) surround optional command arguments. Do not type the square brackets.
`( )`	Group. Parentheses ( `( )` ) identify a group to choose from. Do not type the parentheses.
`\|`	Or. A vertical bar ( `\|` ) separates alternative elements. Type any one of the elements. Do not type the vertical bar.
`...`	Repeatable. An ellipsis ( `...` ) indicates that you can repeat the syntax element as often as required.
`'Literal string'`	Single quotation ( `'` ) marks must surround literal strings in CQL statements. Use single quotation marks to preserve upper case.
`{ key:value }`	Map collection. Braces ( `{ }` ) enclose map collections or key value pairs. A colon separates the key and the value.
`<datatype1,datatype2>`	Set, list, map, or tuple. Angle brackets ( `< >` ) enclose data types in a set, list, map, or tuple. Separate the data types with a comma.
`cql_statement;`	End CQL statement. A semicolon ( `;` ) terminates all CQL statements.
`[ -- ]`	Separate the command line options from the command arguments with two hyphens ( `--` ). This syntax is useful when arguments might be mistaken for command line options.
`' <schema> ... </schema> '`	Search CQL only: Single quotation marks ( `'` ) surround an entire XML schema declaration.
`@xml_entity='xml_entity_type'`	Search CQL only: Identify the entity and literal value to overwrite the XML element in the schema and solrconfig files.

Definition

The short form and long form parameters are comma-separated.

Connection options

-h, --host hostname: The hostname or IP address of a remote node or nodes. When omitted, the default is the local machine.
-p, --port jmx_port: The JMX port number.
-pw, --password jmxpassword: The JMX password for authenticating with secure JMX. If a password is not provided, you are prompted to enter one.
-pwf, --password-file jmx_password_filepath: The filepath to the file that stores JMX authentication credentials.
-u, --username jmx_username: The user name for authenticating with secure JMX.

Command arguments

--set_max_stack_depth number: The depth of the stack traces collected. Changes only the depth of the stack traces that will be collected from the time the parameter is set. Deeper stacks are more unique, so increasing the depth may require increasing stacks_cache_size_mb.; Default: 30
--set_max_stacks_cache_size_mb number: Set the size of the cache for call stack traces. Stack traces are used to debug leaked resources, and use heap memory. Set the amount of heap memory dedicated to each resource by setting the max stacks cache size in MB.; Default: 32
--set_num_access_records number: Set the average number of stack traces kept when a resource is accessed. Currently only supported for chunks in the cache.; Default: 0
--set_sampling_probability number: Set the sampling probability. Each resource is tracked with a sampling probability. Set the sampling probability to 0 to disable tracking and to 1 to enable tracking all the time. A number between 0 and 1 will randomly track a resource. For example, 0.5 will track resources 50% of the time.; Default: 0
CAUTION: Tracking incurs a significant stack trace collection cost for every access and consumes heap space. It should never be enabled unless directed by a support engineer or consultant.
resource: The resource to which the parameters should be applied. If not specified, the parameters affect all resources.

Examples

Print the current memory leak detection status

nodetool leaksdetection

Result:

Current Status:
CachedReadsBufferPool/ByteBuffer - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
DirectReadsBufferPool/ByteBuffer - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
ChunkCache/Chunk                 - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
Memory/Memory                    - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30

Set the sampling probability to 25 percent on the Memory resource

nodetool leaksdetection --set_sampling_probability .25 Memory

Result:

Current Status:
CachedReadsBufferPool/ByteBuffer - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
DirectReadsBufferPool/ByteBuffer - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
ChunkCache/Chunk                 - Sampling probability: 0.000000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30
Memory/Memory                    - Sampling probability: 0.100000, Max stacks cache size MB: 32, num. access records: 0, max stack depth: 30.