IndexPool MBean
The IndexPool MBean exposes metrics around the progress of indexing tasks as they move through the pipeline, and provides a mechanism to tweak the flushing, concurrency, and back-pressure behavior of a core's indexing thread pool.
The IndexPool MBean exposes metrics around the progress of indexing tasks as they move through the pipeline, and provides a mechanism to tweak the flushing, concurrency, and back-pressure behavior of a core indexing thread pool.
The index pool MBean is useful for controlling task submission and flush with these properties:
- Configurable concurrency
- The maximum number of concurrent workers is predefined at construction time. The actual concurrency can be dynamically configured between 1 (synchronous execution) and the given max concurrency.
- Flow control via back pressure
- To reduce memory consumption in case of fast producers, back pressure throttles incoming tasks based on a configurable queue threshold and max pause per task. The implementation is re-entrant if a worker submits a task on the pool, the task is not subjected to any pause.
Path
The index pool MBean is identified by the following path:
com.datastax.bdp.search.keyspace_name.table_name.IndexPool
where:- search is the Mbean type
- keyspace_name.table_name is the search core that the metrics reference
- IndexPool is the MBean name
IndexPool MBean attributes that you can modify
The attributes are effective only until the node is restarted. To make the change
permanent, you must change the corresponding option in
dse.yaml.
- FlushMaxTime
- The maximum time, in milliseconds, to wait before flushing asynchronous index updates, which occurs at Solr commit time or at Cassandra flush time. In dse.yaml (in minutes): flush_max_time_per_core.
- BackPressureThreshold
- The back pressure threshold is the target total number of queued asynchronous indexing requests per core; the back pressure mechanism will throttle incoming requests to keep the queue size as close to the threshold as possible. In dse.yaml: back_pressure_threshold_per_core.
- Concurrency
- The maximum number of concurrent asynchronous indexing threads. In dse.yaml: max_solr_concurrency_per_core.
IndexPool MBean view-only attributes
You can get the following MBean operations:
- BackPressurePauseNanos
- Get the average back pressure pause.
- BackPressurePauseOutliers
- The number of tasks from the scheduled document expiration job or re-indexing that pause longer than the configured maximum pause time.
- IncomingRate
- Get the 1-minute rate of ingested tasks per second.
- MaxConcurrency
- Get the predefined max concurrency level.
- OutgoingRate
- Get the 1-minute rate of processed tasks per second.
- ProcessedTasks
- Get the total number of processed tasks for all workers.
- QueueSize
- Get the current size of each processing queues.
- QueueWeight
- The total weight of all in-flight indexing tasks.
- TaskProcessingTimeNanos
- Get the last processing time for all workers. Could be 0 in case the clock resolution is too coarse.
- Throughput
- The 1-minute rate of work throughput per second.
- TotalQueueSize
- Get the total size of all processing queues. Should be lower than QueueWeight
The location of
the dse.yaml file depends
on the type of installation:
Installer-Services | /etc/dse/dse.yaml |
Package installations | /etc/dse/dse.yaml |
Installer-No Services | install_location/resources/dse/conf/dse.yaml |
Tarball installations | install_location/resources/dse/conf/dse.yaml |