dse.yaml configuration file

Scheduler settings for DSE Search indexes

ttl_index_rebuild_options:
    fixed_rate_period: 300
    initial_delay: 20
    max_docs_per_batch: 4096
    thread_pool_size: 1

ttl_index_rebuild_options

Configures the schedulers in charge of querying for expired records, removing expired records, and the execution of the checks.

fix_rate_period

Time interval in seconds to check for expired data in seconds.

Default: 300

initial_delay

The number of seconds to delay the first TTL check to speed up start-up time.

Default: 20

max_docs_per_batch

The maximum number of documents to check and delete per batch by the TTL rebuild thread. All expired documents are deleted from the index during each check. To avoid memory pressure, their unique keys are retrieved and then deletes are issued in batches.

Default: 4096

thread_pool_size

The maximum number of search indexes (cores) that can execute TTL cleanup concurrently. Manages system resource consumption and prevents many search cores from executing simultaneous TTL deletes.

Default: 1

Reindexing of bootstrapped data

async_bootstrap_reindex: false

async_bootstrap_reindex

For DSE Search, configure whether to asynchronously reindex bootstrapped data.

• true - The node joins the ring immediately after bootstrap and reindexing occurs asynchronously. Do not wait for post-bootstrap reindexing so that the node is not marked down. The dsetool ring command can be used to check the status of the reindexing.
• false - The node joins the ring after reindexing the bootstrapped data.

Default: false

CQL Solr paging

cql_solr_query_paging: off

cql_solr_query_paging

• driver - Respects driver paging settings. Uses Solr pagination (cursors) only when the driver uses pagination. Enabled automatically for DSE SearchAnalytics workloads.
• off - Paging is off. Ignore driver paging settings for CQL queries and use normal Solr paging unless:
  • The current workload is an analytics workload, including SearchAnalytics. SearchAnalytics nodes always use driver paging settings.
  • The cqlsh query parameter paging is set to driver.

  Even when cql_solr_query_paging: off, paging is dynamically enabled with the "paging":"driver" parameter in JSON queries.

Default: off

Solr CQL query option

cql_solr_query_row_timeout: 10000

cql_solr_query_row_timeout

The maximum time in milliseconds to wait for all rows to be read from the database during CQL Solr queries.

Default: 10000 (10 seconds)

DSE Search resource upload limit

solr_resource_upload_limit_mb: 10

solr_resource_upload_limit_mb

Configures the maximum file size of the search index config or schema. Resource files can be uploaded, but the search index config and schema are stored internally in the database after upload.

• 0 - Disable resource uploading.
• upload size - The maximum upload size limit in megabytes (MB) for a DSE Search resource file (search index config or schema).

Default: 10

Shard transport

shard_transport_options:
    netty_client_request_timeout: 60000

DSE Search indexing

# back_pressure_threshold_per_core: 1024
# flush_max_time_per_core: 5
# load_max_time_per_core: 5
# enable_index_disk_failure_policy: false
# solr_data_dir: /MyDir
# solr_field_cache_enabled: false
# ram_buffer_heap_space_in_mb: 1024
# ram_buffer_offheap_space_in_mb: 1024

back_pressure_threshold_per_core

The maximum number of queued partitions during search index rebuilding and reindexing. This maximum number safeguards against excessive heap use by the indexing queue. If set lower than the number of threads per core (TPC), not all TPC threads can be actively indexing.

Default: 1024

flush_max_time_per_core

The maximum time, in minutes, to wait for the flushing of asynchronous index updates that occurs at DSE Search commit time or at flush time.

CAUTION: Expert knowledge is required to change this value.

Always set the wait time high enough to ensure flushing completes successfully to fully sync DSE Search indexes with the database data. If the wait time is exceeded, index updates are only partially committed and the commit log is not truncated which can undermine data durability.

Note: When a timeout occurs, this node is typically overloaded and cannot flush in a timely manner. Live indexing increases the time to flush asynchronous index updates.

Default: 5

load_max_time_per_core

The maximum time, in minutes, to wait for each DSE Search index to load on startup or create/reload operations. This advanced option should be changed only if exceptions happen during search index loading.

Default: 5

enable_index_disk_failure_policy

Whether to apply the configured disk failure policy if IOExceptions occur during index update operations.

• true - Apply the configured Cassandra disk failure policy to index write failures
• false - Do not apply the disk failure policy

Default: false

solr_data_dir

The directory to store index data. See Managing the location of DSE Search data. By default, each DSE Search index is saved in solr_data_dir/keyspace_name.table_name or as specified by the dse.solr.data.dir system property.

Default: A solr.data directory in the cassandra data directory, like /var/lib/cassandra/solr.data

solr_field_cache_enabled

The Apache Lucene® field cache is deprecated. Instead, for fields that are sorted, faceted, or grouped by, set docValues="true" on the field in the search index schema. Then reload the search index and reindex.

Default: false

ram_buffer_heap_space_in_mb

Global Lucene RAM buffer usage threshold for heap to force segment flush. Setting too low can cause a state of constant flushing during periods of ongoing write activity. For near-real-time (NRT) indexing, forced segment flushes also de-schedule pending auto-soft commits to avoid potentially flushing too many small segments.

Default: 1024

ram_buffer_offheap_space_in_mb

Global Lucene RAM buffer usage threshold for offheap to force segment flush. Setting too low can cause a state of constant flushing during periods of ongoing write activity. For NRT, forced segment flushes also de-schedule pending auto-soft commits to avoid potentially flushing too many small segments. When not set, the default is 1024.

Default: 1024

Global Performance Service

# performance_core_threads: 4
# performance_max_threads: 32
# performance_queue_capacity: 32000

Performance Service

Configures the collection of performance metrics on transactional nodes. Performance metrics are stored in the dse_perf keyspace and can be queried using any CQL-based utility, such as cqlsh or any application using a CQL driver. To temporarily make changes for diagnostics and testing, use the dsetool perf subcommands.

graph_events:
   ttl_seconds: 600

graph_events

Graph event information.

ttl_seconds

Number of seconds a record survives before it is expired.

Default: 600

# cql_slow_log_options:
#   enabled: true
#   threshold: 200.0
#   minimum_samples: 100
#   ttl_seconds: 259200
#   skip_writing_to_db: true
#   num_slowest_queries: 5

cql_slow_log_options

Configures reporting distributed sub-queries for search (query executions on individual shards) that take longer than a specified period of time.

enabled

• true - Enables log entries for slow queries.
• false - Does not enable log entries.

Default: true

threshold

The threshold in milliseconds or as a percentile.

• A value greater than 1 is expressed in time and will log queries that take longer than the specified number of milliseconds. For example, 200.0 sets the threshold at 0.2 seconds.
• A value of 0 to 1 is expressed as a percentile and will log queries that exceed this percentile. For example, .95 collects information on 5% of the slowest queries.

Default: 200.0

minimum_samples

The initial number of queries before activating the percentile filter.

Default: commented out (100)

ttl_seconds

Number of seconds a slow log record survives before it is expired.

Default: 259200

skip_writing_to_db

Keeps slow queries only in-memory and does not write data to database.

• true - Keep slow queries only in-memory. Skip writing to database.
• false - Write slow query information in the node_slow_log table. The threshold must be >= 2000 ms to prevent a high load on the database.

Default: commented out (true)

num_slowest_queries

The number of slow queries to keep in-memory.

Default: commented out (5)

cql_system_info_options:
    enabled: false
    refresh_rate_ms: 10000

cql_system_info_options

Configures collection of system-wide performance information about a cluster.

enabled

Enables collection of system-wide performance information about a cluster.

• true - Collect metrics.
• false - Do not collect metrics.

Default: false

refresh_rate_ms

The length of the sampling period in milliseconds; the frequency to update the performance statistics.

Default: 10000 (10 seconds)

resource_level_latency_tracking_options:
    enabled: false
    refresh_rate_ms: 10000

resource_level_latency_tracking_options

Configures collection of object I/O performance statistics.

Tip: See Collecting system level diagnostics.

enabled

Enables collection of object input output performance statistics.

• true - Collect metrics.
• false - Do not collect metrics.

Default: false

refresh_rate_ms

The length of the sampling period in milliseconds; the frequency to update the performance statistics.

Default: 10000 (10 seconds)

db_summary_stats_options:
    enabled: false
    refresh_rate_ms: 10000

db_summary_stats_options

Configures collection of summary statistics at the database level.

Tip: See Collecting database summary diagnostics.

enabled

Enables collection of database summary performance information.

• true - Collect metrics.
• false - Do not collect metrics.

Default: false

refresh_rate_ms

The length of the sampling period in milliseconds; the frequency to update the performance statistics.

Default: 10000 (10 seconds)

cluster_summary_stats_options:
    enabled: false
    refresh_rate_ms: 10000

cluster_summary_stats_options

Configures collection of statistics at a cluster-wide level.

Tip: See Collecting cluster summary diagnostics.

enabled

Enables collection of statistics at a cluster-wide level.

• true - Collect metrics.
• false - Do not collect metrics.

Default: false

refresh_rate_ms

The length of the sampling period in milliseconds; the frequency to update the performance statistics.

Default: 10000 (10 seconds)

spark_cluster_info_options

Configures collection of data associated with Spark cluster and Spark applications.

spark_cluster_info_options:
    enabled: false
    refresh_rate_ms: 10000

Tip: See Monitoring Spark with Spark Performance Objects.

enabled

Enables collection of Spark performance statistics.

• true - Collect metrics.
• false - Do not collect metrics.

Default: false

refresh_rate_ms

The length of the sampling period in milliseconds; the frequency to update the performance statistics.

Default: 10000 (10 seconds)

histogram_data_options:
    enabled: false
    refresh_rate_ms: 10000
    retention_count: 3

histogram_data_options

Histogram data for the dropped mutation metrics are stored in the dropped_messages table in the dse_perf keyspace.

Tip: See Collecting histogram diagnostics.

enabled

• true - Collect metrics.
• false - Do not collect metrics.

Default: false

refresh_rate_ms

The length of the sampling period in milliseconds; the frequency to update the performance statistics.

Default: 10000 (10 seconds)

retention_count

Default: 3

user_level_latency_tracking_options:
    enabled: false
    refresh_rate_ms: 10000
    top_stats_limit: 100
    quantiles: false

user_level_latency_tracking_options

User-resource latency tracking settings.

Tip: See Collecting user activity diagnostics.

enabled

• true - Collect metrics.
• false - Do not collect metrics.

Default: false

refresh_rate_ms

The length of the sampling period in milliseconds; the frequency to update the performance statistics.

Default: 10000 (10 seconds)

top_stats_limit

The maximum number of individual metrics.

Default: 100

quantiles

Default: false

DSE Search Performance Service

solr_slow_sub_query_log_options:
    enabled: false
    ttl_seconds: 604800
    async_writers: 1
    threshold_ms: 3000

solr_slow_sub_query_log_options

See Collecting slow search queries.

enabled

• true - Collect metrics.
• false - Do not collect metrics.

Default: false

ttl_seconds

The number of seconds a record survives before it is expired.

Default: 604800 (about 10 minutes)

async_writers

The number of server threads dedicated to writing in the log. More than one server thread might degrade performance.

Default: 1

threshold_ms

Default: 3000

solr_update_handler_metrics_options:
    enabled: false
    ttl_seconds: 604800
    refresh_rate_ms: 60000

solr_update_handler_metrics_options

Options to collect search index direct update handler statistics over time.

Tip: See Collecting handler statistics.

solr_request_handler_metrics_options:
    enabled: false
    ttl_seconds: 604800
    refresh_rate_ms: 60000

solr_request_handler_metrics_options

Options to collect search index request handler statistics over time.

Tip: See Collecting handler statistics.

solr_index_stats_options:
    enabled: false
    ttl_seconds: 604800
    refresh_rate_ms: 60000

solr_index_stats_options

Options to record search index statistics over time.

Tip: See Collecting index statistics.

solr_cache_stats_options:
    enabled: false
    ttl_seconds: 604800
    refresh_rate_ms: 60000

solr_cache_stats_options

See Collecting cache statistics.

solr_latency_snapshot_options:
    enabled: false
    ttl_seconds: 604800
    refresh_rate_ms: 60000

solr_latency_snapshot_options

See Collecting Apache Solr performance statistics.

Spark Performance Service

spark_application_info_options:
    enabled: false
    refresh_rate_ms: 10000
    driver:
        sink: false
        connectorSource: false
        jvmSource: false
        stateSource: false
    executor:
        sink: false
        connectorSource: false
        jvmSource: false

spark_application_info_options

Collection of Spark application metrics.

enabled

• true - Collect metrics.
• false - Do not collect metrics.

Default: false

refresh_rate_ms

The length of the sampling period in milliseconds; the frequency to update the performance statistics.

Default: 10000 (10 seconds)

driver

Collection that configures collection of metrics at the Spark Driver.

connectorSource

Enables collecting Spark Cassandra Connector metrics at the Spark Driver.

• true - Collect metrics.
• false - Do not collect metrics.

Default: false

jvmSource

Enables collection of JVM heap and garbage collection (GC) metrics from the Spark Driver.

• true - Collect metrics.
• false - Do not collect metrics.

Default: false

stateSource

Enables collection of application state metrics at the Spark Driver.

• true - Collect metrics.
• false - Do not collect metrics.

Default: false

executor

Configures collection of metrics at Spark executors.

sink

Enables collecting metrics collected at Spark executors.

• true - Collect metrics.
• false - Do not collect metrics.

Default: false

connectorSource

Enables collection of Spark Cassandra Connector metrics at Spark executors.

• true - Collect metrics.
• false - Do not collect metrics.

Default: false

jvmSource

Enables collection of JVM heap and GC metrics at Spark executors.

• true - Collect metrics.
• false - Do not collect metrics.

Default: false

Spark resource options

spark_shared_secret_bit_length: 256
spark_security_enabled: false
spark_security_encryption_enabled: false

spark_daemon_readiness_assertion_interval: 1000

resource_manager_options:
   worker_options:
       cores_total: 0.7
       memory_total: 0.6

       workpools:
          - name: alwayson_sql
          cores: 0.25
          memory: 0.25

spark_shared_secret_bit_length

The length of a shared secret used to authenticate Spark components and encrypt the connections between them. This value is not the strength of the cipher for encrypting connections.

Default: 256

spark_security_enabled

In DSE 6.7.4 and later, when DSE authentication is enabled with authentication_options, Spark security is enabled regardless of this setting.

In DSE 6.7.0-6.7.3, enables and disables Spark security based on shared secret infrastructure for mutual authentication and optional encryption between DSE Spark Master and Workers, and communication channels, except the web UI.

Default: false

spark_security_encryption_enabled

In DSE 6.7.4 and later, when DSE authentication is enabled with authentication_options, Spark security encryption is enabled regardless of this setting.

In DSE 6.7.0-6.7.3, enables and disables encryption between DSE Spark Master and Workers, and communication channels, except the web UI. Uses DIGEST-MD5 SASL-based encryption mechanism. Requires spark_security_enabled: true.

Tip: Configure encryption between the Spark processes and DSE with client-to-node encryption in cassandra.yaml.

Default: false

spark_daemon_readiness_assertion_interval

Time interval in milliseconds between subsequent retries by the Spark plugin for Spark Master and Worker readiness to start.

Default: 1000

resource_manager_options

Controls the physical resources used by Spark applications on this node. Optionally add named workpools with specific dedicated resources. See Core management.

worker_options

Configures the amount of system resources that are made available to the Spark Worker.

cores_total

The number of total system cores available to Spark.

Note: For DSE 6.7.7 and later, the SPARK_WORKER_TOTAL_CORES environment variables takes precedence over this setting.

The lowest value that you can assign to Spark Worker cores is 1 core. If the results are lower, no exception is thrown and the values are automatically limited.

Note: Setting cores_total or a workpool's cores to 1.0 is a decimal value, meaning 100% of the available cores will be reserved. Setting cores_total or cores to 1 (no decimal point) is an explicit value, and one core will be reserved.

Default: 0.7

memory_total

The amount of total system memory available to Spark.

• absolute value - Use standard suffixes like M for megabyte and G for gigabyte. For example, 12G.
• decimal value - Maximum fraction of system memory to give all executors for all applications running on a particular node. For example, 0.8.
  When the value is expressed as a decimal, the available resources are calculated in the following way:

  Spark Worker memory = memory_total x (total system memory - memory assigned to DataStax Enterprise)

  The lowest values that you can assign to Spark Worker memory is 64 MB. If the results are lower, no exception is thrown and the values are automatically limited.

Note: For DSE 6.7.7 and later, the SPARK_WORKER_TOTAL_MEMORY environment variables takes precedence over this setting.

Default: 0.6

workpools

A collection of named workpools that can use a portion of the total resources defined under worker_options.

A default workpool named default is used if no workpools are defined in this section. If workpools are defined, the resources allocated to the workpools are taken from the total amount, with the remaining resources available to the default workpool.

The total amount of resources defined in the workpools section must not exceed the resources available to Spark in worker_options.

name

The name of the workpool. A workpool named alwayson_sql is created by default for AlwaysOn SQL. By default, the alwayson_sql workpool is configured to use 25% of the resources available to Spark.

Default: alwayson_sql

cores

The number of system cores to use in this workpool expressed as an absolute value or a decimal value. This option follows the same rules as cores_total.

memory

The amount of memory to use in this workpool expressed as either an absolute value or a decimal value. This option follows the same rules as memory_total.

Spark encryption options

spark_ui_options:
    encryption: inherit
    encryption_options:
        enabled: false
        keystore: resources/dse/conf/.ui-keystore
        keystore_password: cassandra
        require_client_auth: false
        truststore: .truststore
        truststore_password: cassandra
        # Advanced settings
        # protocol: TLS
        # algorithm: SunX509
        # keystore_type: JKS
        # truststore_type: JKS
        # cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]

spark_ui_options

Configures encryption for Spark Master and Spark Worker UIs. These options apply only to Spark daemon UIs, and do not apply to user applications even when the user applications are run in cluster mode.

Tip: To set permissions on roles to allow Spark applications to be started, stopped, managed, and viewed, see Using authorization with Spark

encryption

The source for SSL settings.

• inherit - Inherit the SSL settings from the client_encryption_options in cassandra.yaml.
• custom - Use the following encryption_options in dse.yaml.

Default: inherit

encryption_options

When encryption: custom, configures encryption for HTTPS of Spark Master and Worker UI.

enabled

Enables Spark encryption for Spark client-to-Spark cluster and Spark internode communication.

Default: false

keystore

The keystore for Spark encryption keys.

The relative filepath is the base Spark configuration directory that is defined by the SPARK_CONF_DIR environment variable. The default Spark configuration directory is resources/spark/conf.

Default: resources/dse/conf/.ui-keystore

keystore_password

The password to access the keystore.

Default: cassandra

require_client_auth

Enables custom truststore for client authentication.

• true - Require custom truststore for client authentication.
• false - Do not require custom truststore.

Default: false

truststore

The filepath to the truststore for Spark encryption keys if require_client_auth: true.

The relative filepath is the base Spark configuration directory that is defined by the SPARK_CONF_DIR environment variable. The default Spark configuration directory is resources/spark/conf.

Default: resources/dse/conf/.ui-truststore

truststore_password

The password to access the truststore.

Default: cassandra

protocol

The Transport Layer Security (TLS) authentication protocol. The TLS protocol must be supported by JVM and Spark. TLS 1.2 is the most common JVM default.

Default: JVM default

algorithm

The key manager algorithm.

Default: SunX509

keystore_type

Valid types are JKS, JCEKS, PKCS11, and PKCS12. For file-based keystores, use PKCS12.

Attention: DataStax supports PKCS11 as a keystore_type on nodes with cassandra or advanced workloads. The cassandra workload support is specific to DSE 6.7.7 and later releases. The advanced workload support is specific to DSE 6.7.9 and later. If PKCS11 is needed, in server_encryption_options or client_encryption_options, specify the keystore_type as PKCS11 and the keystore as NONE. PKCS11 is not supported in DSE 6.0.x and 5.1.x releases. PKCS11 is not supported as a truststore_type.

Default: JKS

truststore_type

Valid types are JKS, JCEKS, and PKCS12.

Default: commented out (JKS)

cipher_suites

A comma-separated list of cipher suites for Spark encryption. Enclose the list in square brackets.

• TLS_RSA_WITH_AES_128_CBC_SHA
• TLS_RSA_WITH_AES_256_CBC_SHA
• TLS_DHE_RSA_WITH_AES_128_CBC_SHA
• TLS_DHE_RSA_WITH_AES_256_CBC_SHA
• TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
• TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

Default: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]

Starting Spark drivers and executors

spark_process_runner:
    runner_type: default
    run_as_runner_options: 
        user_slots:
            - slot1
            - slot2

spark_process_runner:

Configures how Spark driver and executor processes are created and managed. See Running Spark processes as separate users.

runner_type

• default - Use the default runner type.
• run_as - Spark applications run as a different OS user than the DSE service user.

run_as_runner_options

When runner_type: run_as, Spark applications run as a different OS user than the DSE service user.

user_slots

The list slot users to separate Spark processes users from the DSE service user.

Default: slot1, slot2