Test your DataStax Enterprise (DSE) cluster

Before deploying to production, verify your configuration and test your cluster with simulated production workloads in isolation and at scale.

Testing helps you find and resolve issues before applications and configuration changes go live. Metrics collected during testing help you set measurable benchmarks, such as throughput and latency, that you can use to monitor performance in production and project future capacity needs.

After deploying to production, continue to monitor, reconfigure, and test your cluster as needed.

Monitoring and testing are universal best practices for application development. This guide provides specific advice for testing DSE clusters with a focus on new deployments. It doesn’t cover all aspects of synthetic testing and full-stack observability.

Prerequisites

This cluster testing guide assumes that you have reviewed and implemented the cluster capacity planning guidance and recommended production settings as applicable to your environment and workloads:
To monitor and test a cluster, you need a cluster with DSE installed on at least one node.

Ideally, your test cluster should mimic your production environment as closely as possible in terms of configuration, node count, and workload characteristics.

Run system checks

You can run the following system-level checks when planning a DataStax Enterprise (DSE) deployment or investigating cluster performance issues.

Run these checks on each node in a cluster, or use diagnostic scripts to gather this information in bulk. DataStax recommends using diagnostic scripts for large clusters with many nodes and multi-server deployments where it can be tedious to run individual checks on each node.

Some checks can be completed while the nodes are idle. Others require activity on the node or network traffic flowing to and from the node. You might need to revisit these checks during your simulated workload tests.

DSE 6.8 performs some of these checks on system start. Potential misconfigurations are output in the start-up logs. For example:

WARN [main] 2020-02-13 11:00:47,676 SigarLibrary.java:174 - Cassandra server running in degraded mode. Is swap disabled? : false, Address space adequate? : true, nofile limit adequate? : true, nproc limit adequate? : true
WARN [main] 2020-02-13 11:00:47,677 StartupChecks.java:311 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl.

Operating system configuration

Review and check all recommended production settings according to your environment and workloads. For example:

Disable Hugepage defragmentation.
Disable swap.
Avoid excessive heap size.
Synchronize system clocks on all nodes in the cluster and on all clients (such as Cassandra drivers) that interact with the cluster.
Set kernel parameters for user resource limits:.
- memlock: Set maximum locked-in-memory address space to unlimited.
- nofile: Allow at least 100,000 open files.
- nproc: Allow at least 32,000 processes.
- as: Set the address space to unlimited.
- vm.max_map_count: Set the maximum memory map areas per process to 1,048,575 or more.
Use a recent version of the Linux kernel for enhanced SSD support and better overall performance.
Make sure that settings persist after restart by setting them in configuration files. For example:
- Linux kernel parameters are specified in the /etc/sysctl.conf file, and they are automatically loaded from this file on restart.
- Swap is disabled through the /etc/fstab file.
- Resource limits are configured through the /etc/security/limits.conf file.

Network

Check network-related errors:

Use ifconfig to check the number of packets with errors for RX and TX rows.

Ideally, this should be between 0 and 1 percent of the total packets. Errors in more than 1 percent of the packets is a strong indicator of network issues that must be remediated.
- errors: Total number of packets received with errors, including CRC, overruns, and others. This indicates malfunctioning network hardware in the server or elsewhere in the network.
- overruns: Number of received packets that experienced FIFO overruns, which occur when a buffer is full and the kernel is unable to empty. You might need to investigate and tune the Linux kernel’s TCP parameters, as described in Operating system messages.
- carrier (TX only): Number of packets that have experienced loss of carriers. This can indicate a flapping link.
Check network latency and throughput for node-to-node connections and client-to-cluster connections.

For simple checks, use ping. For more advanced checks, use iperf, mtr, iftop, or other tools.

Inside the datacenter network, latency should not be more than 1 ms.
Make sure required ports are open and firewalls aren’t blocking communication. Specifically, check connectivity to dedicated communication ports between DSE nodes and components like DSE OpsCenter and Cassandra drivers.

Use nc -zv host port to check for existing connections to specific ports.
Check that the Linux kernel is configured with optimal TCP settings for networking.

Disks

Don’t use NAS devices for DSE data.
Don’t use traditional SAN storage for DSE data in on-premise deployments.

Virtual SAN storage is acceptable for cloud deployments only.
Check disk type (spinning or SSD) using a tool like smartctl.
Check that no errors are reported by S.M.A.R.T., such as errors reported by smartmontools or errors visible in the system logs.
Check if RAID is used.

Typically this isn’t necessary. If it is used, make sure you use a recommended RAID level.
Check disk latencies using tools like iostat or iostat-cli, and then search for output containing %iowait, avgrq-sz, avgqu-sz, await, r_await, and w_await:
- await: Average elapsed time in milliseconds for an I/O request to be computed, from start to finish. Values greater than 5ms indicate that the I/O system can’t cope with the required throughput.
- r_await and w_await: Indicate bottlenecks in read or write operations.
- avgqu-sz: Average queue length for requests issued to a given disk device. For SSD disks, this is typically less than 10. For HDDs, it can be greater than 10.
Use separate disks for DSE data and commit logs, and make sure they are separate from system disks.
If using DSE Search, use a separate disk for DSE Search data, even with SSDs.
Check that the recommended disk settings are applied, particularly read ahead.

These settings are different for SSDs and spinning disks.
Use a recommended and highly performant Extended File System format.

CPU

Disable CPU frequency scaling.
Use lscpu or check /proc/cpuinfo to determine the CPU model, number of cores, frequency, and other information.
On the systems with multiple physical CPUs, use lstopo or numactl to ensure that all PCI components are in NUMA mode.
Check /proc/interrupts for interrupts between the different CPUs, and that interrupts are not assigned to the same CPU. This can lead to overloading of a specific CPU. If necessary, tune the SMP affinity for specific interrupt requests.
Check CPU stats reported by Linux tools such as iostat and vmstat.

Large amounts of steal time or CPU-ready time can indicate that your virtual machine is over-allocated, you have noisy neighbors (other services that are running on the same server as your virtual machine), or both.

Operating system messages

Run dmesg -T to get system messages with a human-readable timestamp, and then search for any entries about out-of-memory (OOM) killer activity, segfaults, disk problems, network issues (typically TCP SYN flood), and other related issues. For example:

For OOM killer, search for messages formatted like Out of memory: Killed process NNN, and then check if the errors were caused by a DSE process.
For process segfaults, search for messages formatted like name[proc_id]: segfault at, and then check if the errors were caused by a DSE process.
For TCP SYN flood, search for messages formatted like TCP: Possible SYN flooding on port NNN, and then refer to the documentation for your specific Linux distribution to resolve the issue. For example, see kernel: Possible SYN flooding on port. Pay particular attention to messages with ports belonging to DSE.
For disk problems, search for messages formatted like disk_id: failed command or disk_id: exception where disk_id is a device identifier like ata7.00, depending on the disk type.

`/tmp` directory must be executable

The /tmp directory must be executable because it is used for extraction and loading of the native code. For more information, see Secure the /tmp directory.

Define SLAs

For any application, you must calculate and set Service Level Agreements (SLAs) that define acceptable performance thresholds and failure tolerance.

This guide focuses on SLAs for DSE clusters. A full explanation of SLAs is beyond the scope of this document.

SLAs are measured in percentiles. However, this doesn’t mean that the highest percentiles are the best for evaluating SLAs.

For example, when measuring request latency, use the following guidelines for SLA percentile targets:

Latency percentiles for requests
Percentile	Recommendation
100	Not recommended because it isn’t meaningful in practice. This percentile is often irrelevant to testing and based on factors external to the database.
99.9	An aggressive target that is best for latency-sensitive workloads only. Typically requires high-performance hardware and careful tuning.
99	The most common target for SLAs in practice. It represents a reasonable balance between performance and hardware/tuning for most workloads.
95	A relaxed target that is best for workloads where latency isn’t noticed or less critical. Not recommended for latency-sensitive workloads, such as high-traffic, user-facing interfaces.

Your cluster’s performance statistics and metrics that inform SLAs, such as latency and throughput, are influenced by the underlying hardware and configurations, in addition to other factors like data modeling and application logic. For example, the following metrics are based on clusters with more data on disk than RAM. The measurement was in the 99th percentile using nodetool tablehistograms.

Impact of hardware on latency
Latency (99p)	Hardware
Less than `10ms`	Premium, high-performance CPU nodes with NVM flash storage
Less than `150ms`.	Common SATA based SSDs
Over `500ms`	Common SATA based SSDs

Latency metrics vary significantly due to factors like controller, contention, and data model. Tuning can shift this metric by an order of magnitude. This can result in some hardware approaching NVM specifications.

Enable monitoring and collect metrics

Monitoring DataStax Enterprise (DSE) clusters is essential for detecting, investigating, and resolving issues in your clusters. This includes statistics regarding database activity, client applications, and the operating system.

JMX and observability

Cassandra uses Java Management Extensions (JMX) tools for cluster monitoring and some management tasks. You can access JMX directly and through tools like nodetool and jmxterm. For more information about JMX, MBeans, and supported observability tools, see Monitor DataStax Enterprise (DSE) clusters.

Important metrics for cluster testing

DSE exposes many keyspace, table, and subsystem metrics for observation and analysis. To help you get started with cluster performance monitoring, this guide focuses on the metrics that are typically considered the most valuable for cluster initialization and testing. Most categories include some related MBeans, recommended thresholds, and other guidance for investigating potential performance issues.

These are general recommendations. You might need to change thresholds or monitor different metrics depending on your environment and workloads.

For more information about MBeans and available metrics, see Monitor DataStax Enterprise (DSE) clusters.

All JMX objects are found at the MBean path org.apache.cassandra.metrics. For example, you can retrieve global write latency at org.apache.cassandra.metrics:type=ClientRequest,scope=Write,name=Latency.

Client requests

Measures how the system performs from the point of view of the client application. Important metrics include:

Coordinator-level latency for read operations, especially 95th and 99th percentiles.
Coordinator-level latency for write operations, especially 95th and 99th percentiles.
Pending (queued) writes.
Number of client connections.

Examples of JMX MBeans:

Coordinator-level read and write latency (global):

org.apache.cassandra.metrics:type=ClientRequest,scope=Write,name=Latency
org.apache.cassandra.metrics:type=ClientRequest,scope=Read,name=Latency

Alert when 99th percentile latency is above 200 ms for more than 1 minute.

Number of pending writes (global):
```
org.apache.cassandra.metrics:type=ClientRequest,name=viewPendingMutations
```
Alert if pending writes exceeds 10,000 for more than a short period of time. This can indicate an issue with tuning or sizing.

Active client connections (global):

org.apache.cassandra.metrics:type=Client,name=connectedNativeClients
org.apache.cassandra.metrics:type=Client,name=connectedThriftClients

There are many other metrics related to read and write operations to get a complete picture of query performance. For example, see Collecting slow queries.

Threadpools (TP)

Measures data processing and task throughput, such as compaction and flushing of data. Important metrics include:

Number of blocked threads, such as memtable flush writer and memtable pool allocations.
Number of aborted threads, such as aborted compactions.
Number of pending threads, such as pending compactions and pending flushes.
Threads-per-core (TPC).

Examples of JMX MBeans:

Number of pending compaction tasks (global, keyspace, or table):
```
org.apache.cassandra.metrics:type=Compaction,name=PendingTasks
org.apache.cassandra.metrics:type=Table,keyspace=ks,scope=*,name=PendingCompactions
```
The number of acceptable pending compactions depends on the type of compaction strategy:
- UnifiedCompactionStrategy (UCS): Depends on mode.
- SizedTieredCompactionStrategy (STCS): Should not exceed 100 for more than short periods.
- LeveledCompactionStrategy (LCS): Not an accurate measure for pending compactions. Staying over 100 is generally an indication of stress. Over 1000 is pathological.
- TimeWindowCompactionsStrategy (TWCS): Similar to STCS.
- DateTieredCompactionStrategy (DTCS): Deprecated. Replace with TWCS.

Number of blocked compaction tasks (global):

org.apache.cassandra.metrics:type=ThreadPools,path=internal,scope=CompactionExecutor,name=CurrentlyBlockedTasks

Alert when this value is greater than 0.

Number of aborted compaction tasks (global):
```
org.apache.cassandra.metrics:type=Compaction,name=CompactionsAborted
```
Alert when this value is greater than 0.
Number of threads blocked by memtable allocation (global):
```
org.apache.cassandra.metrics:type=MemtablePool,name=BlockedOnAllocation
```
Alert when this value is greater than 0.
Number of blocked memtable flush writer tasks (global):
```
org.apache.cassandra.metrics:type=ThreadPools,path=internal,scope=MemtableFlushWriter,name=CurrentlyBlockedTasks
```
Alert when this value is greater than 0.

Always investigate these alerts because this condition can lead to severe performance degradation for writes.

This condition has many causes, including failing disks and excessive disk operations.

Number of pending flushes (global):

org.apache.cassandra.metrics:type=Table,name=PendingFlushes

Number of commit log segments waiting on commit:
```
org.apache.cassandra.metrics:type=CommitLog,name=WaitingOnCommit,name=Count
org.apache.cassandra.metrics:type=CommitLog,name=WaitingOnCommit,name=99thPercentile
```
Alert when this value sustains a significant value, such as 30, for more than 1 minute. The acceptable threshold depends on your environment and workloads.

Individual tables

Measure performance of individual tables.

At minimum, monitor your most important tables. Monitoring every metric for all tables can be noisy and reduce visibility of alerts on important tables.

Important metrics include:

Partition size.
Number of SSTables overall.
Number of SSTables read per request.
Number of tombstones scanned during read request.
Coordinator-level read and write latencies.

Examples of JMX MBeans:

Coordinator-level read and write latency (keyspace or table):

org.apache.cassandra.metrics:type=Table,keyspace=ks,scope=*,name=ReadLatency
org.apache.cassandra.metrics:type=Table,keyspace=ks,scope=*,name=WriteLatency

Alert when the 99th percentile exceeds 200 ms for more than 1 minute.

Maximum partition size in bytes (global, keyspace, or table):
```
org.apache.cassandra.metrics:type=Table,name=MaxPartitionSize
org.apache.cassandra.metrics:type=Table,keyspace=ks,scope=*,name=MaxPartitionSize
```
Alert when partitions are greater than 100 MB.

Development teams must investigate these alerts because they indicate data model issues.
Number of SSTables (global, keyspace, or table):
```
org.apache.cassandra.metrics:type=Table,name=LiveSSTableCount
org.apache.cassandra.metrics:type=Table,keyspace=ks,scope=*,name=LiveSSTableCount
```
Alert when there are more than 200 SSTables per table, depending on the compaction strategy.

Too many large tables can cause performance degradation.

Inter-cluster communication

Measures how data exchange happens in the cluster, including replication and hinted handoff. Important metrics include:

Number of dropped mutations and other messages.
Total number of timeouts, and timeouts per host.
Cross-datacenter latency.
System hints (number of hints on disk).
Hint replay (number of failed and timed-out hint messages).

Examples of JMX MBeans:

Inter-node latency (global or datacenter):

org.apache.cassandra.metrics:type=Messaging,name=CrossNodeLatency
org.apache.cassandra.metrics:type=Messaging,name=DATACENTER_NAME-Latency

For datacenter-specific metrics, the name must include the datacenter name.

Number of dropped mutations (global, keyspace, or table):

org.apache.cassandra.metrics:type=Table,name=DroppedMutations
org.apache.cassandra.metrics:type=Table,keyspace=ks,scope=*,name=DroppedMutations

Alert when this value is greater than zero.

Number of timeouts (global or node):
```
org.apache.cassandra.metrics:type=MessagingService,name=TotalTimeouts
org.apache.cassandra.metrics:type=MessagingService,name=TimeoutsPerHost
```
Alert when this number increases significantly within a 5 or 15 minute window.

This can indicate network problems and similar issues.
Number of hints stored on individual node:
```
org.apache.cassandra.metrics:type=Storage,name=TotalHints
```
Alert when this value is greater than zero.

Any non-zero value indicates that some nodes aren’t avaiable.

By design, metrics like total hints and system generated hints increase whenever a node is down or a datacenter link is down. These values should decrease after reconnecting.

If generated system hints don’t decrease for a long time, investigate why system hints aren’t being replayed successfully.

Hint replay rates:

org.apache.cassandra.metrics:type=HintsService,name=HintsSucceeded
org.apache.cassandra.metrics:type=HintsService,name=HintsFailed
org.apache.cassandra.metrics:type=HintsService,name=HintsTimedOut

Java Virtual Machine (JVM)

Measures memory usage and task execution in the JVM. Important metrics include:

Amount of memory used.
Duration of garbage collection pauses.

If heap usage shows a pattern of huge spikes with long pauses, adjust the heap.

Examples of JMX MBeans:

Information about Java’s garbage collection caused by Mac GC Elapsed and similar:
```
org.apache.cassandra.metrics:type=GCInspector
```

Operating system and hardware

Measures the utilization and availability of system resources on the node. Important metrics include:

CPU usage on the node.
Amount of disk space available.

I/O wait metrics greater than 5 indicate that the CPU is waiting too long on I/O and is a sign of being I/O bound.

For CPUs that use hyper-threading, keep in mind that for each physical processor core, the operating system addresses two virtual (logical) cores, and then shares the workload between them. This means that if half your threads are at 100% CPU usage, then the CPU capacity is at maximum, even if the tool reports 50% usage. To see the actual thread usage, use the ttop command.

Log warnings and errors

Either through metrics or log files, monitor for the following strings that you might need to investigate:

SlabPoolCleaner
WARN
ERROR
GCInspector
hints

Other SLA-related metrics

In addition to the previously mentioned metrics, make sure you monitor all metrics related to your SLAs.

Simulate production workloads

The hardware used for load testing should mimic the parameters and number of nodes in the production environment. Performing load testing on a single node, or in environments where the number of nodes is equal to the replication factor, results in incorrect measurements.

In a test environment, run simulated production workloads, evaluating your cluster’s performance at each stage. If your cluster performs poorly or reports errors, investigate the issues, reconfigure your cluster, and then rerun the tests until your cluster meets your desired performance targets.

Make sure your tests cover a range of scenarios:

Reads and writes: Common queries, uncommon queries, prepared statements, BATCH, LTWs, and known slow queries, such as SELECT *.
Database operations: nodetool commands, memtable flushes, compactions.
Uncommon or undesirable events: Outages, bootstrapping new and replacement nodes, repairs, disaster recovery, and failures.

There are many ways to run these tests. For example, you can create test scripts that use a Cassandra driver or cqlsh to run read/write operations against the cluster. Compared with production applications, you might need to include extended logging, tracing, or tailing to capture more detailed information about the cluster’s behavior during the tests.

For each of the following scenarios, questions and key metrics are provided to help you evaluate cluster performance:

Simulate average read/write traffic.

Some level of I/O must continue throughout all tests to adequately measure node performance under baseline I/O stress in addition to each test scenario. Maximum throughput under ideal circumstances isn’t a realistic measurement of cluster performance. As you perform the remaining tests, you can understand how each scenario impacts performance. Then, you can adjust your performance targets or reconfigure your cluster and repeat the tests until your cluster performs as desired.
Simulate production outages:
1. For small clusters, take one or two nodes offline. For large clusters, take enough nodes offline that the outage is impactful and measurable in testing.
  
  Do your queries start failing?
  
  Do the nodes start working without any change when they come back online?
2. Take a datacenter offline while processing reads and writes.
  
  How long can the cluster be offline before system.hints build up to unmanageable levels?
Simulate operations and queries that can stress the cluster and overload nodes, potentially taking nodes offline or occupying them for long periods of time.

These tests help you understand your clusters resilience to stress, malformed or suboptimal queries, and other workload spikes.
1. Run resource-intensive tasks that use CPU on multiple nodes, such as cluster-wide compactions, schema changes, or memtable flushes.
2. Run resource-intensive range queries, such as SELECT * and other known slow, inefficient, or long-running queries.
  
  For example, the following query reads up to 100,000,000 rows from a given table without paging:
  PAGING OFF; SELECT * FROM keyspace1.table1 LIMIT 100000000;
Simulate production operations during average read/write load:
1. Run repair to ensure that you have enough overhead to complete a repair.
2. Bootstrap a new node into the cluster.
  
  Note how long it takes to bootstrap the new node.
3. Add a datacenter.
4. Change the schema by adding new tables, removing tables, and altering existing tables.
  
  If you tested schema changes in the previous steps, make sure you test different types of changes. For example, altering one column might not have the same impact as removing an entire table.
5. Replace a failed node using the simulated outage scenarios described in the previous steps.
  
  Note how long it takes to bootstrap the replacement node.
6. Repeat these tests during heavy read/write load to simulate high traffic periods.
7. If you haven’t done so already, repeat these tests during the outage and overload scenarios in the previous steps.
These tests ensure that your cluster can tolerate normal and abnormal database operations under normal and heavy I/O load. This is also a good opportunity to practice these procedures and events in a non-production environment.
Establish transactions per second (TPS) targets.

Determine how many TPS you expect your cluster to handle. Ideally, align this number with the node count in your test cluster. Alternately, test with a ratio of TPS to nodes to evaluate how the cluster behaves.

If the cluster does not meet your SLA requirements, adjust your targets or look for the root cause. For example, does the cluster meet SLAs if the TPS level is lower?
Run a real replication factor on at least 5 nodes.

If you plan to run more than 5 nodes in production, run the same number of nodes that you expect to run in production.

If you plan to run 3 nodes in production, test with 5 nodes. You might find that some queries that were faster with 3 nodes are slower with 5 nodes. For example, secondary indexes are often slower at scale.

Use the same replication strategy that you plan to use in production.
Simulate a production datacenter setup.

Each datacenter adds additional write costs. If your production cluster has more datacenters than your test cluster, your writes might be greater than you anticipate. For example, if your replication factor is 3, and your test cluster has 2 datacenters, each write goes to 4 nodes in 2 datacenters. If your production cluster uses a replication factor of 3 with 5 datacenters, each write goes to 7 nodes with 5 datacenters. This means production writes are almost twice as expensive on the coordinator node because, in each remote datacenter, the forwarding coordinator sends the data to the other replicas in its datacenter.
Simulate your network topology with the same latency and pipe.

Test with your expected TPS.

Be aware that remote links can be a limiting factor for many use cases and expected bandwidth.
Simulate expected node density.

Once you’ve determined your target per-node density, add data to the test cluster until the target capacity is surpassed. Observe the following as you approach, reach, and surpass the target density:
- What happens to your SLAs when approaching the target density?
- What happens when surpassing the target density?
- How long does bootstrap take with maximum or excess density?
- How long does repair take with maximum or excess density?

Test with NoSQLBench

NoSQLBench is an open-source testing tool that emulates real application workloads in your development environment. This means that you can run performance, sizing, and data model tests without writing your own testing harness or requiring live production workloads.

With NoSQLBench, you can:

Run common testing workloads directly from the command line.
Generate virtual datasets of arbitrary size, with deterministic data and statistically shaped values.
Design custom workloads that emulate your application, contained in a single file, based on statement templates with no IDE or coding required.
Plot your results in a Docker and Grafana stack on Linux with a single command line option.
Optionally customize the runtime behavior of NoSQLBench for advanced testing, including the use of a JavaScript environment.

For more information, see the following:

Test DSE Search

The solr_stress demo is a benchmark tool that simulates the number of requests for reading and writing data on a DSE Search cluster using a specified sequential or random set of iterations. See Use the solr_stress demo to test DSE Search performance.

Test your DataStax Enterprise (DSE) cluster

Prerequisites

Run system checks

Operating system configuration

Network

Disks

CPU

Operating system messages

`/tmp` directory must be executable

Define SLAs

Enable monitoring and collect metrics

JMX and observability

Important metrics for cluster testing

Simulate production workloads

Test with NoSQLBench

Test DSE Search

See also

Was this helpful?

Give Feedback