Capacity planning and hardware selection for DataStax Enterprise implementations
General guidelines
Follow these guidelines when choosing hardware for your DataStax database:
-
Hardware choices depend on your particular use case. The right balance of CPUs, memory, disks, number of nodes, and network are vastly different for environments with static data that are accessed infrequently than for volatile data that is accessed frequently.
-
The suggested guidelines are the minimum required. You may need to increase CPU capacity, memory, and disk space beyond the recommended minimums.
-
Be sure to read Anti-patterns for important information about SAN storage, NAS devices, and NFS.
-
Thoroughly test your configuration before deployment.
|
In addition, for optimal performance in Linux environments, DataStax recommends using the most recent version of the Linux operating system. Newer versions of Linux handle highly concurrent workloads more efficiently. |
CPUs
|
DSE 6.x was optimized to scale to a high number of CPUs. For some workloads, a cluster with 8 CPUs per node does not perform optimally. Instead, to achieve the best performance, DataStax strongly recommends using a smaller number of nodes, but with each node having 16 CPUs; in effect providing at least the same total number of CPUs across your cluster. |
Insert-heavy workloads are CPU-bound in DSE before becoming memory-bound. All writes go to the commit log, but the database is so efficient in writing that the CPU is the limiting factor. The DataStax database is highly concurrent and uses as many CPU cores as available. Recommendations:
-
Minimum dedicated hardware per node for production: 16 CPU cores (logical).
-
Dedicated hardware in development in non-loading testing environments: 2 CPU cores (logical) are sufficient.
-
DSE Graph: graph queries can be a CPU bottleneck due to query optimization and result set preparation. Therefore more CPU cores or more powerful CPU cores can be beneficial.
-
Production scaling:
-
The most effective way to scale DataStax Enterprise is to add more servers.
-
DSE 6.x can benefit from more CPU cores with its thread-per-core architecture.
-
DSE 6.x makes more effective use of IO than earlier version. Therefore it benefits from a faster storage layer.
-
Memory
The more memory a DataStax Enterprise (DSE) node has, the better read performance. More RAM also allows memory tables (memtables) to hold more recently written data. Larger memtables lead to a fewer number of SSTables being flushed to disk, more data held in the operating system page cache, and fewer files to scan from disk during a read. The ideal amount of RAM depends on the anticipated size of your hot data.
Recommended minimum memory for dedicated hardware and virtual environments
| Node type | System memory | Heap |
|---|---|---|
Production |
||
Transactional (Apache Cassandra™) |
32 GB |
8 GB |
DSE Analytics |
32 GB to 512 GB |
|
DSE Search |
32 GB to 512 GB |
|
Vector Search |
32 GB to 512 GB |
|
DSE Graph |
Add 2 to 4 GB to your particular combination of DSE Search or DSE Analytics. For a large dedicated graph cache, add more RAM. |
|
Development (non-load testing environments) |
||
Any |
8 GB |
4 GB |
Any |
16 GB |
8 GB |
Spinning disks versus solid state drives (local only)
|
For cloud deployments, contact the DataStax Services team. |
Solid state drives (SSDs) are recommended for all DataStax Enterprise nodes. The NAND Flash chips that power SSDs provide extremely low-latency response times for random reads while supplying ample sequential write performance for compaction operations. In recent years, drive manufacturers have improved overall endurance, usually in conjunction with spare (unexposed) capacity. Additionally, because PBW/DWPD ratings are probabilistic estimates based on worst case scenarios, such as random write workloads, and because the database does only large sequential writes, drives significantly exceed their endurance ratings. However, it is important to plan for drive failures and have spares available. A large variety of SSDs are available from server vendors and third-party drive manufacturers.
Longevity is a key factor when purchasing SSDs. The best recommendation is to make the decision based on how difficult it is to change drives when they fail, not on workload of the drive. Remember, your data is protected because the database replicates data across the cluster. Buying strategies include:
-
If drives are quickly available, buy the cheapest drives that provide the performance you want.
-
If it is more challenging to swap the drives, consider higher endurance models, possibly starting in the mid range, and then choose replacements of higher or lower endurance based on the failure rates of the initial model chosen.
For additional help in determining the most cost-effective option for a given deployment and workload, contact the DataStax Services team.
Disk space
Disk space depends on usage, so it is important to understand the mechanism. The database writes data to disk when appending data to the commit log for durability and when flushing memtables to SSTable data files for persistent storage. The commit log has a different access pattern (read/writes ratio) than the pattern for reading data from SSTables. This is more important for spinning disks than for SSDs.
SSTables are periodically compacted. Compaction improves performance by merging and rewriting data and discarding old data. However, depending on the type of compaction and size of the compactions, disk utilization and data directory volume temporarily increases during compaction. For this reason, be sure to leave an adequate amount of free disk space available on a node.
The following table provides guidelines for the minimum disk space requirements based on the compaction strategy:
|
DateTieredCompactionStrategy (DTCS) is deprecated in Cassandra 3.0.8/3.8. |
| Compaction strategy | Minimum requirements | ||
|---|---|---|---|
UnifiedCompactionStrategy (UCS) |
|
||
SizeTieredCompactionStrategy (STCS) |
Sum of all the SSTables compacting must be smaller than the remaining disk space.
|
||
LeveledCompactionStrategy (LCS) |
Generally 10%. Worse case: 50% when the Level 0 backlog exceeds 32 SSTables (LCS uses STCS for Level 0). |
||
TimeWindowCompactionStrategy (TWCS) |
TWCS requirements are similar to STCS. TWCS requires a maximum disk space overhead of approximately 50% of the total size of SSTables in the last created bucket. To ensure adequate disk space, determine the size of the largest bucket or window ever generated and calculate 50% of that size. |
Additional resources:
Estimating usable disk capacity
To estimate how much data your nodes can hold, calculate the usable disk capacity per node and then multiply that by the number of nodes in your cluster. For a production cluster, DataStax recommends separating the commit log and data directories on different disks.
-
Start with the raw capacity of the physical disks:
raw_capacity = disk_size * number_of_data_disks -
Calculate the usable disk space accounting for file system formatting overhead (roughly 10 percent):
formatted_disk_space = (raw_capacity * 0.9) -
Calculate the recommended working disk capacity:
usable_disk_space = formatted_disk_space * (0.5 to 0.8)
During normal operations, the database routinely requires disk capacity for compaction and repair operations. For optimal performance and cluster health, DataStax recommends not filling your disks to capacity, but running at 50% to 80% capacity. See compaction strategy and size of the compactions.
Minimum Disk Space Recommendations
|
Thoroughly test before deploying to production. DataStax highly recommends testing with tools such as NoSQLBench or cassandra-stress tool with your desired configuration. Be sure to test common administrative operations, such as bootstrap, repair, and failure, to make certain your hardware selections meet your business needs. See Testing Your Cluster Before Production. |
For DSE Search, also test using solr_stress.
DataStax Enterprise capacity policy
The following per Node data capacity limitations apply:
-
For DataStax Enterprise version 6.0 and earlier: 1TB per node
-
For DataStax Enterprise version 6.7 and later: 2TB per node
If you have questions regarding node data capacity limitations, contact your DataStax account team.
Exceeding this data capacity limitations may have the following effects.
-
Depending on write throughput, hardware, and compaction strategy, compactions may get behind.
-
Streaming operations such as bootstrap, repair, and replacing nodes may take longer to complete.
Higher capacity nodes work best with low to moderate write throughput and no indexing.
One special case is for time-series data. Use the TimeWindowCompactionStrategy to scale larger than these limits if:
-
the time-series data is written once and never updated.
-
the data has a clustering column that is time based.
-
Readqueries cover specific time-bounded ranges of data rather than its full history. -
the TWCS windows are configured appropriately.
Capacity per node (node density)
Node capacity is highly dependent on the environment. Determining node density depends on many factors, including:
-
DataStax has tested and certified densities greater than 1TB per node for only DSE 6.7 and later.
-
Data frequency change and access frequency.
-
Using HDDs or SSDs.
-
Storage speed and whether the storage is local.
-
SLAs (service-level agreements) and ability to handle outages.
-
Data compression.
-
Compaction strategy: choice of compaction strategy depends on whether the workload is write-intensive, read-intensive, or time dependent. See [Disk Space].
-
Network performance: remote links likely limit storage bandwidth and increase latency.
-
Replication factor: See About Data Distribution and Replication.
Capacity and I/O
When choosing disks for your nodes, consider both capacity (how much data you plan to store) and I/O (the write/read throughput rate). Some workloads are best served by using less expensive SATA disks and scaling disk capacity and I/O by adding more nodes (with more RAM).
Number of disks - HDD
DataStax recommends using at least two disks per node: one for the commit log and the other for the data directories. At a minimum, the commit log should be on its own partition.
Commit log disk - HDD
The disk need not be large, but it should be fast enough to receive all of your writes as appends (sequential I/O).
Commit log disk - SSD
Unlike spinning disks, there is less of a penalty for sharing commit logs and data directories on SSD than there is on HDD. DataStax recommends separating commit logs and data for highest performance and resiliency.
Data disks
Use one or more disks per node and make sure they are large enough for the data volume and fast enough to both satisfy reads that are not cached in memory and to keep up with compaction.
DSE Search - SSD
Because DSE Search is very IO intensive, transactional data and search data must be on different SSDs. Otherwise, the SSD can be overrun from both workloads.
RAID on data disks
It is generally not necessary to use a Redundant Array of Independent Disks (RAID) for the following reasons:
-
Data is replicated across the cluster based on the replication factor you have chosen.
-
DataStax Enterprise includes a JBOD (Just a bunch of disks) feature for disk management. Because the database responds according to your availability/consistency requirements to a disk failure either by stopping the affected node or by denylisting the failed drive, you can deploy nodes with large disk arrays without the overhead of RAID 10. You can configure the database to stop the affected node or denylist the drive according to your availability/consistency requirements. Also see Recovering from a single disk failure using JBOD.
RAID on the commit log disk
Generally RAID is not needed for the commit log disk.
Replication adequately prevents data loss.
If you need extra redundancy, use RAID 1.
Extended file systems
DataStax recommends deploying on XFS or ext4.
On ext2 or ext3, the maximum file size is 2TB even using a 64-bit kernel.
On ext4 the filesystem limitations is 16TB and not to be confused with DSE disk limits.
Because the database can use almost half your disk space for a single file when using SizeTieredCompactionStrategy (STCS), use XFS when using large disks, particularly if using a 32-bit kernel.
XFS file size limits are 16TB max on a 32-bit kernel, and essentially unlimited on 64-bit.
|
Per DSE node data density should not be confused with the Extended File Systems limitations. See DataStax Enterprise capacity policy for the per node data density requirements. |
Estimating partition size
For efficient operation, partitions must be sized within certain limits.
Two measures of partition size are the number of values in a partition and the partition size on disk.
The practical limit of cells per partition is 2 billion.
Sizing the disk space is more complex, and involves the number of rows and the number of columns, primary key columns and static columns in each table.
Each application has different efficiency parameters, but a good rule-of-thumb is to keep the number of rows per partition below 100,000 items and the partition size under 100 MB.
Network
Minimum recommended bandwidth: 1000 Mb/s (gigabit).
A distributed data store puts load on the network to handle read/write requests and replication of data across nodes. Be sure that your network can handle inter-node traffic without bottlenecks. DataStax recommends binding your interfaces to separate Network Interface Cards (NIC). You can use public or private NICs depending on your requirements.
The database efficiently routes requests to replicas that are geographically closest to the coordinator node and chooses a replica in the same rack when possible. The database always chooses replicas located in the same datacenter over replicas in a remote datacenter.
Firewall
If using a firewall, make sure that nodes within a cluster can communicate with each other. See Securing DataStax Enterprise ports.
