Implementation or design patterns that are ineffective and/or counterproductive in Apache Cassandra production installations. Correct patterns are suggested in most cases.
- Storage area network
- Network attached storage
- Shared network file systems
- Excessive heap space size
- Cassandra's rack feature in single-token architecture deployments
- SELECT ... IN or index lookups
- Using the Byte Ordered Partitioner
- Reading before writing
- Load balancers
- Insufficient testing
- Too many keyspaces or tables
- Lack of familiarity with Linux
SAN storage is not recommended for on-premises deployments.
- SAN ROI (return on investment) does not scale along with that of Apache Cassandra, in terms of capital expenses and engineering resources.
- In distributed architectures, SAN generally introduces a bottleneck and single point of failure because Cassandra's IO frequently surpasses the array controller's ability to keep up.
- External storage, even when used with a high-speed network and SSD, adds latency for all operations.
- Heap pressure is increased because pending I/O operations take longer.
- When the SAN transport shares operations with internal and external Cassandra traffic, it can saturate the network and lead to network availability problems.
Taken together these factors can create problems that are difficult to resolve in production. In particular, new users deploying Cassandra with SAN must first develop adequate methods and allocate sufficient engineering resources to identify these issues before they become a problem in production. For example, methods are needed for all key scaling factors, such as operational rates and SAN fiber saturation.
Storing SSTables on a network attached storage (NAS) device is not recommended. Using a NAS device often results in network related bottlenecks resulting from high levels of I/O wait time on both reads and writes. The causes of these bottlenecks include:
- Router latency.
- The Network Interface Card (NIC) in the node.
- The NIC in the NAS device.
If you are required to use NAS for your environment, contact the Cassandra community.
It is recommended to use the default heap space size for most use cases. Exceeding this size can impair the Java virtual machine's (JVM) ability to perform fluid garbage collections (GC). The following table shows a comparison of heap space performances reported by a Cassandra user:
|Heap||CPU utilization||Queries per second||Latency|
|40 GB||50%||750||1 second|
|8 GB||5%||8500 (not maxed out)||10 ms|
For information on heap sizing, see Tuning Java resources.
This information applies only to single-token architecture, not to virtual nodes. See Data distribution.
Defining one rack for the entire cluster is the simplest and most common implementation. Multiple racks should be avoided for the following reasons:
- Most users tend to ignore or forget rack requirements that racks should be organized in an alternating order. This order allows the data to get distributed safely and appropriately.
- Many users are not using the rack information effectively. For example, setting up with as many racks as nodes (or similar non-beneficial scenarios).
- Expanding a cluster when using racks can be tedious. The procedure typically involves several node moves and must ensure that racks are distributing data correctly and evenly. When clusters need immediate expansion, racks should be the last concern.
To use racks correctly:
- Use the same number of nodes in each rack.
- Use one rack and place the nodes in different racks in an alternating pattern. The rack feature of Cassandra benefits from quick and fully functional cluster expansions. Once the cluster is stable, you can swap nodes and make the appropriate moves to ensure that nodes are placed in the ring in an alternating fashion with respect to the racks.
Also see Data replication.
SELECT ... IN and index lookups (formerly secondary indexes) should be avoided except for specific scenarios. See When not to use IN in SELECT and When not to use an index in Indexing in CQL for Cassandra 2.2.
The Byte Ordered Partitioner (BOP) is not recommended.
Use virtual nodes. Vnodes allow each node to own a large number of small ranges distributed throughout the cluster. Using vnodes saves you the effort of generating tokens and assigning tokens to your nodes. If not using vnodes, these partitioners are recommended because all writes occur on the hash of the key and are therefore spread out throughout the ring amongst tokens range. These partitioners ensure that your cluster evenly distributes data by placing the key at the correct token using the key's hash value.
Cassandra was designed to avoid the need for load balancers. Putting load balancers between Cassandra and Cassandra clients is harmful to performance, cost, availability, debugging, testing, and scaling. All high-level clients, such as the Java and Python drivers for Cassandra, implement load balancing directly. For available drivers, see DataStax drivers
Be sure to test at scale and production loads. This the best way to ensure your system will function properly when your application goes live. The information you gather from testing is the best indicator of what throughput per node is needed for future expansion calculations.
Each Cassandra keyspace has a certain amount of overhead space that uses JVM memory. Each table uses approximately 1 MB of memory. For example, 3,500 tables would use about 3.5 GB of JVM memory. Using too many tables, or by extension, too many keyspaces will bloat the memory requirements. A good rule of thumb is to keep the number of tables within a cluster to 1,000 at most, and aim for 500 or less.
Linux has a great collection of tools. Become familiar with the Linux built-in tools. It will help you greatly and ease operation and management costs in normal, routine functions. The essential list of tools and techniques to learn are:
- Parallel SSH and Cluster SSH: The pssh and cssh tools allow SSH access to multiple nodes. This is useful for inspections and cluster wide changes.
- Passwordless SSH: SSH authentication is carried out by using public and private keys. This allows SSH connections to easily hop from node to node without password access. In cases where more security is required, you can implement a bastion host and/or VPN.
- Useful common command-line tools include:
- dstat: Shows all system resources instantly. For example, you can compare disk usage in combination with interrupts from your IDE controller, or compare the network bandwidth numbers directly with the disk throughput (in the same interval).
- top: Provides an ongoing look at CPU processor activity in real time.
- System performance tools: Tools such as iostat, mpstat, iftop, sar, lsof, netstat, htop, vmstat, and similar can collect and report a variety of metrics about the operation of the system.
- vmstat: Reports information about processes, memory, paging, block I/O, traps, and CPU activity.
- iftop: Shows a list of network connections. Connections are ordered by bandwidth usage, with the pair of hosts responsible for the most traffic at the top of list. This tool makes it easier to identify the hosts causing network congestion.
Be sure to use the recommended settings.