Important configuration changes in Cassandra 4.x

Auditing is designed to meet regulatory and security requirements, and is able to audit more types of requests and interactions with the database than Full Query Logging (FQL) or Change Data Capture (CDC) can.

Auditing and FQL have been implemented on a shared design that uses the OpenHFT libraries. By default the chronicle queue library of OpenHFT is used as a binary log (binlog), making the default behavior file-based.

The implementation is file (binlog) based. Different OpenHFT plug-in libraries can be used, the choice of which will influence operability and performance.

There are performance implications involved when enabling Auditing. This depends on the auditing setup and configuration. You should perform testing and benchmarking to evaluate production implications.

Auditing is designed to meet regulatory and security requirements, and is able to audit more types of requests and interactions with the database than Full Query Logging (FQL) or Change Data Capture (CDC) can. Because the use of Auditing may be a hard requirement for many, it’s recommended to perform thorough testing and benchmarking prior to deployment in production.

Auditing will still have a performance impact on production clusters, and file-based logging needs to be configured to be operationally safe and sustainable. Ensure appropriate metrics monitoring and alerts are in place around the log files. To learn more about this feature, read this blog post.

Prior to Cassandra 4.0, authorization via user or role was all or nothing with no concept of datacenters. Starting in version 4.0, Cassandra operators can permit and restrict connections from client applications to specific datacenters.

Authorization by datacenter utilizes the existing Authorization feature in Cassandra, and its concept of Roles. This only prevents the connections to coordinators in disallowed datacenters. Data from remote datacenters can still be accessed through a local coordinator. Server-side control of connections to coordinators is important to prevent saturation of network and available connections. This can lead to client applications being unable to connect to the cluster. This is a common problem with analytics datacenters where clients, like Apache Spark, can spawn hundreds or thousands of connections.

The use of datacenter authorization is recommended when there is a need to control and limit connections to coordinators in specific data centers. This is considered to be a safe feature to use immediately where there is benefit, as it touches very few components of the Cassandra codebase. Though, it’s not recommended to rely on this feature to restrict access to data only replicated to certain datacenters. To learn more about this feature, read this blog post.

Starting in version 4.0, the cassandra.yaml configuration file contains a new setting to automatically upgrade SSTables to the latest format after upgrading Cassandra: automatic_sstable_upgrade. It provides an alternative to manually running the upgrade command via nodetool This feature is optional and disabled by default.

SSTable data must be upgraded as part of any upgrade from Cassandra 3.x to 4.x, as the table format has changed. When the automatic_sstable_upgrade setting is enabled and set to true, a Cassandra node will automatically start the process to convert SSTables from the 3.x format to the 4.x format. This process begins after starting Cassandra 4.x for the first time on an upgraded node.

While this feature provides a convenience when upgrading nodes in the cluster, care needs to be taken when using it. Operators of a Cassandra cluster trade off control over the execution of the process for the convenience of never having to manually execute it.

SSTables are stored in sorted order, which should keep the CPU and disk IO usage relatively low. However, too many concurrent executions of the SSTable upgrade process may overwhelm the cluster. If this feature is enabled, upgrading a node may be delayed until the SSTable upgrade on other nodes completes.

Leaving this feature is disabled allows the SSTable format upgrade to be performed in a controlled manner. Specifically, after upgrading all nodes in the cluster to version 4.x, SSTable upgrade can be manually started on nodes individually. This allows for a scenario where the SSTable upgrade is launched based on the health of the cluster. If the cluster is under no load, multiple executions can be started. If the cluster is under load, then it can be run on each node in turn (one node at a time).

It’s generally recommended to leave this feature disabled and to upgrade the SSTables of a node manually.

Full Query Logging (FQL) logs all requests to the CQL interface. FQL can be used for debugging, performance benchmarking, testing, and also for auditing CQL queries. In comparison to audit logging, FQL is dedicated to CQL requests only and comes with features such as the FQL Replay and FQL Compare that are unavailable in audit logging.

Full Query Logging’s primary use is for correctness testing. Use Auditing for security, business accounting, and monetary transaction compliance. Use Diagnostics Events for debugging. Use Change Data Capture (CDC) for runtime duplication/exporting/streaming of the data.

FQL only logs the queries that successfully complete.

FQL appears to have little or no overhead in WRITE only workloads, and a minor overhead in MIXED workloads.

The implementation is file (binlog) based. Different OpenHFT plug-in libraries can be used, which will influence operability and performance.

Minor performance changes are to be expected, and will depend on the setup and configuration. You should perform testing and benchmarking to evaluate production implications. File-based logging needs to be configured to be operational safe and sustainable. Ensure appropriate metrics monitoring and alerts are in place around the log files.

FQL is intended for correctness testing its use will add quality assurance to existing clusters.

Starting in version 4.0, the cassandra.yaml configuration file contains a new setting to configure the replica-aware token allocation algorithm: allocate_tokens_for_local_replication_factor. It’s an alternative to the existing allocate_tokens_for_keyspace setting.

The following is the configuration setting and value that was used in Cassandra 3.11.x to configure the replica-aware token allocation algorithm:

The replication factor of the keyspace defined in the allocate_tokens_for_keyspace setting is used by the replica-aware token allocator when generating tokens for a new node. No other keyspace information is used. Providing a keyspace for this feature introduced additional overhead and complexities when using the feature in a new cluster.

Starting in version 4.0, the newly-added allocate_tokens_for_local_replication_factor setting can be used as an alternative means of specifying the desired replication factor to the replica-aware token allocator. It has a default value of 3 and reduces the complexities and overhead associated with using replica-aware token allocation.

Instead of keeping the allocate_tokens_for_keyspace setting and value, it should be removed or commented out in the configuration file when upgrading to Cassandra 4.x:

This change should be done as part of Update Cassandra configuration files.

The Thrift protocol has been removed starting in Cassandra 4.0. With the removal of the Thrift protocol, the following settings have also been removed from cassandra.yaml:

The above settings no longer appear in the default configuration as of version 4.0, and you must never add them back in cassandra.yaml, otherwise Cassandra will fail to start.

It’s recommended that clients currently using the Thrift protocol change to using the Native Binary protocol.

In Cassandra versions 3.0.19, 3.11.5, and 4.0, two new settings are configurable via cassandra.yaml:

These are hidden settings and not listed in cassandra.yaml. They have dynamic default values:

Clients that send requests over these thresholds will receive ​​OverloadedException error messages.

It’s recommended that you leave the default settings in place, and only tune them when needed.

These settings are exposed via the following Storage Service MBeans which allow them to be modified via JMX:

For more information, refer to CASSANDRA-15519.

Transient replication takes advantage of incremental repairs, allowing consistency levels and storage requirements to be reduced when data is marked as repaired.

Transient replication introduces new terminology for each replica. Replicas are now either full replicas or transient replicas. Full replicas, like replicas previously, store all data. Transient replicas store only unrepaired data.

When Cassandra 4.0 was released, transient replication was explicitly announced as an experimental feature. At the time of its release, transient replication couldn’t be used for:

Transient replication introduces operational overhead when it comes to changing replica factor (RF).

Transient Replication offers the potential of reducing network traffic and storage requirements, e.g.volume sizes and/or number of nodes. However, due to its experimental status, any reliance on this feature should be accompanied with committed involvement, resources, and testing.

Cassandra 4.0 introduces hardware-bound Zero Copy Streaming (ZCS). Streaming is a fundamental operation to many of Cassandra’s internal operations: bootstrapping, decommissioning, host replacement, repairs, rebuilds, etc. Prior to this feature, all data was streamed through the JVMs of the source and destination nodes, making it a slow and garbage collection-intensive process. With hardware-bound streaming, data can be transferred directly from disk to network avoiding the JVM.

ZCS is enabled by setting stream_entire_sstables: true in cassandra.yaml.

Throttling still occurs according to the stream_throughput_outbound setting.

ZCS will only be used when an SSTable contains only partitions that have been requested to be streamed. In practice this means it only works when vnodes are disabled (i.e. num_tokens=1). That is, if an SSTable candidate contains partitions that are additional to those of the requested streaming operation, it will be streamed via the traditional JVM streaming implementation.

ZCS is also unable to be used for any SSTables that contain legacy counters (from Cassandra versions before 2.1).

To summarize, for ZCS to work on a particular SSTable, the following checks must pass:

Activation and frequency of ZCS can be observed in the debug.log file, if it is enabled. The output will be from the CassandraEntireSSTableStreamReader logger.

ZCS provides improved availability and elasticity by reducing the time it takes to stream token range replicas between nodes during bootstrapping, decommissioning, and host replacement operations. ZCS provides improved performance by reducing the load and latency impact streaming large amounts of data through each JVM causes.

But ZCS only works for clusters not using vnodes, i.e. where num_tokens=1.

There are a number of other important streaming improvements introduced in version 4.0, such as Netty non-blocking NIO messaging, parallel keyspace streams, and better stream balancing during bootstrap. These improvements will improve bootstrapping times and overall elasticity of a Cassandra cluster.

To enable parallel streams, increase the value of streaming_connections_per_host in cassandra.yaml. This will help when streams are receiver CPU bound.

ZCS provides no benefits on vnode clusters. Since there is no negative impact, it’s generally recommended to leave ZCS enabled.

Starting in Cassandra 4.0, the cassandra-rackdc.properties file contains the new ec2_naming_scheme setting that allows the Ec2Snitch and Ec2MultiRegionSnitch to switch between legacy and AWS-style naming conventions.

The possible values are legacy and standard (default). In Cassandra 4.x, standard is the default naming convention used. To use the pre-4.x naming convention, this setting must be set to legacy, i.e. you must use the legacy value if you are upgrading a pre-4.0 cluster.

Prior Cassandra 4.0, the Ec2Snitch and Ec2MultiRegionSnitch used datacenter and rack naming conventions inconsistent with those presented in Amazon EC2 APIs. Full AWS-style naming was introduced in Cassandra 4.0 (CASSANDRA-7839) and made the default naming convention for Ec2Snitch and Ec2MultiRegionSnitch.

As part of this change, the ability to control the naming convention was added. This was done using the ec2_naming_scheme setting in the cassandra-rackdc.properties file. The possible values for this setting are legacy and standard.

If the endpoint_snitch setting in cassandra.yaml is set to either Ec2Snitch or Ec2MultiRegionSnitch, then the ec2_naming_scheme setting in the cassandra-rackdc.properties file must be set to legacy when upgrading from Cassandra 3.x to 4.x.