Read request coordination
When DataStax Enterprise (DSE) processes a read request, a coordinator node can send several types of requests to the replica nodes to prepare the response that is sent back to the client.
|
Read requests only involve the replica nodes that contain the required data for the request. For example, if a cluster has 12 nodes, and each keyspace has a replication factor of 3, only 3 of the 12 nodes contain the data for a given keyspace. By extension, the consistency level applies to the relevant replicas only.
For example, a consistency level of |
- Direct (full) request
-
This type of request retrieves the complete data for the read from a replica node.
If the consistency level is
ONEthe coordinator node contacts only one replica node, using the response from that replica to satisfy the read request for the client.At other consistency levels, the coordinator node can compare preview responses from multiple replicas using a digest request.
- Digest request before full request with possible read repair
-
If the consistency level requires more than one node to respond, the coordinator node contacts all replicas for a preview of the data. If there are more replicas than required to satisfy the consistency level, the coordinator node sends full requests to the replicas that respond the fastest.
If a node selected for a full request doesn’t respond fast enough, the request is retried or fails, depending on the consistency level.
If a replica’s response doesn’t match the other nodes, then the coordinator node triggers read repair to patch the mismatched replicas. This is an interim repair to align the replicas in case of subsequent reads on the same rows; it is not a full repair. When a record is inconsistent across replicas, the coordinator node uses the record with the most recent timestamp for the read repair and the response to the read request.
Rapid read protection using speculative_retry
When the replicas selected for a read are down or slow to respond, rapid read protection allows DSE to fallback to other replicas to satisfy the read request.
This retry trigger is set by the speculative_retry table property.
Speculative retries send requests to additional nodes beyond the consistency level. For example, if the coordinator node originally selects 3 nodes, and 1 node doesn’t respond, then it selects another node (4 total) for the retry. This strategy can help reduce the 95 or 99 percentile tail read latencies, but it puts more load on the nodes due to the additional request processing. This strategy also requires that additional replicas, beyond the consistency level, are available.
For example, given a 12-node cluster with a replication factor of 3, assume that nodes 1, 3, and 6 are replicas, and node 10 is chosen as the coordinator node.
If a request is received at consistency level ONE, node 10 sends the request to node 1.
When node 1 fails to respond in time, node 10 sends the request to node 6, which responds in time and fulfills the request.
|
Examples of read consistency levels
The following examples show how DSE processes read requests at consistency levels with one or more datacenters.
Reads at consistency level QUORUM and LOCAL_QUORUM
When reading at QUORUM or LOCAL_QUORUM, it is a best practice to have an odd number of replicas so that the request can be satisfied even if nodes are down.
For example, if there are 3 replicas, only 2 replicas need to provide a full response to satisfy a read at QUORUM.
This makes the cluster more resilient because a node can go down without causing requests to fail.
If any selected replicas are inconsistent, a read repair is triggered by the coordinator node to avoid this inconsistency in future reads. The coordinator node assembles the response to the client using the records with the most recent timestamp in each returned row.
Replicas that aren’t required to satisfy a read can be used for speculative retry, if needed.
For reads at QUORUM in multi-datacenter clusters, the required number of required replicas increases relative to the number of datacenters, but the replicas selected for full requests can be in any datacenter.
For reads at LOCAL_QUORUM in multi-datacenter clusters, there are more replicas across the datacenters, but the selected replicas must be from the same datacenter as the coordinator node.
Read repairs can be triggered, but the coordinator node only selects replicas for full requests and retries from its local datacenter.
Reads at consistency level ONE or LOCAL_ONE
When reading at ONE, it is still a best practice to have replicas so that the request can be satisfied through speculative retry even if nodes are down.
The coordinator node sends a full request directly to the closest replica that stores the data required for the read. The closest node is the node with the lowest network latency, which is usually in the same datacenter as the coordinator node. However, in multi-datacenter clusters, the coordinator node selects the closest replica in any datacenter.
Read repair is never triggered at ONE because only one replica is selected for a full request.
The coordinator node doesn’t send digest requests to compare data before sending the full request.
For reads at LOCAL_ONE in multi-datacenter clusters, the coordinator node selects the closest replica in the same datacenter to send a full request.
Regardless of the number of replicas across datacenters, the coordinator node only selects a node from its local datacenter.
