Read Repair: repair during read path

Describes read repair, repair during read path.

When a read query encounters inconsistent results at a consistency level greater than ONE or LOCAL_ONE, DataStax Distribution of Apache Cassandra™ (DDAC) initiates a read repair. Such read repairs run in the foreground and block application operations until the repair process is complete.
Note: Read queries with a consistency level of ONE or LOCAL_ONE do not block application operations since a data mismatch must exist to trigger the read repair, and, since only one replicas is queried, no comparison occurs, therefore no mismatch.
In more detail:
  1. The coordinator node asks one replica for data and the others for a digest of their data.
  2. If there is a mismatch in the data returned to the coordinator from the replicas, a read is requested from all replicas involved in the query (dictated by the consistency level) and the results are merged.
  3. If a single replica doesn't have all of the latest data for each column, a new record is assembled by mixing and matching columns from different replicas.
  4. After determining the latest version, the record is written back to only the replicas involved in the request.
For example, in the case of a LOCAL_QUORUM read with a replication factor of three, two replicas are queried, so only those two replicas are repaired.
Note: Read repair does not propagate expired tombstones, nor does it consider expired tombstones when actually repairing data. That means that if there is tombstoned data that has not been propagated to all replica nodes before gc_grace_seconds has expired, that data may continue to be returned as live data.

For versions of DDAC earlier than 5.1.12, cluster-wide and local datacenter non-blocking background repairs can also be configured, and are governed by the parameters dclocal_read_repair_chance and read_repair_chance as described in table_options.

In more detail:
  1. If the read repair chance properties are not zero on a table, during each query Cassandra generates a random number between 0.0 and 1.0.
  2. If that random number is less than or equal to read_repair_chance, a non-blocking global read repair is initiated.
  3. If not, Cassandra tests to see if the random number is less than or equal to dc_local_read_repair_chance, and, if it is, a non-blocking read repair is performed in the local DC only.
Note: Cassandra uses a single random value for both of the read repair tests and global read repair, read_repair_chance, is evaluated first. If read_repair_chance is greater than or equal to dclocal_read_repair_chance for a given table, a local DC read repair never occurs.

Read repair cannot be performed on tables that use DateTieredCompactionStrategy (DTCS) - Deprecated, due to the method of checking timestamps used in DTCS compaction. If your table uses DateTieredCompactionStrategy, set read_repair_chance to zero. For other compaction strategies, read_repair_chance is typically set to a value of 0.2.