Retries

Quick overview

What to do when a request failed on a node: retry (same or other node), rethrow, or ignore.

• advanced.retry-policy in the configuration. Default policy retries at most once, in cases that have a high chance of success; you can also write your own.
• can have per-profile policies.
• only kicks in if the query is idempotent.

When a query fails, it sometimes makes sense to retry it: the error might be temporary, or the query might work on a different node. The driver uses a retry policy to determine when and how to retry.

Built-in retry policies

The driver ships with two retry policies: DefaultRetryPolicy –– the default –– and ConsistencyDowngradingRetryPolicy.

The default retry policy should be preferred in most cases as it only retries when it is perfectly safe to do so, and when the chances of success are high enough to warrant a retry.

ConsistencyDowngradingRetryPolicy is provided for cases where the application can tolerate a temporary degradation of its consistency guarantees. Its general behavior is as follows: if, based on the information the coordinator returns, retrying the operation with the initially requested consistency has a chance to succeed, do it. Otherwise, if based on this information, we know that the initially requested consistency level cannot be achieved currently, then:

• For writes, ignore the exception if we know the write has been persisted on at least one replica.
• For reads, try reading again at a weaker consistency level.

Keep in mind that this may break invariants! For example, if your application relies on immediate write visibility by writing and reading at QUORUM only, downgrading a write to ONE could cause that write to go unnoticed by subsequent reads at QUORUM. Furthermore, this policy doesn’t always respect datacenter locality; for example, it may downgrade LOCAL_QUORUM to ONE, and thus could accidentally send a write that was intended for the local datacenter to another datacenter. In summary: only use this retry policy if you understand the consequences.

Since DefaultRetryPolicy is already the driver’s default retry policy, no special configuration is required to activate it. To use ConsistencyDowngradingRetryPolicy instead, the following option must be declared in the driver configuration:

datastax-java-driver.advanced.retry-policy.class = ConsistencyDowngradingRetryPolicy

You can also use your own policy by specifying for the above option the fully-qualified name of a class that implements RetryPolicy.

Behavior

The behavior of both policies will be detailed in the sections below.

The policy has several methods that cover different error cases. Each method returns a RetryVerdict. A retry verdict essentially provides the driver with a RetryDecision to indicate what to do next. There are four possible retry decisions:

• retry on the same node;
• retry on the next node in the query plan for this statement;
• rethrow the exception to the user code (from the session.execute call, or as a failed future if using the asynchronous API);
• ignore the exception. That is, mark the request as successful, and return an empty result set.

onUnavailableVerdict

A request reached the coordinator, but there weren’t enough live replicas to achieve the requested consistency level. The coordinator replied with an UNAVAILABLE error.

If the policy rethrows the error, the user code will get an UnavailableException. You can inspect the exception’s fields to get the amount of replicas that were known to be alive when the error was triggered, as well as the amount of replicas that where required by the requested consistency level.

The default policy triggers a maximum of one retry, to the next node in the query plan. The rationale is that the first coordinator might have been network-isolated from all other nodes (thinking they’re down), but still able to communicate with the client; in that case, retrying on the same node has almost no chance of success, but moving to the next node might solve the issue.

ConsistencyDowngradingRetryPolicy also triggers a maximum of one retry, but instead of trying the next node, it will downgrade the initial consistency level, if possible, and retry the same node. Note that if it is not possible to downgrade, this policy will rethrow the exception. For example, if the original consistency level was QUORUM, and 2 replicas were required to achieve a quorum, but only one replica is alive, then the query will be retried with consistency ONE. If no replica was alive however, there is no point in downgrading, and the policy will rethrow.

onReadTimeoutVerdict

A read request reached the coordinator, which initially believed that there were enough live replicas to process it. But one or several replicas were too slow to answer within the predefined timeout (read_request_timeout_in_ms in cassandra.yaml); therefore the coordinator replied to the client with a READ_TIMEOUT error.

This could be due to a temporary overloading of these replicas, or even that they just failed or were turned off. During reads, Cassandra doesn’t request data from every replica to minimize internal network traffic; instead, some replicas are only asked for a checksum of the data. A read timeout may occur even if enough replicas responded to fulfill the consistency level, but only checksum responses were received (the method’s dataPresent parameter allow you to check if you’re in that situation).

If the policy rethrows the error, the user code will get a ReadTimeoutException (do not confuse this error with DriverTimeoutException, which happens when the coordinator didn’t reply at all to the client).

The default policy triggers a maximum of one retry (to the same node), and only if enough replicas had responded to the read request but data was not retrieved amongst those. That usually means that enough replicas are alive to satisfy the consistency, but the coordinator picked a dead one for data retrieval, not having detected that replica as dead yet. The reasoning is that by the time we get the timeout, the dead replica will likely have been detected as dead and the retry has a high chance of success.

ConsistencyDowngradingRetryPolicy behaves like the default policy when enough replicas responded. If not enough replicas responded however, it will attempt to downgrade the initial consistency level, and retry the same node. If it is not possible to downgrade, this policy will rethrow the exception.

onWriteTimeoutVerdict

This is similar to onReadTimeout, but for write operations. The reason reads and writes are handled separately is because a read is obviously a non mutating operation, whereas a write is likely to be. If a write times out at the coordinator level, there is no way to know whether the mutation was applied or not on the non-answering replica.

If the policy rethrows the error, the user code will get a WriteTimeoutException.

This method is only invoked for idempotent statements. Otherwise, the driver bypasses the retry policy and always rethrows the error.

The default policy triggers a maximum of one retry (to the same node), and only for a BATCH_LOG write. The reasoning is that the coordinator tries to write the distributed batch log against a small subset of nodes in the local datacenter; a timeout usually means that none of these nodes were alive but the coordinator hadn’t detected them as dead yet. By the time we get the timeout, the dead nodes will likely have been detected as dead, and the retry has a high chance of success.

ConsistencyDowngradingRetryPolicy also triggers a maximum of one retry, but behaves differently:

• For SIMPLE and BATCH write types: if at least one replica acknowledged the write, the policy will assume that the write will be eventually replicated, and decide to ignore the error; in other words, it will consider that the write already succeeded, albeit with weaker consistency guarantees: retrying is therefore useless. If no replica acknowledged the write, the policy will rethrow the error.
• For UNLOGGED_BATCH write type: since only part of the batch could have been persisted, the policy will attempt to downgrade the consistency level and retry on the same node. If downgrading is not possible, the policy will rethrow.
• For BATCH_LOG write type: the policy will retry the same node, for the reasons explained above.
• For other write types: the policy will always rethrow.

onRequestAbortedVerdict

The request was aborted before we could get a response from the coordinator. This can happen in two cases:

• if the connection was closed due to an external event. This will manifest as a ClosedConnectionException (network failure) or HeartbeatException (missed heartbeat);
• if there was an unexpected error while decoding the response (this can only be a driver bug).

This method is only invoked for idempotent statements. Otherwise, the driver bypasses the retry policy and always rethrows the error.

Both the default policy and ConsistencyDowngradingRetryPolicy retry on the next node if the connection was closed, and rethrow (assuming a driver bug) in all other cases.

onErrorResponseVerdict

The coordinator replied with an error other than READ_TIMEOUT, WRITE_TIMEOUT or UNAVAILABLE. Namely, this covers OverloadedException, ServerError, TruncateException, ReadFailureException and WriteFailureException.

This method is only invoked for idempotent statements. Otherwise, the driver bypasses the retry policy and always rethrows the error.

Both the default policy and ConsistencyDowngradingRetryPolicy rethrow read and write failures, and retry other errors on the next node.

Hard-coded rules

There are a few cases where retrying is always the right thing to do. These are not covered by RetryPolicy, but instead hard-coded in the driver:

• any error before a network write was attempted: to send a query, the driver selects a node, borrows a connection from the host’s connection pool, and then writes the message to the connection. Errors can occur before the write was even attempted, for example if the connection pool is saturated, or if the node went down right after we borrowed. In those cases, it is always safe to retry since the request wasn’t sent, so the driver will transparently move to the next node in the query plan.
• re-preparing a statement: when the driver executes a prepared statement, it may find out that the coordinator doesn’t know about it, and need to re-prepare it on the fly (this is described in detail here). The query is then retried on the same node.
• trying to communicate with a node that is bootstrapping: this is a rare edge case, as in practice the driver should never try to communicate with a bootstrapping node (the only way is if it was specified as a contact point). It is again safe to assume that the query was not executed at all, so the driver moves to the next node.

Similarly, some errors have no chance of being solved by a retry. They will always be rethrown directly to the user. These include:

• QueryValidationException and any of its subclasses;
• FunctionFailureException;
• ProtocolError.

Using multiple policies

The retry policy can be overridden in execution profiles:

datastax-java-driver {
  advanced.retry-policy {
    class = DefaultRetryPolicy
  }
  profiles {
    custom-retries {
      advanced.retry-policy {
        class = CustomRetryPolicy
      }
    }
    slow {
      request.timeout = 30 seconds
    }
  }
}

The custom-retries profile uses a dedicated policy. The slow profile inherits the default profile’s. Note that this goes beyond configuration inheritance: the driver only creates a single DefaultRetryPolicy instance and reuses it (this also occurs if two sibling profiles have the same configuration).

Each request uses its declared profile’s policy. If it doesn’t declare any profile, or if the profile doesn’t have a dedicated policy, then the default profile’s policy is used.