Phase 3: Enable asynchronous dual reads
In this phase, you can optionally enable asynchronous dual reads. The idea is to test performance and verify that the target cluster can handle your application’s live request load before cutting over from the origin cluster to the target cluster.
|
As you test the performance on the target, be sure to examine the async read metrics. As noted in the section below, you can learn more in Asynchronous read requests metrics. |
Steps
The steps consist of changing the read_mode configuration variable in vars/zdm_proxy_core_config.yml from PRIMARY_ONLY (the default) to DUAL_ASYNC_ON_SECONDARY.
Example:
read_mode: DUAL_ASYNC_ON_SECONDARYBefore making the change, you should still have the origin as the primary cluster, which is the default:
primary_cluster: ORIGIN # or emptyTo apply this change, run the rolling_update_zdm_proxy.yml playbook as explained here.
|
This optional phase introduces an additional check to make sure that the target can handle the load without timeouts or unacceptable latencies. You would typically perform this step once you have migrated all the existing data from the origin cluster and completed all validation checks and reconciliation, if necessary. |
Asynchronous Dual Reads mode
When using the ZDM Proxy, all writes are synchronously sent to both the origin and target clusters. Reads operate differently: with the default read mode, reads are only sent to the primary cluster (Origin by default).
In Phase 4, you will change the read routing so that reads are routed to the target. Before you do this, you might want to temporarily send the reads to both clusters to make sure that the target can handle the full workload of reads and writes.
If you set the proxy’s read_mode configuration variable to DUAL_ASYNC_ON_SECONDARY, then asynchronous dual reads will be enabled.
That change will result in reads being additionally sent to the secondary cluster.
The proxy will return the read response to the client application as soon as the primary cluster’s response arrives.
The secondary cluster’s response will only be used to track metrics. There will be no impact to the client application if the read fails on the secondary cluster, or if the read performance on the secondary cluster is degraded. Therefore, you can use this feature as a safer way to test the full workload on the target before setting the target as the primary cluster in Phase 4
|
In some cases the additional read requests can cause the write requests to fail or timeout on that cluster. This means that, while this feature provides a way to route read requests to the target with a lower chance of having impact on the client application, it doesn’t completely eliminate that chance. |
Validating performance and error rate
Because the client application is not impacted by these asynchronous reads, the only way to measure the performance and error rate of these asynchronous reads are:
-
Check the metrics of the cluster itself
-
Check the asynchronous reads section of the ZDM Proxy metrics
In the ZDM Proxy Grafana dashboard that the ZDM Proxy Automation is able to deploy, there is a section dedicated to asynchronous reads where you can see latency percentiles, error rates, and some other metrics specific to these requests.
For more, see Asynchronous read requests metrics.
Reminder to switch off async dual reads
Once you are satisfied that your target cluster is ready and tuned appropriately to handle the production read load, you can switch your sync reads to the target permanently.
At this point, be sure to also disable async dual reads by reverting read_mode in vars/zdm_proxy_core_config.yml to PRIMARY_ONLY.
For more information and instructions, see Phase 4: Route reads to the target.
