Phase 3: Enable asynchronous dual reads
In this phase, you can optionally enable asynchronous dual reads. The idea is to test performance and verify that Target can handle your application’s live request load before cutting over from Origin to Target.
|
As you test the performance on Target, be sure to examine the async read metrics. As noted in the section below, you can learn more in Asynchronous read requests metrics. |
Steps
The steps consist of changing the read_mode configuration variable in vars/zdm_proxy_core_config.yml from PRIMARY_ONLY (the default) to DUAL_ASYNC_ON_SECONDARY.
Example:
read_mode: DUAL_ASYNC_ON_SECONDARYBefore making the change, you should still have Origin as the primary cluster, which is the default:
primary_cluster: ORIGIN # or emptyTo apply this change, run the rolling_update_zdm_proxy.yml playbook as explained here.
|
This optional phase introduces an additional check to make sure that Target can handle the load without timeouts or unacceptable latencies. You would typically perform this step once you have migrated all the existing data from Origin and completed all validation checks and reconciliation, if necessary. |
Asynchronous Dual Reads mode
When using the ZDM Proxy, all writes are synchronously sent to both Origin and Target. Reads operate differently: with the default read mode, reads are only sent to the primary cluster (Origin by default).
Before changing the read routing so that reads are routed to Target (phase 4), you may want to temporarily send the reads to both clusters, to make sure that Target can handle the full workload of reads and writes.
If you set the proxy’s read mode configuration variable (read_mode) to DUAL_ASYNC_ON_SECONDARY, then asynchronous dual reads will be enabled.
That change will result in reads being additionally sent to the secondary cluster.
The proxy will return the read response to the client application as soon as the primary cluster’s response arrives.
The secondary cluster’s response will only be used to track metrics. There will be no impact to the client application if the read fails on the secondary cluster, or if the read performance on the secondary cluster is degraded. Therefore, this feature can be used as a safer way to test the full workload on Target before making the switch to set Target as the primary cluster (phase 4).
|
In some cases the additional read requests can cause the write requests to fail or timeout on that cluster. This means that, while this feature provides a way to route read requests to Target with a lower chance of having impact on the client application, it doesn’t completely eliminate that chance. |
Validating performance and error rate
Because the client application is not impacted by these asynchronous reads, the only way to measure the performance and error rate of these asynchronous reads are:
-
Check the metrics of the cluster itself
-
Check the asynchronous reads section of the ZDM Proxy metrics
In the ZDM Proxy Grafana dashboard that the ZDM Proxy Automation is able to deploy, there is a section dedicated to asynchronous reads where you can see latency percentiles, error rates, and some other metrics specific to these requests.
For more, see Asynchronous read requests metrics.
Reminder to switch off async dual reads
|
Once you are satisfied that your Target cluster is ready and tuned appropriately to handle the production read load, you can decide to switch your sync reads to Target.
At this point, be sure to also disable async dual reads by reverting |
