You changed the values of some configuration variables in the automation and then rolled them out using the rolling_update_zdm_proxy.yml playbook, but these changes are not taking effect on your ZDM Proxy instances.

The ZDM Proxy configuration comprises a number of variables, but only a subset of these can be changed on an existing deployment in a rolling fashion. The variables that can be changed with a rolling update are listed here.

All other configuration variables excluded from the list above are considered immutable and can only be changed by a redeployment. This is by design: immutable configuration variables should not be changed after finalizing the deployment prior to starting the migration, so allowing them to be changed through a rolling update would risk accidentally propagating some misconfiguration that could compromise the deployment’s integrity.

To change the value of configuration variables that are considered immutable, simply run the deploy_zdm_proxy.yml playbook again. This playbook can be run as many times as necessary and will just recreate the entire ZDM Proxy deployment from scratch with the provided configuration. Please note that this does not happen in a rolling fashion: the existing ZDM Proxy instances will be torn down all at the same time prior to being recreated, resulting in a brief window in which the whole ZDM Proxy deployment will become unavailable.

In the Java 4.x driver logs, the following issues can manifest during session initialization, or after initialization.

JAVA-2905 is a driver bug that manifests itself in this way. It affects Java driver 4.x, and was fixed on the 4.10.0 release.

If you are using spring boot and/or spring-data-cassandra then an upgrade of these dependencies will be necessary to a version that has the java driver fix.

Alternatively, you can force the protocol version on the driver to the max supported version by both clusters. V4 is a good recommendation that usually fits all but if the user is migrating from DSE to DSE then DSE_V1 should be used for DSE 5.x and DSE_V2 should be used for DSE 6.x.

To force the protocol version on the Java driver, check this section of the driver manual. We don’t believe this issue affects Java driver 3.x, but here are the instructions on how to force the version on 3.x, if necessary.

ZDM Proxy logs contain:

Protocol errors like these are a normal part of the handshake process where the protocol version is being negotiated. These protocol version downgrades happen when either the ZDM Proxy or at least one of the clusters doesn’t support the version requested by the client.

V5 downgrades are enforced by the ZDM Proxy but any other downgrade is requested by one of the clusters when they don’t support the version that the client requested. The proxy supports V3, V4, DSE_V1 and DSE_V2.

These log messages are informative only (log level DEBUG).

If you find one of these messages with a higher log level (especially level=error) then there might be a bug. At that point the issue will need to be investigated by the ZDM team. This log message with a log level of ERROR means that the protocol error occurred after the handshake, and this is a fatal unexpected error that results in a disconnect for that particular connection.

ZDM Proxy logs contain:

The control connections of the ZDM Proxy don’t perform protocol version negotiation, they only attempt to use protocol version 3. If one of the origin clusters doesn’t support at least V3 (e.g. Cassandra 2.0, DSE 4.6), then ZDM cannot be used for that migration at the moment. We plan to introduce support for Cassandra 2.0 and DSE 4.6 very soon.

None.

Credentials are incorrect or have insufficient permissions.

There are three sets of credentials in play with ZDM:

This error means that at least one of these three sets of credentials is incorrect or has insufficient permissions.

If the authentication error is preventing the proxy from starting then it’s either the Origin or Target credentials that are incorrect or have insufficient permissions. The log message shows whether it is the Target or Origin handshake that is failing.

If the proxy is able to start up — that is, this message can be seen in the logs:

Proxy started. Waiting for SIGINT/SIGTERM to shutdown.

then the authentication error is happening when a client application tries to open a connection to the proxy. In this case, the issue is with the Client credentials so the application itself is using invalid credentials (incorrect username/password or insufficient permissions).

Note that the proxy startup message has log level INFO so if the configured log level on the proxy is warning or error, you will have to rely on other ways to know whether the ZDM Proxy started correctly. You can check if the docker container is running (or process if docker isn’t being used) or if there is a log message similar to Error launching proxy.

The ZDM Proxy is listening on a custom port (not 9042) and:

The application is specifying the custom port as part of the contact points using the format <proxy_ip_address>:<proxy_custom_port>.

For example, using the Java driver, if the ZDM Proxy instances were listening on port 14035, this would look like:

.addContactPoints("172.18.10.36:14035", "172.18.11.48:14035", "172.18.12.61:14035")

The contact point is used as the first point of contact to the cluster, but the driver discovers the rest of the nodes via CQL queries. However, this discovery process doesn’t discover the ports, just the addresses so the driver uses the addresses it discovers with the port that is configured at startup.

As a result, port 14035 will only be used for the contact point initially discovered, while for all other nodes the driver will attempt to use the default 9042 port.

In the application, ensure that the custom port is explicitly indicated using the .withPort(<customPort>) API. In the above example:

ZDM Proxy logs contain:

The log message indicates that the server doesn’t recognize the word “CALL” in the query string which most likely means that it is an RPC (remote procedure call). From the proxy logs alone, it is not possible to see what method is being called by the query but it’s very likely the RPC that the drivers use to send DSE Insights data to the server.

Most DataStax drivers have DSE Insights reporting enabled by default when they detect a server version that supports it (regardless of whether the feature is enabled on the server side or not). The driver might also have it enabled for Astra DB depending on what server version Astra DB is returning for queries involving the system.local and system.peers tables.

These log messages are harmless but if your need to get rid of them, you can disable the DSE Insights driver feature through the driver configuration. Refer to this property for Java driver 4.x.

Consider a case where you deploy the metrics component of our ZDM Proxy Automation, a Grafana instance is deployed but you cannot login using the usual default admin/admin credentials.

The ZDM Proxy Automation specifies a custom set of credentials instead of relying on the admin/admin ones that are typically the default for Grafana deployments.

Check the credentials that are being used by looking up the vars/zdm_monitoring_config.yml file on the ZDM Proxy Automation directory. These credentials can also be modified before deploying the metrics stack.

ZDM Proxy log contains:

ZDM Proxy has connectivity only to a subset of the nodes.

The control connection (during ZDM Proxy startup) cycles through the nodes until it finds one that can be connected to. For client connections, each proxy instance cycles through its "assigned nodes" only. (The "assigned nodes" are a different subset of the cluster nodes for each proxy instance, generally non-overlapping between proxy instances so as to avoid any interference with the load balancing already in place at client-side driver level. The assigned nodes are not necessarily contact points: even discovered nodes undergo assignment to proxy instances.)

In the example above, the ZDM Proxy doesn’t have connectivity to 10.0.63.20, which was chosen as the origin node for the incoming client connection, but it was able to connect to 10.0.63.163 during startup.

Ensure that network connectivity exists and is stable between the ZDM Proxy instances and all Cassandra / DSE nodes of the local datacenter.

After a ZDM Proxy has been unavailable for some time and it gets back up, the client application takes too long to reconnect.

There should never be a reason to stop a ZDM Proxy instance other than a configuration change but maybe the proxy crashed or the user tried to do a configuration change and took a long time to get the ZDM Proxy back up.

The ZDM Proxy does not send topology events to the client applications so the time it takes for the driver to reconnect to a ZDM Proxy instance is determined by the reconnection policy.

Restart the client application to force an immediate reconnect.

If you expect ZDM Proxy instances to go down frequently, change the reconnection policy on the driver so that the interval between reconnection attempts has a shorter limit.

ZDM Proxy Automation’s logs:

The Astra DevOps API is likely temporarily unavailable.

Download the Astra DB Secure Connect Bundle (SCB) manually and provide its path to the ZDM Proxy Automation as explained here. For information about the SCB, see working with Secure Connect Bundle.

The ZDM Proxy doesn’t start and the following appears on the proxy logs:

There are two possible causes for this:

Start by opening Astra Portal and checking the Status of your database. If it is Hibernated, click the “Resume” button and wait for it to become Active. If it is Active already, then it is likely an issue with permissions.

We recommend starting with a token that has the Database Administrator role in Astra DB to confirm that it is a permissions issue. Refer to Manage user permissions.

Dual reads are enabled and the following messages are found in the ZDM Proxy logs:

The last log message is logged when the async connection runs out of stream ids. The async connection is a connection dedicated to the async reads (asynchronous dual reads feature). This can be caused by timeouts (first log message) or the connection not being able to keep up with the load.

If the log files are being spammed with these messages then it is likely that an outage occurred which caused all responses to arrive after requests timed out (second log message). In this case the async connection might not be able to recover.

Keep in mind that any errors in the async request path (dual reads) will not affect the client application so these log messages might be useful to predict what may happen when the reads are switched over to the TARGET cluster but async read errors/warnings by themselves do not cause any impact to the client.

Starting in version 2.1.0, you can now tune the maximum number of stream ids available per connection, which by default is 2048. You can increase it to match your driver configuration through the zdm_proxy_max_stream_ids property.

If these errors are being constantly written to the log files (for minutes or even hours) then it is likely that only an application OR ZDM Proxy restart will fix it. If you find an issue like this please submit an Issue in our GitHub repo.

Every 10 minutes a message is logged in the ZDM Proxy logs showing a disconnect that was caused by Astra DB.

Astra DB terminates idle connections after 10 minutes of inactivity. If a client application is only sending reads through a connection then the Target (i.e. Astra in this case) connection will not get any traffic because ZDM forwards all reads to the Origin connection.

This issue has been fixed in ZDM Proxy 2.1.0. We encourage you to upgrade to that version or greater. By default, ZDM Proxy now sends heartbeats after 30 seconds of inactivity on a cluster connection, to keep it alive. You can tune the heartbeat interval with the Ansible configuration variable heartbeat_insterval_ms, or by directly setting the ZDM_HEARTBEAT_INTERVAL_MS environment variable if you do not use the ZDM Proxy Automation.

Consider a case where a user runs separate benchmarks against:

The results of these tests show latency/throughput values are worse with ZDM than when connecting to Astra DB or Origin directly.

ZDM will always add additional latency which, depending on the nature of the test, will also result in a lower throughput. Whether this performance hit is expected or not depends on the difference between the ZDM test results and the test results with the cluster that performed the worst.

Writes in ZDM require an ACK from both clusters while reads only require the result from the Origin cluster (or target if the proxy is set up to route reads to the target cluster). This means that if Origin has better performance than Target then ZDM will inevitably have a worse performance for writes.

From our testing benchmarks, a performance degradation of up to 2x latency is not unheard of even without external factors adding more latency, but it is still worth checking some things that might add additional latency like whether the proxy is deployed on the same Availability Zone (AZ) as the Origin cluster or application instances.

Simple statements and batch statements are things that will make the proxy add additional latency compared to normal prepared statements. Simple statements should be discouraged especially with the ZDM Proxy because currently the proxy takes a considerable amount of time just parsing the queries and with prepared statements the proxy only has to parse them once.

If you are using simple statements, consider using prepared statements as the best first step.

Increasing the number of proxies might help, but only if the VMs resources (CPU, RAM or network IO) are near capacity. The ZDM Proxy doesn’t use a lot of RAM, but it uses a lot of CPU and network IO.

Deploying the proxy instances on VMs with faster CPUs and faster network IO might help, but only your own tests will reveal whether it helps, because it depends on the workload type and details about your environment such as network/VPC configurations, hardware, and so on.

ZDM Proxy logs contain:

This could be the case if the origin (DSE) cluster has Metrics Collector enabled to report metrics for DataStax drivers and my_user does not have the required permissions. ZDM Proxy simply passes through these.

There are two options to get this fixed.