Troubleshooting tips

This page provides general troubleshooting advice and describes some common issues you might encounter with Zero Downtime Migration.

For specific error messages, see Troubleshooting scenarios.

You can also contact your DataStax account representative or DataStax Support, if you have a Luna service contract.

ZDM Proxy logs

The ZDM Proxy logs can help you troubleshoot issues with Zero Downtime Migration.

Set the ZDM Proxy log level

Set the ZDM Proxy log level to print the messages that you need.

The default log level is INFO, which is adequate for most logging.

If you need more detail for temporary troubleshooting, you can set the log level to DEBUG. However, this can slightly degrade performance, and DataStax recommends that you revert to INFO logging as soon as possible.

How you set the log level depends on how you deployed the ZDM Proxy:

If you used ZDM Proxy Automation to deploy the ZDM Proxy, set log_level in vars/zdm_proxy_core_config.yml.

You can change this value in a rolling fashion by editing the variable and running the rolling_update_zdm_proxy.yml playbook. For more information, see Change a mutable configuration variable.
If you didn’t use ZDM Proxy Automation to deploy the ZDM Proxy, set the ZDM_LOG_LEVEL environment variable on each proxy instance, and then restart each instance.

Retrieve the ZDM Proxy log files

If you used the ZDM Proxy Automation to deploy ZDM Proxy, then you can get logs for a single proxy instance, and you can use a playbook to retrieve logs for all instances. For instructions and more information, see Access the proxy logs.

If you did not use the ZDM Proxy Automation to deploy ZDM Proxy, you might have to access the logs another way. For example, if you used Docker, you can use the following command to export a container’s logs to a log.txt file:

docker logs my-container > log.txt

Keep in mind that Docker logs are deleted if the container is recreated.

Message levels

Some log messages contain text that sounds like an error, but they are not errors. The message’s level typically indicates severity:

level=debug and level=info: Expected and normal messages that are typically not errors. However, if you enable DEBUG logging, debug messages can help you find the source of a problem.
level=warn: Reports an event that wasn’t fatal to the overall process, but could indicate an issue with an individual request or connection.
level=error: Indicates an issue with the ZDM Proxy, client application, or clusters. These messages require further examination.

If the meaning of a warn or error message isn’t clear, you can submit an issue in the ZDM Proxy GitHub repository.

Common log messages

Here are the most common messages in the ZDM Proxy logs.

ZDM Proxy startup message

If the log level doesn’t filter out info entries, you can look for a Proxy started log message to verify that the ZDM Proxy started correctly. For example:

{"log":"time=\"2023-01-13T11:50:48Z\" level=info
msg=\"Proxy started. Waiting for SIGINT/SIGTERM to shutdown.
\"\n","stream":"stderr","time":"2023-01-13T11:50:48.522097083Z"}

ZDM Proxy configuration message

If the log level doesn’t filter out info entries, the first few lines of a ZDM Proxy log file contain all configuration variables and values in a long JSON string.

For example, this log message has been truncated for readability:

{"log":"time=\"2023-01-13T11:50:48Z\" level=info
msg=\"Parsed configuration: {\\\"ProxyIndex\\\":1,\\\"ProxyAddresses\\\":"...",
...TRUNCATED...
","stream":"stderr","time":"2023-01-13T11:50:48.339225051Z"}

Configuration settings can help with troubleshooting.

To make this message easier to read, pass it through a JSON formatter or paste it into a text editor that can reformat JSON.

Protocol log messages

There are cases where protocol errors are fatal, and they will kill an active connection that was being used to serve requests. However, it is also possible to get normal protocol log messages that contain wording that sounds like an error.

For example, the following DEBUG message contains the phrases force a downgrade and unsupported protocol version, which can sound like errors:

{"log":"time=\"2023-01-13T12:02:12Z\" level=debug msg=\"[TARGET-CONNECTOR]
Protocol v5 detected while decoding a frame. Returning a protocol message
to the client to force a downgrade: PROTOCOL (code=Code Protocol [0x0000000A],
msg=Invalid or unsupported protocol version (5)).\"\n","stream":"stderr","time":"2023-01-13T12:02:12.379287735Z"}

However, level=debug indicates that this is not an error. Instead, this is a normal part of protocol version negotiation (handshake) during connection initialization.

Check your ZDM Proxy version

In the ZDM Proxy logs, the first message contains the version string:

time="2023-01-13T13:37:28+01:00" level=info msg="Starting ZDM proxy version 2.1.0"
time="2023-01-13T13:37:28+01:00" level=info msg="Parsed configuration: ..."

This message is logged immediately before the long Parsed configuration string.

You can also pass the -version flag to the ZDM Proxy to print the version. For example, you can use the following Docker command:

docker run --rm datastax/zdm-proxy:2.x -version
ZDM proxy version 2.1.0

Don’t use --rm when you launch the ZDM Proxy container. This flag will prevent you from accessing the logs when ZDM Proxy stops or crashes.

Report an issue

To report an issue or get additional support, submit an issue in the ZDM component GitHub repositories:

ZDM Proxy repository
ZDM Proxy Automation repository (includes ZDM Proxy Automation and the ZDM Utility)
Cassandra Data Migrator repository
DSBulk Migrator repository

These repositories are public.

Don’t include any proprietary or private information in issues, pull requests, or comments that you make in these repositories.

In the issue description, include as much of the following information as possible, and make sure to remove all proprietary and private information before submitting the issue:

Your ZDM Proxy version.
ZDM Proxy logs, ideally at DEBUG level, if you can easily reproduce the issue and tolerate restarting the proxy instances to apply the log level configuration change.
Database deployment type (DSE, HCD, Cassandra, or Astra DB) and version for the origin and target clusters. The version isn’t required for Astra DB.
Screenshots of the ZDM Proxy metrics dashboards from Grafana or your chosen visualization tool.

Direct read access to your metrics dashboard is preferred, if permitted by your security policy. This is particularly helpful for performance-related issues.
Client application and driver logs.
The driver language and version that the client application is using.

For performance-related issues, provide the following additional information:

Which statement types (simple, prepared, batch) do you use?
If you use batch statements:
- Which driver API do you use to create these batches?
- Are you passing a BEGIN BATCH CQL query string to a simple/prepared statement, or do you use the actual batch statement objects that the drivers allow you to create?
How many parameters does each statement have?
Is CQL function replacement enabled? This feature is disabled by default. To determine if this feature is enabled, check the following variables:
- If you use ZDM Proxy Automation, check the Ansible advanced configuration variable replace_cql_functions.
- If you don’t use ZDM Proxy Automation, check the environment variable ZDM_REPLACE_CQL_FUNCTIONS.