Configure and read ZDM Proxy logs

ZDM Proxy logs can help you verify that your ZDM Proxy instances are operating normally, investigate how processes are executed, and troubleshoot issues.

Set the ZDM Proxy log level

Set the ZDM Proxy log level to print the messages that you need.

The default log level is INFO, which is adequate for most logging.

If you need more detail for temporary troubleshooting, you can set the log level to DEBUG. However, this can slightly degrade performance, and DataStax recommends that you revert to INFO logging as soon as possible.

How you set the log level depends on how you deployed ZDM Proxy:

If you used ZDM Proxy Automation to deploy ZDM Proxy, set log_level in vars/zdm_proxy_core_config.yml, and then run the rolling_update_zdm_proxy.yml playbook. For more information, see Change a mutable configuration variable.
If you didn’t use ZDM Proxy Automation to deploy ZDM Proxy, set the ZDM_LOG_LEVEL environment variable on each proxy instance, and then restart each instance.

Get ZDM Proxy log files

If you used ZDM Proxy Automation to deploy ZDM Proxy, then you can get logs for a single proxy instance, and you can use a playbook to retrieve logs for all instances.

View or tail logs for one instance

ZDM Proxy runs as a Docker container on each proxy host.

To view the logs for a single ZDM Proxy instance, connect to a proxy host, and then run the following command:

docker container logs zdm-proxy-container

To tail (stream) the logs as they are written, use the --follow (-f) option:

docker container logs zdm-proxy-container -f

Keep in mind that Docker logs are deleted if the container is re-created.

Collect logs for multiple instances

ZDM Proxy Automation has a dedicated playbook, collect_zdm_proxy_logs.yml, that you can use to collect logs for all ZDM Proxy instances in a deployment.

You can view the playbook’s configuration in vars/zdm_proxy_log_collection_config.yml, but no changes are required to run it.

Connect to the Ansible Control Host Docker container. You can do this from the jumphost machine by running the following command:
```
docker exec -it zdm-ansible-container bash
```
Result
```
ubuntu@52772568517c:~$
```
Run the log collection playbook:
```
ansible-playbook collect_zdm_proxy_logs.yml -i zdm_ansible_inventory
```
This playbook creates a single zip file, zdm_proxy_logs_TIMESTAMP.zip, that contains the logs from all proxy instances. This archive is stored on the Ansible Control Host Docker container at /home/ubuntu/zdm_proxy_archived_logs.
To copy the archive from the container to the jumphost, open a shell on the jumphost, and then run the following command:
```
docker cp zdm-ansible-container:/home/ubuntu/zdm_proxy_archived_logs/zdm_proxy_logs_TIMESTAMP.zip DESTINATION_DIRECTORY_ON_JUMPHOST
```
Replace the following:
- TIMESTAMP: The timestamp from the name of your log file archive
- DESTINATION_DIRECTORY_ON_JUMPHOST: The path to the directory where you want to copy the archive

Get logs for deployments that don’t use ZDM Proxy Automation

If you didn’t use ZDM Proxy Automation to deploy ZDM Proxy, you must access the logs another way, depending on your deployment configuration and infrastructure.

For example, if you used Docker, you can use the following command to export a container’s logs to a log.txt file:

docker logs my-container > log.txt

Keep in mind that Docker logs are deleted if the container is re-created.

Message levels

Some log messages contain text that seems like an error but they aren’t errors. Instead, the message’s level indicates severity:

level=info: Expected and normal messages that typically aren’t errors.
level=debug: Expected and normal messages that typically aren’t errors. However, they can help you find the source of a problem by providing information about the environment and conditions when the error occurred.

debug messages are only recorded if you set the log level to DEBUG.
level=warn: Reports an event that wasn’t fatal to the overall process but might indicate an issue with an individual request or connection.
level=error: Indicates an issue with ZDM Proxy, the client application, or the clusters. These messages require further examination.

If the meaning of a warn or error message isn’t clear, you can report an issue.

Common log messages

Here are some of the most common messages in the ZDM Proxy logs.

ZDM Proxy startup message

If the log level doesn’t filter out info entries, you can look for a Proxy started log message to verify that ZDM Proxy started correctly. For example:

{"log":"time=\"2023-01-13T11:50:48Z\" level=info
msg=\"Proxy started. Waiting for SIGINT/SIGTERM to shutdown.
\"\n","stream":"stderr","time":"2023-01-13T11:50:48.522097083Z"}

ZDM Proxy configuration message

If the log level doesn’t filter out info entries, the first few lines of a ZDM Proxy log file contain all configuration variables and values in a long JSON string.

The following example log message is truncated for readability:

{"log":"time=\"2023-01-13T11:50:48Z\" level=info
msg=\"Parsed configuration: {\\\"ProxyIndex\\\":1,\\\"ProxyAddresses\\\":"...",
...TRUNCATED...
","stream":"stderr","time":"2023-01-13T11:50:48.339225051Z"}

Configuration settings can help with troubleshooting.

To make this message easier to read, pass it through a JSON formatter or paste it into a text editor that can reformat JSON.

Protocol log messages

There are cases where protocol errors are fatal, and they will kill an active connection that was being used to serve requests. However, it is also possible to get normal protocol log messages that contain wording that sounds like an error.

For example, the following DEBUG message contains the phrases force a downgrade and unsupported protocol version, which can sound like errors:

{"log":"time=\"2023-01-13T12:02:12Z\" level=debug msg=\"[TARGET-CONNECTOR]
Protocol v5 detected while decoding a frame. Returning a protocol message
to the client to force a downgrade: PROTOCOL (code=Code Protocol [0x0000000A],
msg=Invalid or unsupported protocol version (5)).\"\n","stream":"stderr","time":"2023-01-13T12:02:12.379287735Z"}

However, level=debug indicates that this is not an error. Instead, this is a normal part of protocol version negotiation (handshake) during connection initialization.