Set up the ZDM Proxy Automation with ZDM Utility

This page explains how to use the ZDM Utility to set up the Ansible Control Host container for the ZDM Proxy Automation.

After completing the setup tasks in ZDM Utility, see the next topic for subsequent steps to use ZDM Proxy Automation, which you will use to deploy ZDM Proxy instances and the monitoring stack.

Once completed, you will have a working and fully monitored ZDM Proxy deployment.

Introduction

The ZDM Proxy Automation uses Ansible, which deploys and configures the ZDM Proxy instances and monitoring stack via playbooks. This step expects that the infrastructure has been already provisioned. See Deployment and infrastructure considerations, which include the infrastructure requirements.

Configuring a machine to serve as the Ansible Control Host is very easy using the ZDM Utility. This is a Golang (Go) executable program that runs anywhere. This utility prompts you for a few configuration values, with helpful embedded explanations and error handling, then automatically creates the Ansible Control Host container ready for you to use. From this container, you will be able to easily configure and run the ZDM Proxy Automation Ansible playbooks.

ZDM Proxy connections from Docker container created by ZDM Utility

Prerequisites

You must have already provisioned the ZDM infrastructure, which means you must have the server machines ready, and know their IP addresses. These can be in the cloud provider of your choice or on-premise.
Docker needs to be installed on the machine that will be running the Ansible Control Host container. For comprehensive installation instructions, please refer to the official Docker documentation.
The docker command must not require superuser privileges. The instructions to do this can be found here.

That’s it!

Please note that the manual, non-superuser installation of Docker described above is to be done on the machine that will run the Ansible Control Host. From that point, the automation will take care of installing Docker on the ZDM Proxy machines without further intervention.

Jumphost

In this guide, we’ll use a jumphost to run the Ansible Control Host container.

A jumphost is a server on a network used to access and manage devices in a separate security zone, providing a controlled means of access between them. The jumphost can be, for example, a Linux server machine that is able to access the server machines that you wish to use for your ZDM Proxy deployment.

The jumphost will serve three purposes:

Accessing the ZDM Proxy machines
Running the Ansible Control Host container, from which the ZDM Proxy Automation can be run
Running the ZDM monitoring stack, which uses Prometheus and Grafana to expose the metrics of all the ZDM Proxy instances in a preconfigured dashboard.

To simplify accessing the jumphost and ZDM Proxy instances from your machine, create a custom SSH configuration file. The following steps will assume that this file exists.

Let’s get started.

Proxy deployment setup on the jumphost

To run the ZDM Proxy Automation, the Ansible Control Host needs to be able to connect to all other instances of the ZDM Proxy deployment. For this reason, it needs to have the SSH key required by those instances.

Add SSH keys to the jumphost

From your local machine, transfer (scp) the SSH private key for the ZDM deployment to the jumphost. Example:

scp -F <path to zdm_ssh_config> <zdm key> jumphost:

Now connect to the jumphost.

ssh -F <path to zdm_ssh_config> jumphost

Running the ZDM Utility

From the jumphost, download the ZDM Utility’s executable. Releases are available here:

https://github.com/datastax/zdm-proxy-automation/releases

The downloadable archive name format is zdm-util-<platform>-<version>.

Download example with <version> placeholder. Latest GitHub release (latest by date) :

wget https://github.com/datastax/zdm-proxy-automation/releases/download/<version>/zdm-util-linux-amd64-<version>.tgz

Here’s an example to wget ZDM Proxy 2.1.0:

wget https://github.com/datastax/zdm-proxy-automation/releases/download/v2.1.0/zdm-util-linux-amd64-v2.1.0.tgz

Once downloaded, unzip it. Here’s an example with ZDM Proxy 2.1.0:

tar -xvf zdm-util-linux-amd64-v2.1.0.tgz

Run the ZDM Utility. Here’s an example with ZDM Proxy 2.1.0:

./zdm-util-v2.1.0

The utility prompts you for a few configuration values, then creates and initializes the Ansible Control Host container.

The ZDM Utility will store the configuration that you provide into a file named ansible_container_init_config in the current directory. If you run the utility again, it will detect the file and ask you if you wish to use that configuration or discard it. If the configuration is not fully valid, you will be prompted for the missing or invalid parameters only.

You can also pass a custom configuration file to the ZDM Utility with the optional command-line parameter -utilConfigFile. Example:

Here’s an example with ZDM Proxy 2.1.0:

./zdm-util-v2.1.0 -utilConfigFile your_config_file

The ZDM Utility will validate each variable that you enter. In case of invalid variables, it will display specific messages to help you fix the problem.

You have five attempts to enter valid variables. You can always run the ZDM Utility again, if necessary.

Enter the path to, and name of, the SSH private key to access the proxy hosts. Example:
```
~/my-zdm-key
```
Enter the common prefix of the private IP addresses of the proxy hosts. Example:
```
172.18.*
```
You’re asked if you have an existing Ansible inventory file. If you do, and you transferred it to the jumphost, you can just specify it. If you do not, the ZDM Utility will create one based on your answers to prompts and save it. Here we’ll assume that you do not have one. Enter n.
The created file will be named zdm_ansible_inventory in your working directory.
Next, indicate if this deployment is for local testing and evaluation (such as when you’re creating a demo or just experimenting with the ZDM Proxy). In this example, we’ll enter n because this scenario is for a production deployment.
Now enter at least three proxy private IP addresses for the machines that will run the ZDM Proxy instances, for a production deployment. (If we had indicated above that we’re doing local testing in dev, only one proxy would have been required.) Example values entered at the ZDM Utility’s prompt, for production:
```
172.18.10.137
172.18.11.88
172.18.12.191
```
To finish entering private IP addresses, simply press ENTER at the prompt.

Optionally, when prompted, you can enter the private IP address of your Monitoring instance, which will use Prometheus to store data and Grafana to visualize it into a preconfigured dashboard. It is strongly recommended exposing the ZDM Proxy metrics in the preconfigured dashboard that ships with the ZDM Proxy Automation for easy monitoring. You can skip this step if you haven’t decided which machine to use for monitoring, or if you wish to use your own monitoring stack.

We highly recommend that you configure a monitoring instance, unless you intend to use a monitoring stack that you already have. For migrations that may run for multiple days, it is essential that you use metrics to understand the performance and health of the ZDM Proxy instances.

You cannot rely solely on information in the logs. They report connection or protocol errors, but do not give you enough information on how the ZDM Proxy is working and how each cluster is responding. Metrics, however, provide especially helpful data and the graphs show you how they vary over time. The monitoring stack ships with preconfigured Grafana dashboards that are automatically set up as part of the monitoring deployment.

For details about the metrics you can observe in these preconfigured Grafana dashboards, see this section of the troubleshooting tips.

You can choose to deploy the monitoring stack on the jumphost or on a different machine, as long as it can connect to the ZDM Proxy instances over TCP on ports 9100 (to collect host-level metrics) and on the port on which the ZDM Proxy exposes its own metrics, typically 14001.

In this example, we’ll enter the same IP of the Ansible control host (the jumphost machine on which we’re running the ZDM Utility). Example:

172.18.100.128

At this point, the ZDM Utility:

Has created the Ansible Inventory to the default file, zdm_ansible_inventory.
Has written the ZDM Utility configuration to the default file, ansible_container_init_config.
Presents a summary of the configuration thus far, and prompts you to Continue. Example:

A summary of the configuration provided is displayed in the terminal

If you agree, enter Y to proceed.

The ZDM Utility now:

Creates and downloads the image of the Ansible Docker container for you.
Creates, configures and starts the Ansible Control Host container.
Displays a message. Example:

Ansible Docker container success messages

Depending on your circumstances, you can make different choices in the ZDM Utility, which will result in a path that is slightly different to the one explained here. The utility will guide you through the process with meaningful, self-explanatory messages and help you rectify any issue that you may encounter.

The successful outcome will always be a configured Ansible Control Host container ready to run the ZDM Proxy Automation.