Exporting and visualizing metrics with Prometheus and Docker

Use the preconfigured Docker images provided by DataStax to quickly start exporting and visualizing metrics for DSE clusters.

To quickly get started with DSE Metrics Collector, download and install Docker, which provides the ability to run DataStax preconfigured dashboards. If you already have a monitoring and visualization solution, see Manually exporting and visualizing metrics with Prometheus.

Warning: To avoid the loss of metrics data, before using the Docker examples provided in this repository, make sure you have a good understanding of Prometheus's default data retention period and how to adjust it. By default, Prometheus is configured to only retain 15 days' worth of data. If you want to keep a longer history of your metrics data, revise docker-compose.yml to add the --storage.tsdb.retention flag onto the Prometheus runtime command line. Specify an appropriate value for your environment.

Prerequisites

  1. If necessary, download and install DataStax Enterprise (DSE) 6.7 or later.
  2. Download and install Docker.
  3. Download and install Docker Compose.
  4. Clone the DSE Metrics Collector Dashboards repository from GitHub.
Note: You must install Docker, Docker Compose, and the DSE Metrics Collector Dashboards repository on a node where DSE is not installed.

Procedure

  1. On your DSE cluster, check the configuration for DSE Metrics Collector:
    dsetool insights_config --show_config
    {
      "mode" : "DISABLED",
      "config_refresh_interval_in_seconds" : 30,
      "metric_sampling_interval_in_seconds" : 30,
      "data_dir_max_size_in_mb" : 1024,
      "node_system_info_report_period" : "PT1H"
    }
  2. On the DataStax Enterprise (DSE) cluster that DSE Metrics Collector reads from, create a /collectd directory in one of the following locations:
    • Tarball: installation_location/resources/dse/collectd/etc/collectd
    • Package: /etc/dse/collectd
  3. To enable the Prometheus server, create a configuration file on a DataStax Enterprise (DSE) cluster that DSE Metrics Collector reads from.

    For clusters managed by Lifecycle Manager (LCM), you can use LCM to export metrics collection to Prometheus and other supported monitoring tools.

    1. Create a prometheus.conf file in the directories you created in the previous step:
      sudo touch prometheus.conf
    2. Add the following code to the prometheus.conf configuration file and save it:
      vi prometheus.conf
      LoadPlugin write_prometheus
      
      <Plugin write_prometheus>
       Port “9103”
      </Plugin>
    3. On any node in the cluster, disable and re-enable DSE Metrics Collector to propagate the changes:
      dsetool insights_config --mode DISABLED
      Note: Disabling DSE Metrics Collector can take up to 30 seconds to propagate across the cluster. Wait at least 30 seconds before re-enabling DSE Metrics Collector.
      dsetool insights_config --mode ENABLED_WITH_LOCAL_STORAGE
  4. Ensure that the node is listening on port 9103, in addition to the typical DSE ports:
    netstat -lnt
    Active Internet connections (only servers)
    Proto Recv-Q Send-Q Local Address           Foreign Address     State
    tcp        0      0 10.200.176.166:9042     0.0.0.0:*           LISTEN
    tcp        0      0 0.0.0.0:22              0.0.0.0:*           LISTEN
    tcp        0      0 10.200.176.166:7000     0.0.0.0:*           LISTEN
    tcp        0      0 127.0.0.1:7199          0.0.0.0:*           LISTEN
    tcp        0      0 10.200.176.166:8609     0.0.0.0:*           LISTEN
    tcp        0      0 127.0.0.1:39940         0.0.0.0:*           LISTEN
    tcp6       0      0 :::9103                 :::*                LISTEN
    tcp6       0      0 10.200.176.166:61621    :::*                LISTEN
    tcp6       0      0 :::22                   :::*                LISTEN
    Note: If you configured Prometheus but port 9103 is not listening, read Error messages when libaio not installed (DSE 6.0 and 6.7) and ensure that libaio is installed and Epoll is working. The collectd daemon will not start if Epoll is not working.
  5. On the node where you cloned the DSE Metrics Collector Dashboard repository, navigate to the /prometheus directory and modify the tg_dse.json file to include the IP address of each node in your cluster under “targets”. The list of IP addresses is comma-delimited, and each IP address must be enclosed in double quotes.
    vi tg_dse.json
    [
      {
        "targets": [
          "10.100.110.96:9103",
          "10.100.100.95:9103",
          "10.100.100.97:9103"
        ]
      }
    ]
  6. From the cloned directory containing the DSE Metrics Collector repository, start and attach the Docker containers. This command also starts the included Prometheus and Grafana servers.
    docker-compose up
  7. View the Prometheus and Grafana dashboards where DSE Metrics Collector is reporting metrics:
    1. To view the Prometheus dashboard, which displays each DSE node as a target, open a browser and navigate to http://localhost:9090/targets.
    2. To view the Grafana dashboards that are linked to local Prometheus data source, open a browser and navigate to http://localhost:3000/dashboards.
  8. List the name of all Docker containers to obtain the container IDs for the Prometheus and Grafana images:
    docker ps -a
    CONTAINER ID        IMAGE                 COMMAND
    78d3324982ba        grafana/grafana       "/run.sh"
    aadc3d4ad1d1        prom/prometheus       "/bin/prometheus --c…"

Results

The Grafana dashboards display metrics for the defined clusters.