Exporting and visualizing metrics with Prometheus and Docker

Use the preconfigured Docker images provided by DataStax to quickly start exporting and visualizing metrics for DSE clusters.

To quickly get started with DSE Metrics Collector, download and install Docker to use the DataStax preconfigured dashboards. If you already have a monitoring and visualization solution, see Manually exporting and visualizing metrics with Prometheus.

Prerequisites

  1. If necessary, download and install DSE.
  2. Download and install Docker.
  3. Download and install Docker Compose.
  4. Clone the DSE Metrics Collector Dashboards repository from GitHub.
Note: Docker, Docker Compose, and the DSE Metrics Collector Dashboards repository must all be located on a node where DSE is not installed.

Procedure

  1. On your DSE cluster, enable the DSE Metrics Collector:
    dsetool insights_config --show_config
    {
      "mode" : "DISABLED",
      "config_refresh_interval_in_seconds" : 30,
      "metric_sampling_interval_in_seconds" : 30,
      "data_dir_max_size_in_mb" : 1024,
      "node_system_info_report_period" : "PT1H"
    }
  2. To check the configuration for DSE Metrics Collector:
    dsetool insights_config --show_config
    {
      "mode" : "ENABLED_WITH_LOCAL_STORAGE",
      "config_refresh_interval_in_seconds" : 30,
      "metric_sampling_interval_in_seconds" : 30,
      "data_dir_max_size_in_mb" : 1024,
      "node_system_info_report_period" : "PT1H"
    }
  3. On the DataStax Enterprise (DSE) cluster that DSE Metrics Collector reads from, create a /collectd directory in one of the following locations:
    • Tarball: installation_location/resources/dse/collectd/etc/collectd
    • Package: /etc/dse/collectd
  4. To enable the Prometheus server, create a configuration file.
    1. Create a prometheus.conf file in the directories you created in the previous step:
      sudo touch prometheus.conf
    2. Add the following code to the prometheus.conf configuration file and save it:
      vi prometheus.conf
      LoadPlugin write_prometheus
      
      <Plugin write_prometheus>
       Port “9103”
      </Plugin>
    3. On any node in the cluster, disable and re-enable DSE Metrics Collector to propagate the changes:
      dsetool insights_config --mode DISABLED
      Note: Disabling DSE Metrics Collector can take up to 30 seconds to propagate across the cluster. Wait at least 30 seconds before re-enabling DSE Metrics Collector.
      dsetool insights_config --mode ENABLED_WITH_LOCAL_STORAGE
  5. Ensure that the node is listening on port 9103, in addition to the typical DSE ports:
    netstat -lnt
    Active Internet connections (only servers)
    Proto Recv-Q Send-Q Local Address           Foreign Address     State
    tcp        0      0 10.200.176.166:9042     0.0.0.0:*           LISTEN
    tcp        0      0 0.0.0.0:22              0.0.0.0:*           LISTEN
    tcp        0      0 10.200.176.166:7000     0.0.0.0:*           LISTEN
    tcp        0      0 127.0.0.1:7199          0.0.0.0:*           LISTEN
    tcp        0      0 10.200.176.166:8609     0.0.0.0:*           LISTEN
    tcp        0      0 127.0.0.1:39940         0.0.0.0:*           LISTEN
    tcp6       0      0 :::9103                 :::*                LISTEN
    tcp6       0      0 10.200.176.166:61621    :::*                LISTEN
    tcp6       0      0 :::22                   :::*                LISTEN
  6. On the node where you cloned the DSE Metrics Collector Dashboard repository, navigate to the /prometheus directory and modify the tg_dse.json file to include the IP address of each node in your cluster under “targets”. The list of IP addresses is comma-delimited, and each IP address must be enclosed in double quotes.
    vi tg_dse.json
    [
      {
        "targets": [
          "10.100.110.96:9103",
          "10.100.100.95:9103",
          "10.100.100.97:9103"
        ]
      }
    ]
  7. From the directory where you cloned the DSE Metrics Collector repository, start and attach the Docker containers. This command also starts the included Prometheus and Grafana servers.
    docker-compose up
  8. View the Prometheus and Grafana dashboards where DSE Metrics Collector is reporting metrics:
  9. List the name of all Docker containers to obtain the container IDs for the Prometheus and Grafana images:
    docker ps -a
    CONTAINER ID        IMAGE                 COMMAND
    78d3324982ba        grafana/grafana       "/run.sh"
    aadc3d4ad1d1        prom/prometheus       "/bin/prometheus --c…"

Results

The Grafana dashboards display metrics for the defined clusters.