Exporting metrics using preconfigured Docker images

Use the preconfigured Docker images provided by DataStax to quickly start exporting and visualizing metrics for DSE clusters.

To quickly get started with DSE Metrics Collector, download and install Docker, which provides the ability to run DataStax preconfigured dashboards. If you already have a monitoring and visualization solution, see Exporting metrics with an existing Prometheus server.

Warning: Before using the Docker examples provided in this repository, take steps to avoid the loss of metrics data. By default, Prometheus stores data in local storage for only 15 days. You can configure Prometheus's default data retention to keep a longer history of your metrics data. Revise docker-compose.yml to add the --storage.tsdb.retention.time flag on the Prometheus runtime command line. Specify an appropriate value for your environment.

Prerequisites

  1. If necessary, Download and install DataStax Enterprise (DSE) 5.1.14 or later.
  2. On a node where DSE is not installed:
Note: You must install Docker, Docker Compose, and the DSE Metrics Collector Dashboards repository on a node where DSE is not installed.

Procedure

  1. On your DSE cluster, check the configuration for DSE Metrics Collector:
    dsetool insights_config --show_config
    {
      "mode" : "DISABLED",
      "config_refresh_interval_in_seconds" : 30,
      "metric_sampling_interval_in_seconds" : 30,
      "data_dir_max_size_in_mb" : 1024,
      "node_system_info_report_period" : "PT1H"
    }
    If the mode is not enabled, then enable DSE Metrics Collector.
  2. On the DSE cluster that DSE Metrics Collector reads from, create a /collectd directory:
    • Package installations: Create the /etc/dse/collectd directory on all nodes in the cluster and set the required user permissions.
    • Tarball installations: The installation_location/resources/dse/collectd/etc/collectd exists by default.
  3. On the server where Prometheus is running, ensure the prometheus.yml and tg_dse.json files are collocated in the same directory.
  4. To enable the Prometheus server, create a configuration file on a DSE cluster that DSE Metrics Collector reads from.
    1. Create a prometheus.conf file in the collectd directories you created in the previous step:
      sudo touch prometheus.conf
    2. Define the Prometheus plugin:
      LoadPlugin write_prometheus
      <Plugin write_prometheus>
      Port “9103”
      </Plugin>
    3. On any node in the cluster, disable and re-enable DSE Metrics Collector to propagate the changes:
      dsetool insights_config --mode DISABLED
      Note: Disabling DSE Metrics Collector can take up to 30 seconds to propagate across the cluster. Wait at least 30 seconds before re-enabling DSE Metrics Collector.
      dsetool insights_config --mode ENABLED_WITH_LOCAL_STORAGE
  5. Ensure that the node is listening on port 9103, in addition to the typical DSE ports:
    netstat -lnt
    Active Internet connections (only servers)
    Proto Recv-Q Send-Q Local Address           Foreign Address     State
    tcp        0      0 10.100.176.166:9042     0.0.0.0:*           LISTEN
    tcp        0      0 0.0.0.0:22              0.0.0.0:*           LISTEN
    tcp        0      0 10.100.176.166:7000     0.0.0.0:*           LISTEN
    tcp        0      0 127.0.0.1:7199          0.0.0.0:*           LISTEN
    tcp        0      0 10.100.176.166:8609     0.0.0.0:*           LISTEN
    tcp        0      0 127.0.0.1:39940         0.0.0.0:*           LISTEN
    tcp6       0      0 :::9103                 :::*                LISTEN
    tcp6       0      0 10.100.176.166:61621    :::*                LISTEN
    tcp6       0      0 :::22                   :::*                LISTEN
    Note: If you configured Prometheus but port 9103 is not listening, ensure that libaio is installed and Epoll is working. The collectd daemon will not start if Epoll is not working.
  6. On the node where you cloned the DSE Metrics Collector Dashboard repository, navigate to the /prometheus directory and modify the tg_dse.json file to include the IP address of each node in your cluster under "targets".
    The list of IP addresses is comma-delimited. Enclose each IP address in double quotation marks.
  7. From the cloned directory containing the DSE Metrics Collector repository, start and attach the Docker containers:
    docker-compose up
    This command also starts the included Prometheus and Grafana servers.
  8. View the Prometheus and Grafana dashboards where DSE Metrics Collector is reporting metrics:
    1. To view the Prometheus dashboard, which displays each DSE node as a target, open a browser and navigate to http://prometheus_server_IP_address:9090, where prometheus_server_IP_address is the IP address where the Prometheus server is running, and 9090 is the default port where Prometheus runs.
    2. To view the Grafana dashboards that are linked to local Prometheus data source, open a browser and navigate to http://localhost:3000/dashboards.
  9. List the name of all Docker containers to obtain the container IDs for the Prometheus and Grafana images:
    docker ps -a
    CONTAINER ID        IMAGE                 COMMAND
    78d3324982ba        grafana/grafana       "/run.sh"
    aadc3d4ad1d1        prom/prometheus       "/bin/prometheus --c…"

Results

The Grafana dashboards display metrics for the defined clusters.