Exporting metrics with an existing Prometheus server

Use an existing Prometheus server to export metrics.

After configuring DSE Metrics Collector, you can export metrics to an existing Prometheus server. These steps use Prometheus as the monitoring solution and Grafana as the dashboard. However, other monitoring and visualization tools can be used with DSE Metrics Collector.

If you already have a Prometheus server running, this method of exporting and visualizing metrics enables use of the existing server.

Prerequisites

  1. If necessary, Download and install DataStax Enterprise (DSE) 5.1.14 or later.
  2. Download and install Prometheus, which is used as the monitoring solution, or use an existing Prometheus server.
  3. Download and install Grafana, which is used as the visualization dashboard.
  4. Download the prometheus.yml file from GitHub and save it in a directory on the server where Prometheus is running. The prometheus.yml file is preconfigured to scrape metrics reported by DSE Metrics Collector. Each metric uses a specific regex for the metric type.
  5. Download the tg_dse.json file and save it in the same directory as the prometheus.yml file. The prometheus.yml file references this file to obtain the IP addresses for each cluster node.
  6. DataStax provides preconfigured Grafana dashboards. Download the prometheus.yml file from GitHub and save it in a directory on the server where Prometheus is running.

    The prometheus.yml file is preconfigured to scrape metrics reported by DSE Metrics Collector. Each metric uses a specific regex for the metric type.

  7. Download the tg_dse.json file and save it in the same directory as the prometheus.yml file.

    The prometheus.yml file references this file to obtain the IP addresses for each cluster node.

Procedure

  1. On the server where Prometheus is running, ensure the prometheus.yml and tg_dse.json files are collocated in the same directory.
  2. On the DSE cluster that DSE Metrics Collector reads from, create a /collectd directory:
    • Package installations: Create the /etc/dse/collectd directory on all nodes in the cluster and set the required user permissions.
    • Tarball installations: The installation_location/resources/dse/collectd/etc/collectd exists by default.
  3. To enable the Prometheus server, create a configuration file on a DSE cluster that DSE Metrics Collector reads from.
    1. Create a prometheus.conf file in the collectd directories you created in the previous step:
      sudo touch prometheus.conf
    2. Define the Prometheus plugin:
      LoadPlugin write_prometheus
      <Plugin write_prometheus>
      Port “9103”
      </Plugin>
    3. On any node in the cluster, disable and re-enable DSE Metrics Collector to propagate the changes:
      dsetool insights_config --mode DISABLED
      Note: Disabling DSE Metrics Collector can take up to 30 seconds to propagate across the cluster. Wait at least 30 seconds before re-enabling DSE Metrics Collector.
      dsetool insights_config --mode ENABLED_WITH_LOCAL_STORAGE
  4. Ensure that the node is listening on port 9103, in addition to the typical DSE ports:
    netstat -lnt
    Note: If you configured Prometheus but port 9103 is not listening, read Error messages when libaio not installed and ensure that libaio is installed and Epoll is working. The collectd daemon will not start if Epoll is not working.
    Active Internet connections (only servers)
    Proto Recv-Q Send-Q Local Address           Foreign Address     State
    tcp        0      0 10.200.176.166:9042     0.0.0.0:*           LISTEN
    tcp        0      0 0.0.0.0:22              0.0.0.0:*           LISTEN
    tcp        0      0 10.200.176.166:7000     0.0.0.0:*           LISTEN
    tcp        0      0 127.0.0.1:7199          0.0.0.0:*           LISTEN
    tcp        0      0 10.200.176.166:8609     0.0.0.0:*           LISTEN
    tcp        0      0 127.0.0.1:39940         0.0.0.0:*           LISTEN
    tcp6       0      0 :::9103                 :::*                LISTEN
    tcp6       0      0 10.200.176.166:61621    :::*                LISTEN
    tcp6       0      0 :::22                   :::*                LISTEN
  5. Modify the downloaded tg_dse.json file to include the IP address of each node in the cluster under "targets".
    The list of IP addresses is comma-delimited. Enclose each IP address in double quotation marks.
  6. Start Prometheus with the --config option to specify the prometheus.yml file you downloaded from the DSE Metrics Collector Dashboards repository.
    For example, if you saved the prometheus.yml file in the /etc/dse/tmp directory, the command would look like this:
    ./prometheus --config.file=/etc/dse/tmp/prometheus.yml
  7. Verify the Prometheus and Grafana targets:
    1. To view the Prometheus dashboard, which displays each DSE node as a target, open a browser and navigate to http://prometheus_server_IP_address:9090, where prometheus_server_IP_address is the IP address where the Prometheus server is running, and 9090 is the default port where Prometheus runs.
    2. To view the targets that Prometheus is monitoring, open a browser and navigate to http://prometheus_server_IP_address:9090/targets. Each of the nodes specified in the tg_dse.json file displays as an endpoint, with the current status and last scrape time.

What's next

Create a Grafana dashboard using the Prometheus data source

Creating a Grafana dashboard using the Prometheus data source

Steps to create a Grafana dashboard when an existing Prometheus server is used to export DSE metrics.

You can create a Grafana dashboard to visualize metrics. The Prometheus plugin is included with Grafana. These steps apply when an existing Prometheus server exports DSE metrics.

Prerequisites

Complete the required configuration in Exporting metrics with an existing Prometheus server.

Procedure

  1. Start Grafana and enable the Prometheus plugin, which is included with Grafana by default.
    Note: You must have admin privileges for your organization to add data sources in Grafana.
    1. Log in to your Grafana instance at http://grafana_server_IP_address:3000, where grafana_server_IP_address is the IP address where your Grafana server is running.
    2. Open the side menu by selecting the Grafana icon in the upper left.
    3. In the side menu, select Configuration > Data Sources.
    4. On the Configuration page, select Add data source.
    5. Enter a name for your data source.
    6. Select Prometheus as the data source Type.
    7. Enter the URL where your Prometheus server is running.
    8. Select Server or Browser, depending on where your Prometheus instance is running.
    9. Select Save & Test.
  2. Create a Grafana dashboard using the Prometheus data source.
    1. Log in to your Grafana instance.
    2. In the side menu, select Create > Import.
    3. Select Upload .json file and select one of the preconfigured Grafana dashboard JSON files you downloaded earlier.
      For example, dse-cluster-condensed.json.
    4. Enter a name for the dashboard and select Import.

Results

The dashboard displays in your Grafana instance.