Manually exporting and visualizing metrics with Prometheus

Use an existing Prometheus server to export metrics and configure Grafana to visualize metrics in a dashboard.

After configuring DSE Metrics Collector, you can export metrics and visualize them in a dashboard. In the following examples, Prometheus is the monitoring solution and Grafana is used as the dashboard. However, other monitoring and visualization tools can be used with DSE Metrics Collector.

If you already have a Prometheus server running, this method of exporting and visualizing metrics enables use of the existing server.

Prerequisites

  1. If necessary, Download and install DSE 6.0.5 or later.
  2. Download and install Prometheus, which is used as the monitoring solution, or use a Prometheus server that is already running.
  3. Download and install Grafana, which is used as the visualization dashboard.
  4. Download the prometheus.yaml file from GitHub and save it in a directory on the server where Prometheus is running. The prometheus.yaml file is preconfigured to scrape metrics reported by DSE Metrics Collector. Each metric uses a specific regex for the metric type.
  5. Download the tg_dse.json file and save it in the same directory as the prometheus.yaml file. The prometheus.yaml file references this file to obtain the IP addresses for each cluster node.
  6. Download the JSON files for the preconfigured Grafana dashboards provided by DataStax. Place these files in a local directory that is accessible.
Quick check

On the server where Prometheus is running, the prometheus.yaml file and the tg_dse.json file are collocated in the same directory.

Procedure

  1. To enable the Prometheus server, create a configuration file on a DataStax Enterprise (DSE) cluster that DSE Metrics Collector reads from.
    1. Create a prometheus.conf file in one of the following directories:
      • Tarball: installation_location/resources/dse/collectd/etc/collectd
      • Package: /etc/dse/collectd
      sudo touch prometheus.conf
      Tip: If the /collectd directory does not exist, create it, and then create the prometheus.conf file.
    2. Add the following code to the prometheus.conf configuration file and save it:
      vi prometheus.conf
      LoadPlugin write_prometheus
      
      <Plugin write_prometheus>
       Port “9103”
      </Plugin>
    3. On any node in the cluster, disable and re-enable DSE Metrics Collector to propagate the changes:
      dsetool insights_config --mode DISABLED
      Note: Disabling DSE Metrics Collector can take up to 30 seconds to propagate across the cluster. Wait at least 30 seconds before re-enabling DSE Metrics Collector.
      dsetool insights_config --mode ENABLED_WITH_LOCAL_STORAGE
  2. Modify the downloaded tg_dse.json file to include the IP address of each node in the cluster under “targets”. The list of IP addresses is comma-delimited, and each IP address must be enclosed in double quotes.
    vi tg_dse.json
    [
      {
        "targets": [
          "10.100.110.96:9103",
          "10.100.100.95:9103",
          "10.100.100.97:9103"
        ]
      }
    ]
  3. Start Prometheus with the --config option to specify the prometheus.yaml file you downloaded from the DSE Metrics Collector Dashboards repository. For example, if you saved the prometheus.yaml file in the /etc/dse/tmp directory, the command would look like this:
    ./prometheus --config.file=/etc/dse/tmp/prometheus.yaml
  4. Verify the Promtheus and Grafana targets:
    1. Open a browser and navigate to http://prometheus_server_IP_address:9090, where prometheus_server_IP_address is the IP address where the Prometheus server is running, and 9090 is the default port where Prometheus runs.
    2. Navigate to http://prometheus_server_IP_address:9090/targets to view the targets that Prometheus is monitoring. Each of the nodes specified in the tg_dse.json file displays as an endpoint, with the current status and last scrape time.
  5. After verifying that Prometheus is scraping the endpoints where DSE Metrics Collector is running, start Grafana and enable the Prometheus plugin, which is included with Grafana by default (separate installation not required).
    Note: You must have Admin privileges for your organization to add data sources in Grafana.
    1. Log in to your Grafana instance at http://grafana_server_IP_address:3000, where grafana_server_IP_address is the IP address where your Grafana server is running.
    2. Open the side menu by clicking the Grafana icon in the upper left.
    3. In the side menu, select Configuration > Data Sources.
    4. On the Configuration page, click Add data source.
    5. Enter a name for your data source.
    6. Select Prometheus as the data source Type.
    7. Enter the URL where your Prometheus server is running.
    8. Select Server or Browser, depending on where your Prometheus instance is running.
    9. Click Save & Test.
  6. Create a Grafana dashboard using the Prometheus data source.
    1. Log in to your Grafana instance.
    2. In the side menu, click Create > Import.
    3. Click Upload .json file and select the one of the JSON files you downloaded. For example, dse-cluster-condensed.json.
    4. Enter a name for the dashboard and click Import.

Results

The dashboard displays in your Grafana instance.