Exporting metrics with an existing Prometheus server

Use an existing Prometheus server to export metrics.

1. On the server where Prometheus is running, ensure the prometheus.yml and tg_dse.json files are collocated in the same directory.
2. On the DSE cluster that DSE Metrics Collector reads from, create a /collectd directory:
   • Package installations: Create the /etc/dse/collectd directory on all nodes in the cluster and set the required user permissions.
   • Tarball installations: The installation_location/resources/dse/collectd/etc/collectd exists by default.
3. Configure the Prometheus server from which the DSE Metrics Collector reads.
   1. Create a prometheus.conf file in the collectd directories:

      sudo touch prometheus.conf

   2. Define the Prometheus plugin:

      LoadPlugin write_prometheus
      
      <Plugin write_prometheus>
       Port "9103"
      </Plugin>

   3. To propagate the changes, disable and then re-enable DSE Metrics Collector on any node in the cluster:

      dsetool insights_config --mode DISABLED

      Note: Disabling DSE Metrics Collector can take up to 30 seconds to propagate across the cluster. Wait at least 30 seconds before re-enabling DSE Metrics Collector.

      dsetool insights_config --mode ENABLED_WITH_LOCAL_STORAGE

4. Ensure collectd is listening on port 9103, in addition to the typical DSE ports:

   netstat -lnt

   Active Internet connections (only servers)
   Proto Recv-Q Send-Q Local Address           Foreign Address     State
   tcp        0      0 10.100.176.166:9042     0.0.0.0:*           LISTEN
   tcp        0      0 0.0.0.0:22              0.0.0.0:*           LISTEN
   tcp        0      0 10.100.176.166:7000     0.0.0.0:*           LISTEN
   tcp        0      0 127.0.0.1:7199          0.0.0.0:*           LISTEN
   tcp        0      0 10.100.176.166:8609     0.0.0.0:*           LISTEN
   tcp        0      0 127.0.0.1:39940         0.0.0.0:*           LISTEN
   tcp6       0      0 :::9103                 :::*                LISTEN
   tcp6       0      0 10.100.176.166:61621    :::*                LISTEN
   tcp6       0      0 :::22                   :::*                LISTEN

   Note: If collectd is not listening on port 9103, ensure libaio is installed and Epoll is working. The collectd daemon will not start if Epoll is not working.
5. Modify the downloaded tg_dse.json file to include the IP address of each node in the cluster under "targets".
   The list of IP addresses is comma-delimited. Enclose each IP address in double quotation marks.
6. Start Prometheus with the --config option to specify the prometheus.yml file you downloaded from the DSE Metrics Collector Dashboards repository.
   For example, if you saved the prometheus.yml file in the /etc/dse/tmp directory, the command would look like this:

   ./prometheus --config.file=/etc/dse/tmp/prometheus.yml

7. Verify the Prometheus and Grafana targets:
   1. To view the Prometheus dashboard, which displays each DSE node as a target, open a browser and navigate to http://prometheus_server_IP_address:9090, where prometheus_server_IP_address is the IP address where the Prometheus server is running, and 9090 is the default port where Prometheus runs.
   2. To view the targets that Prometheus is monitoring, open a browser and navigate to http://prometheus_server_IP_address:9090/targets. Each of the nodes specified in the tg_dse.json file displays as an endpoint, with the current status and last scrape time.