Exporting metrics using preconfigured Docker images
Use the preconfigured Docker images provided by DataStax to quickly start exporting and visualizing metrics for DSE clusters.
To quickly get started with DSE Metrics Collector, download and install Docker, which provides the ability to run DataStax preconfigured dashboards. If you already have a monitoring and visualization solution, see Exporting metrics with an existing Prometheus server.
Warning: Before using the Docker examples provided in this repository, take steps to
avoid the loss of metrics data. By default, Prometheus stores data in local storage
for only 15 days. You can configure Prometheus's default data retention to keep a longer history of your
metrics data. Revise
docker-compose.yml
to add the
--storage.tsdb.retention.time
flag on the Prometheus runtime
command line. Specify an appropriate value for your environment.Tip: For clusters managed by Lifecycle Manager (LCM) 6.7 and later, you can
use LCM to export metrics collection to Prometheus,
which replaces steps 1-3
in this procedure.
Prerequisites
- If necessary, download and install DataStax Enterprise (DSE 6.7.0 or later | 6.0.5 or later | 5.1.14 or later).
- On a host where DSE is not installed:
Procedure
-
On the DSE cluster that DSE Metrics Collector reads from, create a
/collectd directory:
- Package installations: Create the /etc/dse/collectd directory on all nodes in the cluster and set the required user permissions.
- Tarball installations: The installation_location/resources/dse/collectd/etc/collectd exists by default.
-
Configure the Prometheus server from which the DSE Metrics Collector
reads.
-
Ensure collectd is listening on port 9103, in addition to the typical DSE
ports:
netstat -lnt
Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 10.100.176.166:9042 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp 0 0 10.100.176.166:7000 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:7199 0.0.0.0:* LISTEN tcp 0 0 10.100.176.166:8609 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:39940 0.0.0.0:* LISTEN tcp6 0 0 :::9103 :::* LISTEN tcp6 0 0 10.100.176.166:61621 :::* LISTEN tcp6 0 0 :::22 :::* LISTEN
Note: If collectd is not listening on port 9103, ensure libaio is installed and Epoll is working. The collectd daemon will not start if Epoll is not working. -
On the node where you cloned the DSE Metrics Collector Dashboard repository,
modify repo_location/prometheus/tg_dse.json file to
include the IP address of each node in your cluster under
"targets"
.[ { "targets": [ "127.0.0.1:9103", "10.101.34.229:9103" "10.101.34.250:9013" ], "labels": { "cluster": "test_cluster" } } ]
The list of IP addresses is comma-delimited. Enclose each IP address in double quotation marks. -
From the cloned directory containing the DSE Metrics Collector repository,
attach the Docker containers.
-
List the name of all Docker containers to obtain the container IDs for the
Prometheus and Grafana images:
docker ps -a
CONTAINER ID IMAGE COMMAND 78d3324982ba grafana/grafana "/run.sh" aadc3d4ad1d1 prom/prometheus "/bin/prometheus --c…"
-
View the Prometheus and Grafana dashboards where DSE Metrics Collector is
reporting metrics:
- To view the Prometheus dashboard, which displays each DSE node as a target, open a browser and navigate to http://prometheus_server_IP_address:9090, where prometheus_server_IP_address is the IP address where the Prometheus server is running, and 9090 is the default port where Prometheus runs.
- To view the Grafana dashboards that are linked to local Prometheus data source, open a browser and navigate to http://localhost:3000/dashboards.
Results
