Metric Reporter Dashboards for Cassandra and DSE in Kubernetes

DataStax provides GitHub-hosted templates and build scripts for metrics reporter dashboards using Prometheus and Grafana. The dashboards allow you to check the health of DSE or open-source Apache Cassandra® resources in a Kubernetes (K8s) cluster. Refer to the hosted start page containing resources for each environment:

Use the git clone command to clone the repo for your environment and follow these steps.

Python scripts for dashboards and configurations

The dashboards and Prometheus and Grafana configuration files are transformed through Python scripts:

Cassandra Metrics Collector scripts (Cassandra)
DSE Metrics Reporter scripts

Run the scripts for your database type:

bin/clean.py && bin/build.py

The generated files integrate with each custom resource (CR) defined by the Prometheus Operator and Grafana Operator. These operators are available on Operator Hub.

The Python-generated files are written to the generated directory.

Prometheus Operator setup

The Prometheus Operator handles the orchestration, configuration, and deployment of Kubernetes resources that are required for a High Availability (HA) Prometheus installation. Rather than specifying a list of Cassandra or DSE nodes in a JSON file, this setup directs Prometheus to monitor a Kubernetes Service that exposes all nodes through DNS. This mapping of hosts is handled by a ServiceMonitor CR defined by the Cassandra Operator.

The following steps illustrate how to install the Prometheus Operator, deploy a service monitor pointed at a Cassandra or DSE cluster (with metric relabeling), and deploy a HA Prometheus cluster connected to the service monitor.

Install the OperatorHub Lifecycle Manager (OLM).

The following command installs a number of CRs and an OLM operator that handles installing other operators:
```
curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/0.15.1/install.sh | bash -s 0.15.1
```
Install the Prometheus Operator.

The following example pulls down and starts the Prometheus Operator along with a configuration indicating that the Cassandra Operator watches the default namespace for resources it manages. If your installation uses a different namespace, change the values in operator.yaml and re-apply the YAML file to the cluster.
- Cassandra example:
  kubectl create -f dashboards/k8s-build/generated/prometheus/operator.yaml
- DSE example, specifying the cass-operator namespace:
  kubectl create -n cass-operator -f k8s-build/generated/prometheus/operator.yaml
Configure and install the Service Monitor:
1. Edit the Service Monitor’s YAML file to include the appropriate labels that match your cluster’s service.
  
  For example, if your cluster service has the label cassandra.datastax.com/cluster: cluster-name, then this mapping is included in the service monitor YAML file under spec.selector.matchLabels. As a convenience, DataStax provided that example in the generated service_monitor.yaml.
  
  You can use kubectl to check the labels on your service. For example, using the cass-operator namespace:
  kubectl get svc -n cass-operator --show-labels=true
2. Apply the Service Monitor resource to the cluster:
  - Cassandra example:
    
    kubectl apply -f dashboards/k8s-build/generated/prometheus/service_monitor.yaml
  - DSE example:
    
    kubectl apply -n cass-operator -f k8s-build/generated/prometheus/service_monitor.yaml
Configure and install the Prometheus deployment.

Similar to the service and service monitor mapping in the previous steps, the instance must be provided with a set of labels to match service monitors to Prometheus deployments.

The following section in the Prometheus CR maps the deployment to all service monitors with the label cassandra.datastax.com/cluster: cluster-name:
```
  serviceMonitorSelector:
    matchLabels:
      cassandra.datastax.com/cluster: cluster-name
```
After making any necessary adjustments for your environment, apply the instance.yaml file to the cluster:
- Cassandra example:
  kubectl apply -f dashboards/k8s-build/generated/prometheus/instance.yaml
- DSE example:
  kubectl apply -n cass-operator -f k8s-build/generated/prometheus/instance.yaml

Grafana Operator setup

The Grafana Operator handles the orchestration, configuration, and deployment of Kubernetes resources required for a Grafana installation. Instead of configuring datasources, dashboards, and deployments from the graphical UI, everything is configured through YAML files in this repo.

The following steps illustrate how to install the Grafana operator, the data source pointed at the previously deployed Prometheus, a collection of dashboards, and an instance of the Grafana application.

Install the OperatorHub Lifecycle Manager (OLM).

The following command installs a number of CRs and an OLM operator that handles installing other operators:
```
curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/0.15.1/install.sh | bash -s 0.15.1
```
Install the Grafana Operator.

The following example pulls down and starts the Grafana Operator along with a configuration indicating that the Cassandra Operator watches the default namespace for resources it manages. If your installation uses a different namespace, change the values in operator.yaml and re-apply the YAML file to the cluster.
- Cassandra example:
  kubectl create -f dashboards/k8s-build/generated/grafana/operator.yaml
- DSE example, specifying the cass-operator namespace:
  kubectl create -n cass-operator -f k8s-build/generated/grafana/operator.yaml
Configure and install the GrafanaDataSource:
1. Edit the datasource YAML file to point at the instantiated Prometheus cluster.
  
  If you don’t know which value to use, run kubectl get svc, and then get the Prometheus service name from the output.
2. After editing the datasource configuration file, apply the resource to the cluster:
  - Cassandra example:
    
    kubectl apply -f dashboards/k8s-build/generated/grafana/datasource.yaml
  - DSE example:
    
    kubectl apply -n cass-operator -f k8s-build/generated/grafana/datasource.yaml
Configure and install the GrafanaDashboard:
1. Edit the YAML file with appropriate labels.
  
  The following examples use the label app=grafana. If needed, modify for your environment.
2. Apply the resource to the cluster:
  - Cassandra example:
    
    kubectl apply -f dashboards/k8s-build/generated/grafana/
  - DSE example:
    
    kubectl apply -n cass-operator -f k8s-build/generated/grafana/
Configure and install the Grafana deployment.

The deployment must be informed of the labels to check for matching dashboards. The following section in the Grafana CR maps the deployment to all dashboards with the label app=grafana:
```
  dashboardLabelSelector:
    - matchExpressions:
        - key: app
          operator: In
          values:
            - grafana
```
Apply the instance.yaml file to the cluster:
- Cassandra example:
  kubectl apply -f dashboards/k8s-build/generated/grafana/instance.yaml
- DSE example:
  kubectl apply -n cass-operator -f k8s-build/generated/grafana/instance.yaml
To check the Grafana instance, port forward the grafana instance, and then view it in a browser.

These examples assume the instance is available at http://127.0.0.1:3000/ with the default credentials. Don’t use the default credentials in production.
- Cassandra example:
  kubectl port-forward svc/grafana-service 3000
- DSE example:
  kubectl port-forward -n cass-operator svc/grafana-service 3000

Metric Reporter Dashboards for Cassandra and DSE in Kubernetes

Python scripts for dashboards and configurations

Prometheus Operator setup

Grafana Operator setup

See also

Was this helpful?

Give Feedback