Metric reporter dashboards for Cassandra and DSE in Kubernetes

DataStax provides GitHub-hosted templates and build scripts for metrics reporter dashboards using Prometheus and Grafana. The dashboards allow you to check the health of DSE or open-source Apache Cassandra® resources in a Kubernetes cluster. Refer to the hosted start page containing resources for each environment:

Use git clone to clone the repo for your environment and follow the steps in this topic.

Python scripts for dashboards and configurations

The dashboards plus Prometheus and Grafana configuration files are transformed via Python scripts under bin:

bin (Cassandra)
bin (DSE)

Run:

bin/clean.py && bin/build.py

The generated files will integrate with the Custom Resources defined by the Prometheus and Grafana operators that are available on Operator Hub.

Note: The Python-generated files are written to the generated directory.

Prometheus Operator setup

The Prometheus Operator handles the orchestration, configuration, and deployment of Kubernetes resources required for a High Availability (HA) Prometheus installation. Rather than specifying a list of Cassandra or DSE nodes in a JSON file, this setup directs Prometheus to monitor a Kubernetes Service that exposes all nodes via DNS. This mapping of hosts is handled by a ServiceMonitor Custom Resource defined by the operator.

The following steps illustrate how to install the Prometheus Operator, deploy a service monitor pointed at a Cassandra or DSE cluster (with metric relabeling), and deploy a HA Prometheus cluster connected to the service monitor.

Install the OperatorHub Lifecycle Manager (OLM)
The following command installs a number of custom resource definitions and an OLM operator that handles installing other operators:
```
curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/0.15.1/install.sh | bash -s 0.15.1
```
Install the Prometheus Operator
The following example pulls down and starts the Prometheus Operator along with a configuration indicating it should watch the default namespace for resources it manages. If your installation uses a different namespace, change the values in operator.yaml and re-apply it to the cluster.
Cassandra example:
```
kubectl create -f dashboards/k8s-build/generated/prometheus/operator.yaml
```
DSE example specifying cass-operator as the namespace:
```
kubectl create -n cass-operator -f k8s-build/generated/prometheus/operator.yaml
```
Configure and install the Service Monitor
Before installing the Service Monitor, edit the service monitor's YAML to include the appropriate labels that match your cluster's service. For example if your cluster service has the label cassandra.datastax.com/cluster: cluster-name, this mapping would be included in the service monitor YAML under spec.selector.matchLabels. As a convenience, DataStax provided that example in the generated service_monitor.yaml.
Tip: For example, to check the labels on your service if the namespace name is cass-operator:
```
kubectl get svc -n cass-operator --show-labels=true
```
With the service monitor's configuration file updated, apply the resource to the cluster.
Cassandra example:
```
kubectl apply -f dashboards/k8s-build/generated/prometheus/service_monitor.yaml
```
DSE example:
```
kubectl apply -n cass-operator -f k8s-build/generated/prometheus/service_monitor.yaml
```
Configure and install the Prometheus deployment
Similar to the service and service monitor mapping above, the instance must be provided with a set of labels to match service monitors to Prometheus deployments. The following section in the Prometheus Custom Resource maps the deployment to all service monitors with the label cassandra.datastax.com/cluster: cluster-name
```
  serviceMonitorSelector:
    matchLabels:
      cassandra.datastax.com/cluster: cluster-name 
```
After making any necessary adjustments for your environment, apply the instance.yaml file to the cluster.
Cassandra example:
```
kubectl apply -f dashboards/k8s-build/generated/prometheus/instance.yaml
```
DSE example:
```
kubectl apply -n cass-operator -f k8s-build/generated/prometheus/instance.yaml
```

Grafana Operator setup

The Grafana Operator handles the orchestration, configuration, and deployment of Kubernetes resources required for a Grafana installation. Instead of configuring datasources, dashboards, and deployments via the graphical UI, everything is configured via YAML files in this repo.

The following steps illustrate how to install the Grafana operator, data source pointed at the previously deployed Prometheus, a collection of dashboards, and an instance of the Grafana application.

Install the OperatorHub Lifecycle Manager (OLM)
The following command installs a number of custom resource definitions and an OLM operator that handles installing other operators:
```
curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/0.15.1/install.sh | bash -s 0.15.1
```
Install the Grafana Operator
The following example pulls down and starts the Grafana Operator along with a configuration indicating it should watch the default namespace for resources it manages. If your installation uses a different namespace, change the values in operator.yaml and re-apply it to the cluster.
Cassandra example:
```
kubectl create -f dashboards/k8s-build/generated/grafana/operator.yaml
```
DSE example specifying cass-operator as the namespace:
```
kubectl create -n cass-operator -f k8s-build/generated/grafana/operator.yaml
```
Configure and install the GrafanaDataSource
Before installing the GrafanaDataSource, edit the datasource YAML to point at the instantiated Prometheus cluster. If you are not sure what value to use, check the output via kubectl get svc for the Prometheus service name.
With the datasource configuration file updated, apply the resource to the cluster.
Cassandra example:
```
kubectl apply -f dashboards/k8s-build/generated/grafana/datasource.yaml
```
DSE example:
```
kubectl apply -n cass-operator -f k8s-build/generated/grafana/datasource.yaml
```
Configure and install the GrafanaDashboard
Before installing the GrafanaDashboard , edit the YAML with appropriate labels. In this example, a label of app=grafana is used. If needed, modify for your environment.
With the configuration file updated, apply the resource to the cluster.
Cassandra example:
```
kubectl apply -f dashboards/k8s-build/generated/grafana/
```
DSE example:
```
kubectl apply -n cass-operator -f k8s-build/generated/grafana/
```
Configure and install the Grafana deployment
The deployment must be informed of the labels to check for matching dashboards. The following section in the Grafana Custom Resource maps the deployment to all dashboards with the label app=grafana
```
  dashboardLabelSelector:
    - matchExpressions:
        - key: app
          operator: In
          values:
            - grafana
```
After adjustments apply the instance.yaml file to the cluster.
Cassandra example:
```
kubectl apply -f dashboards/k8s-build/generated/grafana/instance.yaml
```
DSE example:
```
kubectl apply -n cass-operator -f k8s-build/generated/grafana/instance.yaml
```
Check the Grafana instance
Port forward to the grafana instance and check it out at http://127.0.0.1:3000/ (username: admin, password: secret)
CAUTION: Never use documented credentials in production environments!
Cassandra example:
```
kubectl port-forward svc/grafana-service 3000
```
DSE example:
```
kubectl port-forward -n cass-operator svc/grafana-service 3000
```

What's next?

Learn about repair options for Apache Cassandra clusters in Kubernetes.