Metric Reporter Dashboards for Cassandra and DSE in Kubernetes

DataStax provides GitHub-hosted templates and build scripts for metrics reporter dashboards using Prometheus and Grafana. The dashboards allow you to check the health of DSE or open-source Apache Cassandra^® resources in a Kubernetes (K8s) cluster. Refer to the hosted start page containing resources for each environment:

Use the git clone command to clone the repo for your environment and follow these steps.

Python scripts for dashboards and configurations

The dashboards and Prometheus and Grafana configuration files are transformed through Python scripts under bin:

bin (Cassandra)
bin (DSE)

Run:

bin/clean.py && bin/build.py

The generated files integrate with the Custom Resources defined by the Prometheus and Grafana operators. These operators are available on Operator Hub.

The Python-generated files are written to the generated directory.

Prometheus Operator setup

The Prometheus Operator handles the orchestration, configuration, and deployment of Kubernetes resources that are required for a High Availability (HA) Prometheus installation. Rather than specifying a list of Cassandra or DSE nodes in a JSON file, this setup directs Prometheus to monitor a Kubernetes Service that exposes all nodes through DNS. This mapping of hosts is handled by a ServiceMonitor Custom Resource defined by the operator.

The following steps illustrate how to install the Prometheus Operator, deploy a service monitor pointed at a Cassandra or DSE cluster (with metric relabeling), and deploy a HA Prometheus cluster connected to the service monitor.

Install the OperatorHub Lifecycle Manager (OLM).

The following command installs a number of custom resource definitions and an OLM operator that handles installing other operators:
```
curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/0.15.1/install.sh | bash -s 0.15.1
```
Install the Prometheus Operator.

The following example pulls down and starts the Prometheus Operator along with a configuration indicating that the operator watches the default namespace for resources it manages. If your installation uses a different namespace, change the values in operator.yaml and re-apply the YAML file to the cluster.

Cassandra example:
```
kubectl create -f dashboards/k8s-build/generated/prometheus/operator.yaml
```
DSE example specifying cass-operator as the namespace:
```
kubectl create -n cass-operator -f k8s-build/generated/prometheus/operator.yaml
```
Configure and install the Service Monitor.

Before installing the Service Monitor, edit the service monitor’s YAML file to include the appropriate labels that match your cluster’s service. For example, if your cluster service has the label

cassandra.datastax.com/cluster: cluster-name

then this mapping is included in the service monitor YAML file under spec.selector.matchLabels. As a convenience, DataStax provided that example in the generated service_monitor.yaml.
For example, to check the labels on your service if the namespace name is cass-operator, run:

kubectl get svc -n cass-operator --show-labels=true
With the service monitor’s configuration file updated, apply the resource to the cluster.

Cassandra example:
```
kubectl apply -f dashboards/k8s-build/generated/prometheus/service_monitor.yaml
```
DSE example:
```
kubectl apply -n cass-operator -f k8s-build/generated/prometheus/service_monitor.yaml
```
Configure and install the Prometheus deployment.

Similar to the service and service monitor mapping described in the previous example, the instance must be provided with a set of labels to match service monitors to Prometheus deployments. The following section in the Prometheus Custom Resource maps the deployment to all service monitors with the label.

cassandra.datastax.com/cluster: cluster-name
```
  serviceMonitorSelector:
    matchLabels:
      cassandra.datastax.com/cluster: cluster-name
```
After making any necessary adjustments for your environment, apply the instance.yaml file to the cluster.

Cassandra example:
```
kubectl apply -f dashboards/k8s-build/generated/prometheus/instance.yaml
```
DSE example:
```
kubectl apply -n cass-operator -f k8s-build/generated/prometheus/instance.yaml
```

Grafana Operator setup

The Grafana Operator handles the orchestration, configuration, and deployment of Kubernetes resources required for a Grafana installation. Instead of configuring datasources, dashboards, and deployments from the graphical UI, everything is configured through YAML files in this repo.

The following steps illustrate how to install the Grafana operator, the data source pointed at the previously deployed Prometheus, a collection of dashboards, and an instance of the Grafana application.

Install the OperatorHub Lifecycle Manager (OLM).

The following command installs a number of custom resource definitions and an OLM operator that handles installing other operators:
```
curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/0.15.1/install.sh | bash -s 0.15.1
```
Install the Grafana Operator.

The following example pulls down and starts the Grafana Operator along with a configuration indicating that the operator watches the default namespace for resources it manages. If your installation uses a different namespace, change the values in operator.yaml and re-apply the YAML file to the cluster.

Cassandra example:
```
kubectl create -f dashboards/k8s-build/generated/grafana/operator.yaml
```
DSE example specifying cass-operator as the namespace:
```
kubectl create -n cass-operator -f k8s-build/generated/grafana/operator.yaml
```
Configure and install the GrafanaDataSource.

Before installing the GrafanaDataSource, edit the datasource YAML file to point at the instantiated Prometheus cluster. If you are not sure what value to use, run the kubectl get svc command and check the output for the Prometheus service name.

With the datasource configuration file updated, apply the resource to the cluster.

Cassandra example:
```
kubectl apply -f dashboards/k8s-build/generated/grafana/datasource.yaml
```
DSE example:
```
kubectl apply -n cass-operator -f k8s-build/generated/grafana/datasource.yaml
```
Configure and install the GrafanaDashboard.

Before installing the GrafanaDashboard , edit the YAML with appropriate labels. This example uses a label of app=grafana. If needed, modify for your environment.

With the configuration file updated, apply the resource to the cluster.

Cassandra example:
```
kubectl apply -f dashboards/k8s-build/generated/grafana/
```
DSE example:
```
kubectl apply -n cass-operator -f k8s-build/generated/grafana/
```
Configure and install the Grafana deployment.

The deployment must be informed of the labels to check for matching dashboards. The following section in the Grafana Custom Resource maps the deployment to all dashboards with the label app=grafana.
```
  dashboardLabelSelector:
    - matchExpressions:
        - key: app
          operator: In
          values:
            - grafana
```
After the adjustments, apply the instance.yaml file to the cluster.

Cassandra example:
```
kubectl apply -f dashboards/k8s-build/generated/grafana/instance.yaml
```
DSE example:
```
kubectl apply -n cass-operator -f k8s-build/generated/grafana/instance.yaml
```
Check the Grafana instance.

Port forward the grafana instance and review it in the browser http://127.0.0.1:3000/ (username: admin, password: secret).

Never use documented credentials in production environments!

Cassandra example:
```
kubectl port-forward svc/grafana-service 3000
```
DSE example:
```
kubectl port-forward -n cass-operator svc/grafana-service 3000
```

What’s next?

Learn about repair options for Apache Cassandra clusters in Kubernetes.