Metric reporter dashboards for Cassandra and DSE in Kubernetes
Metric reporter dashboards for Cassandra and DSE in Kubernetes
git clone
to clone the repo for your environment and follow the
steps in this topic.Python scripts for dashboards and configurations
bin
: bin/clean.py && bin/build.pyThe generated files will integrate with the Custom Resources defined by the Prometheus and Grafana operators that are available on Operator Hub.
generated
directory.Prometheus Operator setup
The Prometheus Operator handles the orchestration, configuration, and deployment of
Kubernetes resources required for a High Availability (HA) Prometheus installation. Rather
than specifying a list of Cassandra or DSE nodes in a JSON file, this setup directs
Prometheus to monitor a Kubernetes Service that exposes all nodes via DNS. This mapping of
hosts is handled by a ServiceMonitor
Custom Resource defined by the
operator.
The following steps illustrate how to install the Prometheus Operator, deploy a service monitor pointed at a Cassandra or DSE cluster (with metric relabeling), and deploy a HA Prometheus cluster connected to the service monitor.
- Install the OperatorHub Lifecycle Manager (OLM)
The following command installs a number of custom resource definitions and an OLM operator that handles installing other operators:
curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/0.15.1/install.sh | bash -s 0.15.1
- Install the Prometheus Operator
The following example pulls down and starts the Prometheus Operator along with a configuration indicating it should watch the
default
namespace for resources it manages. If your installation uses a different namespace, change the values in operator.yaml and re-apply it to the cluster.Cassandra example:kubectl create -f dashboards/k8s-build/generated/prometheus/operator.yaml
DSE example specifyingcass-operator
as the namespace:kubectl create -n cass-operator -f k8s-build/generated/prometheus/operator.yaml
- Configure and install the Service Monitor
Before installing the Service Monitor, edit the service monitor's YAML to include the appropriate labels that match your cluster's service. For example if your cluster service has the label
cassandra.datastax.com/cluster: cluster-name
, this mapping would be included in the service monitor YAML underspec.selector.matchLabels
. As a convenience, DataStax provided that example in the generated service_monitor.yaml.Tip: For example, to check the labels on your service if the namespace name isWith the service monitor's configuration file updated, apply the resource to the cluster.cass-operator
:kubectl get svc -n cass-operator --show-labels=true
Cassandra example:kubectl apply -f dashboards/k8s-build/generated/prometheus/service_monitor.yaml
DSE example:kubectl apply -n cass-operator -f k8s-build/generated/prometheus/service_monitor.yaml
- Configure and install the Prometheus deployment
Similar to the service and service monitor mapping above, the instance must be provided with a set of labels to match service monitors to Prometheus deployments. The following section in the Prometheus Custom Resource maps the deployment to all service monitors with the label
cassandra.datastax.com/cluster: cluster-name
serviceMonitorSelector: matchLabels: cassandra.datastax.com/cluster: cluster-name
After making any necessary adjustments for your environment, apply theinstance.yaml
file to the cluster.Cassandra example:kubectl apply -f dashboards/k8s-build/generated/prometheus/instance.yaml
DSE example:kubectl apply -n cass-operator -f k8s-build/generated/prometheus/instance.yaml
Grafana Operator setup
The Grafana Operator handles the orchestration, configuration, and deployment of Kubernetes resources required for a Grafana installation. Instead of configuring datasources, dashboards, and deployments via the graphical UI, everything is configured via YAML files in this repo.
The following steps illustrate how to install the Grafana operator, data source pointed at the previously deployed Prometheus, a collection of dashboards, and an instance of the Grafana application.
- Install the OperatorHub Lifecycle Manager (OLM)
The following command installs a number of custom resource definitions and an OLM operator that handles installing other operators:
curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/0.15.1/install.sh | bash -s 0.15.1
- Install the Grafana Operator
The following example pulls down and starts the Grafana Operator along with a configuration indicating it should watch the
default
namespace for resources it manages. If your installation uses a different namespace, change the values in operator.yaml and re-apply it to the cluster.Cassandra example:kubectl create -f dashboards/k8s-build/generated/grafana/operator.yaml
DSE example specifyingcass-operator
as the namespace:kubectl create -n cass-operator -f k8s-build/generated/grafana/operator.yaml
- Configure and install the GrafanaDataSource
Before installing the GrafanaDataSource, edit the datasource YAML to point at the instantiated Prometheus cluster. If you are not sure what value to use, check the output via
kubectl get svc
for the Prometheus service name.With the datasource configuration file updated, apply the resource to the cluster.
Cassandra example:kubectl apply -f dashboards/k8s-build/generated/grafana/datasource.yaml
DSE example:kubectl apply -n cass-operator -f k8s-build/generated/grafana/datasource.yaml
- Configure and install the GrafanaDashboard
Before installing the
GrafanaDashboard
, edit the YAML with appropriate labels. In this example, a label ofapp=grafana
is used. If needed, modify for your environment.With the configuration file updated, apply the resource to the cluster.
Cassandra example:kubectl apply -f dashboards/k8s-build/generated/grafana/
DSE example:kubectl apply -n cass-operator -f k8s-build/generated/grafana/
- Configure and install the Grafana deployment
The deployment must be informed of the labels to check for matching dashboards. The following section in the Grafana Custom Resource maps the deployment to all dashboards with the label
app=grafana
dashboardLabelSelector: - matchExpressions: - key: app operator: In values: - grafana
After adjustments apply the instance.yaml file to the cluster.Cassandra example:kubectl apply -f dashboards/k8s-build/generated/grafana/instance.yaml
DSE example:kubectl apply -n cass-operator -f k8s-build/generated/grafana/instance.yaml
- Check the Grafana instance Port forward to the grafana instance and check it out at http://127.0.0.1:3000/ (username: admin, password: secret)CAUTION: Never use documented credentials in production environments!Cassandra example:
kubectl port-forward svc/grafana-service 3000
DSE example:kubectl port-forward -n cass-operator svc/grafana-service 3000
What's next?
Learn about repair options for Apache Cassandra clusters in Kubernetes.