Integrate with an external Prometheus and Grafana instance

Astra Streaming exposes some of the Pulsar metrics through Prometheus endpoints. The Prometheus configuration information to scrape Astra Streaming metrics into an external Prometheus server is found in the Astra Portal.

This page explains how to set up a Prometheus server in a Kubernetes cluster and configure it to scrape Astra Streaming metrics.

Prerequisites

An Astra DB account
A Kubernetes cluster

Configure and deploy Prometheus

This example uses the Prometheus Community Kubernetes Helm Charts to set up a Prometheus server and a Grafana server in a Kubernetes cluster.

Download the Astra Streaming Prometheus configuration from the Astra Portal. For more information, see Scrape metrics from Astra Streaming.
Use the downloaded file to create a config.yml.

This example uses a job named astra-pulsar-metrics-demo. The values in your config.yml depend on your tenant configuration.
config.yml
```
  - job_name: "astra-pulsar-metrics-demo"

    scheme: 'https'
    metrics_path: '/pulsarmetrics/TENANT_NAME'
    authorization:
      credentials: 'PULSAR_JWT_TOKEN'

    static_configs:
      - targets: [https://prometheus-PROVIDER-REGION.streaming.datastax.com/pulsarmetrics/TENANT_NAME]
```
Do not add other Prometheus configurations, like scrape_interval or evaluation_interval yet. These are added later when configuring the Helm chart.

Create a Kubernetes secret with the above configuration file, and then apply it:

kubectl create secret generic SECRET_NAME \
  --from-file=config.yml \
  --dry-run=client -oyaml > k8s-additional-scrape-config-demo.yaml
kubectl apply -f k8s-additional-scrape-config-demo.yaml

You can change the name of the YAML output file, if desired.

Create a customized values file (custom-values.yml) for the Prometheus Community Kubernetes Helm chart. In the 'additionalScrapeConfigsSecret' section, make sure the name and key match the name if your Kubernetes secret and config file name, such as astra-secret and config.yml.
custom-values.yml
```
prometheus:
   prometheusSpec:
      scrapeInterval: 60s
      evaluationInterval: 15s
      additionalScrapeConfigsSecret:
         enabled: true
         name: SECRET_NAME
         key: CONFIG_YAML
```

Deploy the Helm chart:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm upgrade --install kubeprom -f custom-values.yaml prometheus-community/kube-prometheus-stack

For macOS users who run Docker Desktop as the underlying Kubernetes cluster, run the following commands to ensure the deployment is successful. If you don’t run this command, you might encounter a problem when initializing the Prometheus node exporter daemon set.

# Patch Prometheus node exporter daemon set
kubectl patch ds kubeprom-prometheus-node-exporter --type "json" -p '[{"op": "remove", "path" : "/spec/template/spec/containers/0/volumeMounts/2/mountPropagation"}]'

# Confirm Prometheus node exporter daemonset is running properly
kubectl rollout status daemonset \
  kubeprom-prometheus-node-exporter \
  --timeout 60s

To confirm that Astra Streaming metrics are integrated with your external Prometheus server, go to your external Prometheus server UI. Make sure the additional scrape job is in UP status. If not, review the configuration instructions and YAML examples to ensure your configuration is correct.

If the additional scrape job returns a 401 Unauthorized error, make sure your Pulsar JWT isn’t expired. For more information, see Manage tokens.

Integrate with an external Prometheus and Grafana instance

Prerequisites

Configure and deploy Prometheus

See also

Was this helpful?

Give Feedback