Scrape Astra Streaming metrics with Prometheus

Prometheus collects system metrics by scraping targets at intervals. These metrics are used to monitor deployments, generate alerts, and diagnose problems.

This doc will show you how to scrape an Astra Streaming tenant with Prometheus.

Prerequisites:

Astra Streaming tenant
Docker installed locally

Get configuration file from Astra Streaming

To start connecting your Astra Streaming tenant with Prometheus, select Prometheus in the Astra Streaming Connect tab.

A new configuration file will be generated in the Connect tab that looks like this:

global:
  scrape_interval: 60s
  evaluation_interval: 60s

scrape_configs:
  - job_name: "astra-pulsar-metrics"

    scheme: 'https'
    metrics_path: '/pulsarmetrics/tenant-1'
    authorization:
      credentials: '<pulsar-token>'

    static_configs:
      - targets: [https://prometheus-aws-useast2.dev.streaming.datastax.com/pulsarmetrics/tenant-1]

This example prometheus.yml will scrape https://prometheus-aws-useast2.dev.streaming.datastax.com/pulsarmetrics/tenant-1 every 60 seconds with the supplied Pulsar token.
The job_name is added as a label to any timeseries scraped with this configuration.

Copy and paste the configuration code, or download it as a .yml file (it will be called prometheus.yml).

Build Prometheus with custom yml file

Prometheus runs with a prometheus.yml file found either locally or in a Docker container. For this example, we’ll tell Docker to run Prometheus from our downloaded prometheus.yml file.

Pull the Prometheus Docker image with docker pull prom/prometheus.
Bind-mount your modified prometheus.yml file by running the Prometheus Docker container with a modified path in the -v argument.
```
docker run \
    -p 9090:9090 \
    -v /<your-path>/prometheus.yml:/etc/prometheus/prometheus.yml \
    prom/prometheus
```

If you receive output similar to below, you’re ready to scrape with Prometheus.

ts=2022-05-10T20:40:30.877Z caller=main.go:1199 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=2.75025ms db_storage=584ns remote_storage=708ns web_handler=167ns query_engine=416ns scrape=262.125µs scrape_sd=12.208µs notify=667ns notify_sd=792ns rules=1.042µs tracing=2.959µs
ts=2022-05-10T20:40:30.877Z caller=main.go:930 level=info msg="Server is ready to receive web requests."

If you encounter a mounts denied permissions error, ensure your local directory is shared with Docker in File Sharing → Resources.

Scrape with Prometheus

Open your Prometheus dashboard at localhost:9090. In the Status → Targets window, you should see the endpoint targeted in static_configs in an UP state.
Navigate to the Graph window. Enter pulsar_in_messages_total in the Expression field and select Execute. Prometheus will now display total incoming Pulsar messages to your Astra Streaming cluster.
Produce a few messages in your tenant with the Pulsar CLI or the Astra Streaming Websocket.
Your Prometheus graph displays the number of incoming Pulsar messages with each 60 second scrape.

You’re scraping your Astra Streaming tenant with Prometheus!

Content encoding

Astra Streaming supports content encoding with either gzip or deflate.
With the example from above still running, use a curl request to decompress your Prometheus scrape data.

Deflate
Gzip

curl -v -H 'Authorization: Bearer $BEARER_TOKEN' -H 'Accept-encoding: deflate' https://prometheus-aws-useast1.streaming.datastax.com/pulsarmetrics/tenant-1 --compressed

curl -v -H 'Authorization: Bearer $BEARER_TOKEN' -H 'Accept-encoding: gzip' https://prometheus-aws-useast1.streaming.datastax.com/pulsarmetrics/tenant-1 | zcat

Deflate or Gzip will extract your scraped metrics in a format like below:

# TYPE pulsar_topics_count untyped
pulsar_topics_count{app="pulsar",cluster="pulsar-aws-useast1",component="broker",controller_revision_hash="pulsar-aws-useast1-broker-7444bf6f64",instance="192.168.2.120:8080",job="broker",kubernetes_namespace="pulsar",kubernetes_pod_name="pulsar-aws-useast1-broker-1",namespace="mk-tenant/default",release="astraproduction-aws-useast1-pulsar",statefulset_kubernetes_io_pod_name="pulsar-aws-useast1-broker-1",prometheus="pulsar/astraproduction-aws-useast-prometheus",prometheus_replica="prometheus-astraproduction-aws-useast-prometheus-0"} 1 1654550685678

Metrics exposed by Astra Streaming

Here’s a list of Prometheus metrics exposed by Astra Streaming.

Cluster operational metrics are not exposed to individual cluster tenants. A tenant can only access its own metrics on the broker or function worker pods.

What’s next?

For more on monitoring Astra Streaming with Prometheus and Grafana, see Monitoring Streaming Tenants.
Follow our getting started guide to get started with Astra now.
For more on Prometheus, see the Prometheus docs.