Pulsar Heartbeat

Pulsar Heartbeat monitors the availability, tracks the performance, and reports failures of the Pulsar cluster. It produces synthetic workloads to measure end-to-end message pubsub latency.

Pulsar Heartbeat is a cloud native application that can be installed by Helm within a Pulsar Kubernetes cluster. It can also monitor multiple Pulsar clusters.

Pulsar Heartbeat is installed automatically for server/VM installations as described in Quick Start for Server/VM installs.

Pulsar Heartbeat supports the following features:

  • Monitor message pubsub and admin REST API endpoint

  • Measure end-to-end message latency from producing to consuming messages

  • Measure message latency over the websocket interface, and Pulsar function

  • Monitor instance availability of broker, proxy, bookkeeper, and zookeeper in a Pulsar Kubernetes cluster

  • Monitor individual Pulsar broker’s health

  • Incident alert integration with OpsGenie

  • Customer configurable alert thresholds and probe test intervals

  • Slack alerts

Configuration

Pulsar Heartbeat is a data driven tool that sources configuration from a yaml or json file. The configuration json file can be specified in the following order of precedence:

  • An environment variable PULSAR_OPS_MONITOR_CFG

  • A command line argument ./pulsar-heartbeat -config /path/to/runtime.yml

  • A default path to ../config/runtime.yml

You can download a template here.

Observability

Pulsar Heartbeat exposes Prometheus compliant metrics at the \metrics endpoint for scraping. The exported metrics are:

Name Type Description

pulsar_pubsub_latency_ms

gauge

end to end message pub and sub latency in milliseconds

pulsar_pubsub_latency_ms_hst

summary

end to end message latency histogram summary over 50%, 90%, and 99% samples

pulsar_websocket_latency_ms

gauge

end to end message pub and sub latency over websocket interface in milliseconds

pulsar_k8s_bookkeeper_offline_counter

gauge

bookkeeper offline instances in Kubernetes cluster

pulsar_k8s_broker_offline_counter

gauge

broker offline instances in the Kubernetes cluster

pulsar_k8s_proxy_offline_counter

gauge

proxy offline instances in the Kubernetes cluster

pulsar_k8s_bookkeeper_zookeeper_counter

gauge

zookeeper offline instances in the Kubernetes cluster

pulsar_monitor_counter

counter

pulsar monitor heartbeat counter

pulsar_tenant_size

gauge

the number of tenants that can be used as a health indicator of admin interface

In-cluster monitoring

Pulsar Heartbeat can be deployed within the same Pulsar Kubernetes cluster.

Kubernetes' pod and service, and individual broker monitoring are only supported within the same Kubernetes cluster deployment.

Docker

Pulsar Heartbeat’s official docker image can be pulled here

Docker compose

./config/runtime.yml or ./config/runtime.json must have a Pulsar jwt and must be configured properly.

$ docker-compose up

Docker example

The runtime.yml/yaml or runtime.json file must be mounted to /config/runtime.yml as the default configuration path.

Run docker container with Pulsar CA certificate if TLS is enabled and expose Prometheus metrics for collection.

$ docker run -d -it -v ./config/runtime.yml:/config/runtime.yml -v /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:/etc/ssl/certs/ca-bundle.crt -p 8080:8080 --name=pulsar-heartbeat datastax/pulsar-heartbeat:latest