Pulsar Heartbeat
Pulsar Heartbeat monitors the availability, tracks the performance, and reports failures of the Pulsar cluster. It produces synthetic workloads to measure end-to-end message pubsub latency.
Pulsar Heartbeat is a cloud native application that can be installed by Helm within a Pulsar Kubernetes cluster. It can also monitor multiple Pulsar clusters.
| Pulsar Heartbeat is installed automatically for server/VM installations as described in Quick Start for Bare Metal/VM installs. |
Pulsar Heartbeat supports the following features:
-
Monitor message pubsub and admin REST API endpoint
-
Measure end-to-end message latency from producing to consuming messages
-
Measure message latency over the websocket interface, and Pulsar function
-
Monitor instance availability of broker, proxy, bookkeeper, and zookeeper in a Pulsar Kubernetes cluster
-
Monitor individual Pulsar broker’s health
-
Incident alert integration with OpsGenie
-
Customer configurable alert thresholds and probe test intervals
-
Slack alerts
Configuration
Pulsar Heartbeat is a data driven tool that sources configuration from a YAML or JSON file. The configuration JSON file can be specified in the following order of precedence:
-
An environment variable
PULSAR_OPS_MONITOR_CFG -
A command line argument
./pulsar-heartbeat -config /path/to/runtime.yml -
A default path to
../config/runtime.yml
You can download a template here.
Observability
Pulsar Heartbeat exposes Prometheus compliant metrics at the \metrics endpoint for scraping. The exported metrics are:
| Name | Type | Description |
|---|---|---|
pulsar_pubsub_latency_ms |
gauge |
end to end message pub and sub latency in milliseconds |
pulsar_pubsub_latency_ms_hst |
summary |
end to end message latency histogram summary over 50%, 90%, and 99% samples |
pulsar_websocket_latency_ms |
gauge |
end to end message pub and sub latency over websocket interface in milliseconds |
pulsar_k8s_bookkeeper_offline_counter |
gauge |
bookkeeper offline instances in Kubernetes cluster |
pulsar_k8s_broker_offline_counter |
gauge |
broker offline instances in the Kubernetes cluster |
pulsar_k8s_proxy_offline_counter |
gauge |
proxy offline instances in the Kubernetes cluster |
pulsar_k8s_bookkeeper_zookeeper_counter |
gauge |
zookeeper offline instances in the Kubernetes cluster |
pulsar_monitor_counter |
counter |
pulsar monitor heartbeat counter |
pulsar_tenant_size |
gauge |
the number of tenants that can be used as a health indicator of admin interface |
In-cluster monitoring
Pulsar Heartbeat can be deployed within the same Pulsar Kubernetes cluster.
Kubernetes' pod and service, and individual broker monitoring are only supported within the same Kubernetes cluster deployment.
Docker
Pulsar Heartbeat’s official docker image can be pulled here.
Docker compose
docker-compose up./config/runtime.yml or ./config/runtime.json must have a Pulsar jwt and must be configured properly.
Docker example
docker run -d -it -v ./config/runtime.yml:/config/runtime.yml -v /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:/etc/ssl/certs/ca-bundle.crt -p 8080:8080 --name=pulsar-heartbeat datastax/pulsar-heartbeat:latestThe runtime.yml/yaml or runtime.json file must be mounted to /config/runtime.yml as the default configuration path.
Run docker container with Pulsar CA certificate if TLS is enabled and expose Prometheus metrics for collection.
