Pulsar Heartbeat
Pulsar Heartbeat monitors the availability, tracks the performance, and reports failures of the Pulsar cluster. It produces synthetic workloads to measure end-to-end message pubsub latency.
Pulsar Heartbeat is a cloud native application that can be installed by Helm within a Pulsar Kubernetes cluster. It can also monitor multiple Pulsar clusters.
| Pulsar Heartbeat is installed automatically for server/VM installations as described in Quick Start for Bare Metal/VM installs. |
Pulsar Heartbeat supports the following features:
-
Monitor message pubsub and admin REST API endpoint
-
Measure end-to-end message latency from producing to consuming messages
-
Measure message latency over the websocket interface, and Pulsar function
-
Monitor instance availability of broker, proxy, bookkeeper, and zookeeper in a Pulsar Kubernetes cluster
-
Monitor individual Pulsar broker’s health
-
Incident alert integration with OpsGenie
-
Customer configurable alert thresholds and probe test intervals
-
Slack alerts
Configuration
Pulsar Heartbeat is a data driven tool that sources configuration from a yaml or json file. The configuration json file can be specified in the following order of precedence:
-
An environment variable
PULSAR_OPS_MONITOR_CFG -
A command line argument
./pulsar-heartbeat -config /path/to/runtime.yml -
A default path to
../config/runtime.yml
You can download a template here.
Observability
Pulsar Heartbeat exposes Prometheus compliant metrics at the \metrics endpoint for scraping. The exported metrics are:
| Name | Type | Description |
|---|---|---|
pulsar_pubsub_latency_ms |
gauge |
end to end message pub and sub latency in milliseconds |
pulsar_pubsub_latency_ms_hst |
summary |
end to end message latency histogram summary over 50%, 90%, and 99% samples |
pulsar_websocket_latency_ms |
gauge |
end to end message pub and sub latency over websocket interface in milliseconds |
pulsar_k8s_bookkeeper_offline_counter |
gauge |
bookkeeper offline instances in Kubernetes cluster |
pulsar_k8s_broker_offline_counter |
gauge |
broker offline instances in the Kubernetes cluster |
pulsar_k8s_proxy_offline_counter |
gauge |
proxy offline instances in the Kubernetes cluster |
pulsar_k8s_bookkeeper_zookeeper_counter |
gauge |
zookeeper offline instances in the Kubernetes cluster |
pulsar_monitor_counter |
counter |
pulsar monitor heartbeat counter |
pulsar_tenant_size |
gauge |
the number of tenants that can be used as a health indicator of admin interface |
In-cluster monitoring
Pulsar Heartbeat can be deployed within the same Pulsar Kubernetes cluster.
| Kubernetes' pod and service, and individual broker monitoring are only supported within the same Kubernetes cluster deployment. |
Docker
Pulsar Heartbeat’s official docker image can be pulled here
Docker compose
./config/runtime.yml or ./config/runtime.json must have a Pulsar jwt and must be configured properly.
$ docker-compose upDocker example
The runtime.yml/yaml or runtime.json file must be mounted to /config/runtime.yml as the default configuration path.
Run docker container with Pulsar CA certificate if TLS is enabled and expose Prometheus metrics for collection.
$ docker run -d -it -v ./config/runtime.yml:/config/runtime.yml -v /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:/etc/ssl/certs/ca-bundle.crt -p 8080:8080 --name=pulsar-heartbeat datastax/pulsar-heartbeat:latest