Monitoring the Starlight for Kafka extension
Starlight for Kafka exposes the following metrics in Prometheus format. You can monitor your clusters with these metrics.
The following types of metrics are available:
-
Counter: a cumulative metric that represents a single monotonically increasing counter. The value increases by default. You can reset the value to zero or restart your cluster.
-
Gauge: a metric that represents a single numerical value that can arbitrarily go up and down.
-
Histogram: a histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets.
-
Summary: similar to a histogram, a summary samples observations (usually things like request durations and response sizes). While it also provides a total count of observations and a sum of all observed values, it calculates configurable quantiles over a sliding time window.
Starlight for Kafka metrics
The Starlight for Kafka metrics are exposed under /metrics` at port 8000 along with Pulsar metrics. Use a different port by configuring the stats_server_port system property.
Request metrics
| Name | Type | Description |
|---|---|---|
kop_server_ALIVE_CHANNEL_COUNT |
Gauge |
The number of alive request channels. |
kop_server_ACTIVE_CHANNEL_COUNT |
Gauge |
The number of active request channels. |
Request metrics
| Name | Type | Description |
|---|---|---|
kop_server_REQUEST_QUEUE_SIZE |
Gauge |
The number of requests in S4K request processing queue of the total request channel. |
kop_server_REQUEST_QUEUED_LATENCY |
Summary |
The requests queued latency calculated in milliseconds. Available labels: request (ApiVersions, Metadata, Produce, FindCoordinator, ListOffsets, OffsetFetch, OffsetCommit, Fetch, JoinGroup, SyncGroup, Heartbeat, LeaveGroup, DescribeGroups, ListGroups, DeleteGroups, SaslHandshake, SaslAuthenticate, CreateTopics, InitProducerId, AddPartitionsToTxn, AddOffsetsToTxn, TxnOffsetCommit, EndTxn, WriteTxnMarkers, DescribeConfigs, DeleteTopics). |
kop_server_REQUEST_PARSE_LATENCY |
Summary |
The requests parse latency from |
kop_server_REQUEST_LATENCY |
Summary |
The requests processing total latency for all Kafka APIs. Available labels: request (ApiVersions, Metadata, Produce, FindCoordinator, ListOffsets, OffsetFetch, OffsetCommit, Fetch, JoinGroup, SyncGroup, Heartbeat, LeaveGroup, DescribeGroups, ListGroups, DeleteGroups, SaslHandshake, SaslAuthenticate, CreateTopics, InitProducerId, AddPartitionsToTxn, AddOffsetsToTxn, TxnOffsetCommit, EndTxn, WriteTxnMarkers, DescribeConfigs, DeleteTopics). |
Response metrics
| Name | Type | Description |
|---|---|---|
kop_server_RESPONSE_BLOCKED_TIMES |
Counter |
The response blocked times due to waiting for process completes. |
kop_server_RESPONSE_BLOCKED_LATENCY |
Summary |
The response blocked latency calculated in milliseconds. |
Producer metrics
| Name | Type | Description |
|---|---|---|
kop_server_PRODUCE_ENCODE |
Summary |
The memory record encode latency. |
kop_server_MESSAGE_PUBLISH |
Summary |
The message publish latency to Pulsar ManagedLedger. |
kop_server_MESSAGE_QUEUED_LATENCY |
Summary |
The message queued latency in S4K message publish queue. |
kop_server_BYTES_IN |
Counter |
The producer bytes in stats. Available labels: topic, partition.
|
kop_server_MESSAGE_IN |
Counter |
The producer message in stats. Available labels: topic, partition.
|
kop_server_BATCH_COUNT_PER_MEMORYRECORDS |
Gauge |
The number of batches in each memory records. |
kop_server_PRODUCE_MESSAGE_CONVERSIONS |
Counter |
The producer message conversions in stats. Available labels: topic, partition.
|
Consumer metrics
| Name | Type | Description |
|---|---|---|
kop_server_PREPARE_METADATA |
Summary |
The prepare metadata latency in milliseconds before starting fetch from Pulsar ManagedLedger. |
kop_server_TOTAL_MESSAGE_READ |
Summary |
The total message read latency in milliseconds in this fetch request. |
kop_server_MESSAGE_READ |
Summary |
The message read latency in milliseconds for one cursor read entry request. |
kop_server_FETCH_DECODE |
Summary |
The message decode latency in milliseconds. |
kop_server_BYTES_OUT |
Counter |
The consumer bytes out stats. Available labels: topic, partition, group.
|
kop_server_MESSAGE_OUT |
Counter |
The consumer message out stats. Available labels: topic, partition, group.
|
kop_server_ENTRIES_OUT |
Counter |
The consumer entries out stats. Available labels: topic, partition, group.
|
kop_server_CONSUME_MESSAGE_CONVERSIONS |
Counter |
The consumer message conversions in stats. Available labels: topic, partition.
|
S4K event metrics
| Name | Type | Description |
|---|---|---|
kop_server_KOP_EVENT_QUEUE_SIZE |
Gauge |
The total number of events in S4K event processing queue. |
kop_server_KOP_EVENT_QUEUED_LATENCY |
Summary |
The events queued latency calculated in milliseconds. Available labels: event (DeleteTopicsEvent, BrokersChangeEvent, ShutdownEventThread). |
kop_server_KOP_EVENT_LATENCY |
Summary |
The events processing total latency for all S4K event types. Available labels: event (DeleteTopicsEvent, BrokersChangeEvent, ShutdownEventThread). |
What’s next?
For more on Starlight for Kafka, see:
