Scale the Pulsar Broker with a Kafka client workload

This guide demonstrates how to configure and run an Apache Kafka® workload on an Apache Pulsar™ cluster with the Broker Auto Scaling feature.

Support for the Kafka wire protocol is provided by Starlight for Kafka.

The client work load is generated using the basic Kafka performance tools.

Install the KAAP Operator and a Pulsar cluster.

The following example installs KAAP stack with the Kafka protocol enabled:

helm install pos helm/pulsar-stack --values helm/examples/kafka/values.yaml

The Kafka protocol is controlled in the deployment’s values.yaml file:

kafka:
        enabled: true
    autoscaler:
        enabled: true
        periodMs: 20000
        min: 2
        max: 10
        lowerCpuThreshold: 0.2
        higherCpuThreshold: 0.4
        scaleUpBy: 1
        scaleDownBy: 1
        stabilizationWindowMs: 120000
    config: {}

Additionally, you can proxy the Kafka connection in the Pulsar Proxy with kafka:enabled:true:

proxy:
        replicas: 1
        resources:
          requests:
            cpu: "0.2"
            memory: "128Mi"
        kafka:
          enabled: true
        config: {}

Deploy the Kafka producers:

kubectl apply -f helm/examples/kafka/kafka-producer-perf.yaml

Get the logs of the Kafka producers:

kubectl logs -f deploy/kafka-client-producer

Deploy the Kafka consumers:

kubectl apply -f helm/examples/kafka/kafka-consumer-perf.yaml

Get the logs of the Kafka consumers:

kubectl logs -f deploy/kafka-client-consumer

Scale the client workloads:
1. Open a terminal, and then get the logs of the KAAP Operator:
  kubectl logs -f deploy/pulsar-operator
2. Use kubectl to scale the client applications up and down:
  kubectl scale deploy/kafka-client-consumer --replicas 10
  As the load increases, the KAAP Operator scales out the Broker STS.

Scale the Pulsar Broker with a Kafka client workload

Was this helpful?

Give Feedback