Have an Apache Kafka® workload you want to control with KAAP Operator? Thanks to Starlight for Kafka, you can run your Kafka workload on a Pulsar cluster, and with KAAP Operator, the scaling of the Kafka pods is handled for you.

Scaling the Pulsar Broker with a Kafka Client Workload

This folder contains a sample configuration and demo about how to run a workload on an Apache Pulsar® cluster with the Broker Auto Scaling feature.

Support for the Kafka wire protocol is provided by Starlight for Kafka.

The client work load is generated using the basic Kafka Performance tools.


  1. Install the operator and a Pulsar cluster. In this case, we’re installing Pulsar Stack with the Kafka protocol enabled.

    helm install pos helm/pulsar-stack --values helm/examples/kafka/values.yaml

    The Kafka protocol is controlled in the deployment’s values.yaml file.

            enabled: true
            enabled: true
            periodMs: 20000
            min: 2
            max: 10
            lowerCpuThreshold: 0.2
            higherCpuThreshold: 0.4
            scaleUpBy: 1
            scaleDownBy: 1
            stabilizationWindowMs: 120000
        config: {}

    Additionally, you can proxy the Kafka connection in the Pulsar Proxy with kafka:enabled:true.

            replicas: 1
                cpu: "0.2"
                memory: "128Mi"
              enabled: true
            config: {}
  2. Deploy the Kafka producers.

    kubectl apply -f helm/examples/kafka/kafka-producer-perf.yaml
  3. See the logs of the Kafka producers.

    kubectl logs -f deploy/kafka-client-producer
  4. Deploy the Kafka consumers.

    kubectl apply -f helm/examples/kafka/kafka-consumer-perf.yaml
  5. See the logs of the Kafka consumers.

    kubectl logs -f deploy/kafka-client-consumer

Scaling the client workloads

  1. Open a terminal and view the logs of the Operator.

    kubectl logs -f deploy/pulsar-operator
  2. You can use Kubectl to scale the client applications up and down.

    kubectl scale deploy/kafka-client-consumer  --replicas 10

As the load increases you will see the Operator scales out the Broker STS.

Was this helpful?

Give Feedback

How can we improve the documentation?

© 2024 DataStax | Privacy policy | Terms of use

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: +1 (650) 389-6000, info@datastax.com