Luna Streaming Installation on Google Kubernetes Environment

This document will cover installation of Luna Streaming on the Google Kubernetes Environment (GKE).

For an example set of production cluster values, see the DataStax production-ready Helm chart.

DataStax recommends these hardware resources for running Luna Streaming in a Kubernetes environment:

  • Helm version 3

  • A Kubernetes cluster

  • Two node pools

    • One function-worker node pool for deploying sink and source connectors, and the other node pool for everything else

  • Must use SSD disks

  • Depending on the cloud provider, the latest 'Storage Driver' should be used, along with the fastest disk type (for example, GP3 in AWS)

  • 5 Zookeeper replicas

  • 3 Bookies

  • 3 Brokers

  • 3 Proxies

For the local machine running the Helm chart, you will need:

  • Helm version 3

  • A Google account with privileges to install and configure Kubernetes resources.

Helm chart values.yaml

The values.yaml will deploy Pulsar with:

  • 1 ZooKeeper cluster with replicaCount:3

  • 1 Bookkeeper cluster with replicaCount:3

  • 1 Broker cluster with replicaCount: 3

    • Ledgers: defaultEnsembleSize: 2, defaultAckQuorum: 2, defaultWriteQuorum: 2

  • 1 Function cluster with a replicaCount: 2


The default_storage parameter in values.yaml controls the default storage class for all persistent volumes created by the Helm chart.

  existingStorageClassName: default

For a component like BookKeeper, which requires stateful storage, we need to override the default_storage class when the BookKeeper Persistent Volume Claims (PVCs) are created.

There are two ways to override default_storage:

  • Leave existingStorageClassName blank and specify the storage class parameters below.

          name: journal
          size: 20Gi
             type: pd-ssd
             fsType: ext4
                replication-type: none
  • Create a custom storage configuration as a yaml file (like the DataStax example) and tell the Helm chart to use that storage configuration when it creates the BookKeeper PVCs.

          name: journal
          size: 20Gi
            existingStorageClassName: bookkeeper-storageclass.yaml

GKE-specific storage class

Here is an example storage-values.yaml as used above for the BookKeeper node. Paste it into a yaml file to provision a persistent SSD with the ext4 file system.

    type: pd-ssd
    fsType: ext4
        replication-type: none

Install Luna Streaming to GKE

Once you have added your storage settings to the storage-values.yaml file, install the Helm chart.

  1. Create the namespace; in this example, we use pulsar.

    kubectl create namespace pulsar
  2. Run this Helm command using the storage-values.yaml provisioning:

    helm install pulsar datastax-pulsar/pulsar --namespace pulsar --values storage_values.yaml --create-namespace
  3. To avoid having to specify the pulsar namespace on each subsequent command, set the namespace context. Example:

    kubectl config set-context $(kubectl config current-context) --namespace=pulsar
  4. Execute kubectl get pods -A to view your running pods.

What’s next?

Once your pods are up and running, continue to Accessing Pulsar Cluster in Cloud to access and manage your cluster.

Was this helpful?

Give Feedback

How can we improve the documentation?

© 2024 DataStax | Privacy policy | Terms of use

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: +1 (650) 389-6000,