Luna Streaming Installation on Amazon Elastic Kubernetes Environment

This document covers installation of Luna Streaming on the Amazon Elastic Kubernetes Environment (EKS).

For an example set of production cluster values, see the DataStax production-ready Helm chart.

DataStax recommends these hardware resources for running Luna Streaming in a Kubernetes environment:

  • Helm version 3

  • A Kubernetes cluster

  • Two node pools

    • One function-worker node pool for deploying sink and source connectors, and the other node pool for everything else

  • Must use SSD disks

  • Depending on the cloud provider, the latest 'Storage Driver' should be used, along with the fastest disk type (for example, GP3 in AWS)

  • 5 Zookeeper replicas

  • 3 Bookies

  • 3 Brokers

  • 3 Proxies

For the local machine running the Helm chart, you will need:

  • Helm version 3

  • An AWS account with privileges to install and configure Kubernetes resources.

Helm chart values.yaml

The values.yaml will deploy Pulsar with:

  • 1 ZooKeeper cluster with replicaCount:3

  • 1 Bookkeeper cluster with replicaCount:3

  • 1 Broker cluster with replicaCount: 3

    • Ledgers: defaultEnsembleSize: 2, defaultAckQuorum: 2, defaultWriteQuorum: 2

  • 1 Function cluster with a replicaCount: 2


The default_storage parameter in values.yaml controls the default storage class for all persistent volumes created by the Helm chart.

  existingStorageClassName: default

For a component like BookKeeper, which requires stateful storage, we need to override the default_storage class when the BookKeeper Persistent Volume Claims (PVCs) are created.

There are two ways to override default_storage:

  • Leave existingStorageClassName blank and specify the storage class parameters below.

          name: journal
          size: 20Gi
             type: gp2
             fsType: ext4
                iopsPerGB: "10"
  • Create a custom storage configuration as a yaml file (like the DataStax example) and tell the Helm chart to use that storage configuration when it creates the BookKeeper PVCs.

          name: journal
          size: 20Gi
          existingStorageClassName: bookkeeper-storageclass.yaml

EKS-specific storage class

Here is an example storage-values.yaml as used above for the BookKeeper node. Paste it into a yaml file to provision a persistent general purpose (gp2) SSD with the ext4 file system.

    type: gp2
    fsType: ext4
      iopsPerGB: "10"

Install Luna Streaming to EKS

Once you have added your storage settings to the storage-values.yaml file, install the Helm chart using the storage-values.yaml provisioning.

  1. Create the namespace. In this example, we use pulsar:

    kubectl create namespace pulsar
  2. Run this Helm command using the storage-values.yaml provisioning:

    helm install pulsar datastax-pulsar/pulsar --namespace pulsar --values storage_values.yaml --create-namespace
  3. To avoid having to specify the pulsar namespace on each subsequent command, set the namespace context. Example:

    `kubectl config set-context $(kubectl config current-context) --namespace=pulsar`
  4. Execute kubectl get pods -A to view your running pods.

What’s next?

Once your pods are up and running, continue to Accessing Pulsar Cluster in Cloud to access and manage your cluster.

Was this helpful?

Give Feedback

How can we improve the documentation?

© 2024 DataStax | Privacy policy | Terms of use

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: +1 (650) 389-6000,