Luna Streaming Installation on Amazon Elastic Kubernetes Environment

This document covers installation of Luna Streaming on the Amazon Elastic Kubernetes Environment (EKS).

For an example set of production cluster values, see the DataStax production-ready Helm chart.

DataStax recommends these hardware resources for running Luna Streaming in a Kubernetes environment:

Helm version 3
A Kubernetes cluster
Two node pools
- One function-worker node pool for deploying sink and source connectors, and the other node pool for everything else
Must use SSD disks
Depending on the cloud provider, the latest 'Storage Driver' should be used, along with the fastest disk type (for example, GP3 in AWS)
5 Zookeeper replicas
3 Bookies
3 Brokers
3 Proxies

For the local machine running the Helm chart, you will need:

Helm version 3
An AWS account with privileges to install and configure Kubernetes resources.

Helm chart `values.yaml`

The values.yaml will deploy Pulsar with:

1 ZooKeeper cluster with replicaCount:3
1 Bookkeeper cluster with replicaCount:3
1 Broker cluster with replicaCount: 3
- Ledgers: defaultEnsembleSize: 2, defaultAckQuorum: 2, defaultWriteQuorum: 2
1 Function cluster with a replicaCount: 2

`default_storage`

The default_storage parameter in values.yaml controls the default storage class for all persistent volumes created by the Helm chart.

default_storage:
  existingStorageClassName: default

For a component like BookKeeper, which requires stateful storage, we need to override the default_storage class when the BookKeeper Persistent Volume Claims (PVCs) are created.

There are two ways to override default_storage:

Leave existingStorageClassName blank and specify the storage class parameters below.

  volumes:
    journal:
      name: journal
      size: 20Gi
      existingStorageClassName:
        storageClass:
         provisioner: kubernetes.io/aws-ebs
         type: gp2
         fsType: ext4
         extraParams:
            iopsPerGB: "10"

Create a custom storage configuration as a yaml file (like the DataStax example) and tell the Helm chart to use that storage configuration when it creates the BookKeeper PVCs.
```
  volumes:
    journal:
      name: journal
      size: 20Gi
      existingStorageClassName: bookkeeper-storageclass.yaml
```

EKS-specific storage class

Here is an example storage-values.yaml as used above for the BookKeeper node. Paste it into a yaml file to provision a persistent general purpose (gp2) SSD with the ext4 file system.

storage_class:
    provisioner: kubernetes.io/aws-ebs
    type: gp2
    fsType: ext4
    extraParams:
      iopsPerGB: "10"

Install Luna Streaming to EKS

Once you have added your storage settings to the storage-values.yaml file, install the Helm chart using the storage-values.yaml provisioning.

Create the namespace. In this example, we use pulsar:
```
kubectl create namespace pulsar
```

Run this Helm command using the storage-values.yaml provisioning:

helm install pulsar datastax-pulsar/pulsar --namespace pulsar --values storage_values.yaml --create-namespace

To avoid having to specify the pulsar namespace on each subsequent command, set the namespace context. Example:
```
`kubectl config set-context $(kubectl config current-context) --namespace=pulsar`
```
Execute kubectl get pods -A to view your running pods.

What’s next?

Once your pods are up and running, continue to Accessing Pulsar Cluster in Cloud to access and manage your cluster.

To learn more about using the Pulsar Admin Console, see Admin Console Tutorial.
To learn more about installing Pulsar on a server or virtual machine, see Quickstart for Server/VM Install.

Was this helpful?

Edit this page