Luna Streaming Installation on Amazon Elastic Kubernetes Environment
This document covers installation of Luna Streaming on the Amazon Elastic Kubernetes Environment (EKS).
For an example set of production cluster values, see the DataStax production-ready Helm chart.
DataStax recommends these hardware resources for running Luna Streaming in a Kubernetes environment:
-
Helm version 3
-
A Kubernetes cluster
-
Two node pools
-
One
function-workernode pool for deploying sink and source connectors, and the other node pool for everything else
-
-
Must use SSD disks
-
Depending on the cloud provider, the latest 'Storage Driver' should be used, along with the fastest disk type (for example, GP3 in AWS)
-
5 Zookeeper replicas
-
3 Bookies
-
3 Brokers
-
3 Proxies
For the local machine running the Helm chart, you will need:
-
Helm version 3
-
An AWS account with privileges to install and configure Kubernetes resources.
Helm chart values.yaml
The values.yaml will deploy Pulsar with:
-
1 ZooKeeper cluster with
replicaCount:3 -
1 Bookkeeper cluster with
replicaCount:3 -
1 Broker cluster with
replicaCount: 3-
Ledgers:
defaultEnsembleSize: 2,defaultAckQuorum: 2,defaultWriteQuorum: 2
-
-
1 Function cluster with a
replicaCount: 2
default_storage
The default_storage parameter in values.yaml controls the default storage class for all persistent volumes created by the Helm chart.
default_storage:
existingStorageClassName: defaultFor a component like BookKeeper, which requires stateful storage, we need to override the default_storage class when the BookKeeper Persistent Volume Claims (PVCs) are created.
There are two ways to override default_storage:
-
Leave
existingStorageClassNameblank and specify the storage class parameters below.volumes: journal: name: journal size: 20Gi existingStorageClassName: storageClass: provisioner: kubernetes.io/aws-ebs type: gp2 fsType: ext4 extraParams: iopsPerGB: "10" -
Create a custom storage configuration as a
yamlfile (like the DataStax example) and tell the Helm chart to use that storage configuration when it creates the BookKeeper PVCs.volumes: journal: name: journal size: 20Gi existingStorageClassName: bookkeeper-storageclass.yaml
EKS-specific storage class
Here is an example storage-values.yaml as used above for the BookKeeper node. Paste it into a yaml file to provision a persistent general purpose (gp2) SSD with the ext4 file system.
storage_class:
provisioner: kubernetes.io/aws-ebs
type: gp2
fsType: ext4
extraParams:
iopsPerGB: "10"Install Luna Streaming to EKS
Once you have added your storage settings to the storage-values.yaml file, install the Helm chart using the storage-values.yaml provisioning.
-
Create the namespace. In this example, we use
pulsar:kubectl create namespace pulsar
-
Run this Helm command using the
storage-values.yamlprovisioning:helm install pulsar datastax-pulsar/pulsar --namespace pulsar --values storage_values.yaml --create-namespace
-
To avoid having to specify the
pulsarnamespace on each subsequent command, set the namespace context. Example:`kubectl config set-context $(kubectl config current-context) --namespace=pulsar`
-
Execute
kubectl get pods -Ato view your running pods.
What’s next?
Once your pods are up and running, continue to Accessing Pulsar Cluster in Cloud to access and manage your cluster.
-
To learn more about using the Pulsar Admin Console, see Admin Console Tutorial.
-
To learn more about installing Pulsar on a server or virtual machine, see Quickstart for Server/VM Install.
