Scaling your Luna Streaming cluster
You can scale Luna Streaming clusters up for more compute capacity and down for less capacity.
The Kubernetes Autoscaling for Apache Pulsar (KAAP) operator takes care of scaling Pulsar cluster components, deploying new clusters, and even migrating your existing cluster to an operator-managed deployment. |
Install Pulsar cluster
For our Pulsar cluster installation, use this Helm chart.
To start the cluster, use the values provided in this YAML file:
diff ~/dev-values.yaml ~/dev-values_large.yaml
40c40,47
< replicaCount: 1
---
> replicaCount: 4
> volumes:
> journal:
> size: 2Gi
> ledgers:
> size: 10Gi
> ranges:
> size: 1Gi
52,54c59,61
< defaultEnsembleSize: 1
< defaultAckQuorum: 1
< defaultWriteQuorum: 1
---
> defaultEnsembleSize: 3
> defaultAckQuorum: 2
> defaultWriteQuorum: 3
-
Create the cluster by installing Pulsar with
dev-values_large.yaml
:helm install pulsar -f ~/dev-values_large.yaml --wait datastax-pulsar/pulsar
Generate data
-
Generate data in your namespaces:
bin/pulsar-admin namespaces set-retention public/default -t -1 -s 100M bin/pulsar-client produce test1 -n 500 -m "sdasdsdsadsadsad" bin/pulsar-client produce test2 -n 500 -m "sdasdsdsadsadsad" bin/pulsar-client produce test3 -n 500 -m "sdasdsdsadsadsad"
-
Ensure all bookies have some ledgers assigned to them:
./bin/bookkeeper shell listledgers -meta | grep pulsar-bookkeeper-3
Scale down bookies
-
To scale down a bookie, change the
replicaCount
value in the YAML file to a number less than the currentreplicaCount
:bookkeeper: replicaCount: 3
-
Upgrade the Helm chart to use the new value:
helm upgrade pulsar -f ~/dev-values_large.yaml --wait datastax-pulsar/pulsar
Now,
bookkeeper-3
will fail to start withorg.apache.bookkeeper.bookie.BookieException$InvalidCookieException
. -
Destroy the bookie’s PVC:
kubectl delete pvc pulsar-bookkeeper-ledgers-pulsar-bookkeeper-3
Scale up your cluster
-
To scale up your cluster, change the
replicaCount
value in the YAML file to a number greater than the currentreplicaCount
:bookkeeper: replicaCount: 5
-
Upgrade the Helm chart to use the new value in the Pulsar cluster:
helm upgrade pulsar -f ~/dev-values_large.yaml --wait datastax-pulsar/pulsar
Recover from a lost Persistent Volume Claim (PVC)
-
If a PVC goes down, track under-replicated ledgers from any live bookie:
./bin/bookkeeper shell listunderreplicated
-
Double-check the bookie id of the failing bookie:
./bin/bookkeeper shell listbookies -a
This command returns
Cannot resolve pulsar-bookkeeper-3.pulsar-bookkeeper.default.svc.cluster.local:3181, bookie is unknown
. -
Decommission the bookie:
./bin/bookkeeper shell decommissionbookie -bookieid pulsar-bookkeeper-3.pulsar-bookkeeper.default.svc.cluster.local:3181
This command deletes the cookie of the decommissioned bookie specified by
-bookieid
.The decommissioning takes some time because it triggers the audit and auto-recovery, which blocks until auto-recovery completes.
-
Restart the failing bookie with Ctrl-K in Kubernetes, or wait for Kubernetes to restart the bookie again.
The bookie will restart and rejoin the cluster. However, as expected with deleted PVC, data is lost.