Scaling your IBM Elite Support for Apache Pulsar cluster

You can scale IBM Elite Support for Apache Pulsar clusters up for more compute capacity and down for less capacity.

IBM Elite Support for Apache Pulsar is the new name for Luna Streaming after IBM acquired DataStax. In this documentation, IBM Elite Support for Apache Pulsar and Luna Streaming refer to the same product.

For technical specifications, inclusions, and exclusions of the IBM Elite Support for Apache Pulsar offering, see the IBM Elite Support for Apache Pulsar page in the IBM documentation.

Kubernetes Autoscaling for Apache Pulsar (KAAP) takes care of scaling Pulsar cluster components, deploying new clusters, and even migrating your existing cluster to an operator-managed deployment.

Install Pulsar cluster

The DataStax Pulsar Helm chart is deprecated. For Kubernetes deployments, use Kubernetes Autoscaling for Apache Pulsar (KAAP) instead.

To start the cluster, use the values provided in the DataStax Pulsar values YAML file:

diff ~/dev-values.yaml ~/dev-values_large.yaml
40c40,47
<   replicaCount: 1
---
>   replicaCount: 4
>   volumes:
>     journal:
>       size: 2Gi
>     ledgers:
>       size: 10Gi
>     ranges:
>       size: 1Gi
52,54c59,61
<     defaultEnsembleSize: 1
<     defaultAckQuorum: 1
<     defaultWriteQuorum: 1
---
>     defaultEnsembleSize: 3
>     defaultAckQuorum: 2
>     defaultWriteQuorum: 3

Create the cluster by installing Pulsar with dev-values_large.yaml:

helm install pulsar -f ~/dev-values_large.yaml --wait datastax-pulsar/pulsar

Generate data

Generate data in your namespaces:

bin/pulsar-admin namespaces set-retention public/default -t -1 -s 100M
bin/pulsar-client produce test1 -n 500 -m "sdasdsdsadsadsad"
bin/pulsar-client produce test2 -n 500 -m "sdasdsdsadsadsad"
bin/pulsar-client produce test3 -n 500 -m "sdasdsdsadsadsad"

Ensure all bookies have some ledgers assigned to them:

./bin/bookkeeper shell listledgers -meta | grep pulsar-bookkeeper-3

Scale down bookies

To scale down a bookie, change the replicaCount value in the YAML file to a number less than the current replicaCount:
```
 bookkeeper:
    replicaCount: 3
```
Upgrade the Helm chart to use the new value:
```
helm upgrade pulsar -f ~/dev-values_large.yaml --wait datastax-pulsar/pulsar
```
Now, bookkeeper-3 will fail to start with org.apache.bookkeeper.bookie.BookieException$InvalidCookieException.

Destroy the bookie’s PVC:

kubectl delete pvc pulsar-bookkeeper-ledgers-pulsar-bookkeeper-3

Scale up your cluster

To scale up your cluster, change the replicaCount value in the YAML file to a number greater than the current replicaCount:
```
 bookkeeper:
    replicaCount: 5
```

Upgrade the Helm chart to use the new value in the Pulsar cluster:

helm upgrade pulsar -f ~/dev-values_large.yaml --wait datastax-pulsar/pulsar

Recover from a lost Persistent Volume Claim (PVC)

If a PVC goes down, track under-replicated ledgers from any live bookie:
```
./bin/bookkeeper shell listunderreplicated
```
Double-check the bookieid of the failing bookie:
```
./bin/bookkeeper shell listbookies -a
```
Decommission the bookie:
```
./bin/bookkeeper shell decommissionbookie -bookieid pulsar-bookkeeper-3.pulsar-bookkeeper.default.svc.cluster.local:3181
```
This command deletes the cookie of the decommissioned bookie specified by -bookieid.

The decommissioning takes some time because it triggers the audit and auto-recovery, which blocks until auto-recovery completes.
Restart the failing bookie with Ctrl-K in Kubernetes, or wait for Kubernetes to restart the bookie again.

The bookie will restart and rejoin the cluster. However, as expected with deleted PVC, data is lost.

Scaling your IBM Elite Support for Apache Pulsar cluster

Install Pulsar cluster

Generate data

Scale down bookies

Scale up your cluster

Recover from a lost Persistent Volume Claim (PVC)

Was this helpful?

Give Feedback