• Glossary
  • Support
  • Downloads
  • DataStax Home
Get Live Help
Expand All
Collapse All

DataStax Streaming Home

Luna Streaming Documentation

    • Luna Streaming FAQs
    • Luna Streaming Installation on Azure Kubernetes Service
    • Luna Streaming Installation on Amazon Elastic Kubernetes Environment
    • Luna Streaming Installation on Google Kubernetes Environment
    • Install & Upgrade
      • Quick Start for Helm Chart installs
      • Quick Start for Bare Metal/VM installs
      • Supported Software
    • Components
      • Pulsar Admin Console
      • Install Pulsar Admin Console on Server/VM
      • Pulsar Heartbeat
      • Heartbeat on VM/Server
      • Pulsar Beam
      • Pulsar SQL
      • Starlight for Luna Streaming
    • Operations
      • Luna Streaming Authentication
      • Luna Streaming Functions
      • Luna Streaming IO connectors
      • Elasticsearch sink connector
      • Scaling your Luna Streaming cluster
      • Troubleshooting
  • Luna Streaming Documentation
  • Operations
  • Scaling your Luna Streaming cluster
2.10-3.x 2.10-2.x 2.10-1.x
Edit this Page

Scaling your Luna Streaming cluster

This page will show you how to scale Luna Streaming clusters up for more compute capacity, or down for less.

Installing Pulsar cluster

For our Pulsar cluster installation, use this Helm chart.

To start the cluster, use the values provided in this YAML file.

$ diff ~/dev-values.yaml ~/dev-values_large.yaml
40c40,47
<   replicaCount: 1
---
>   replicaCount: 4
>   volumes:
>     journal:
>       size: 2Gi
>     ledgers:
>       size: 10Gi
>     ranges:
>       size: 1Gi
52,54c59,61
<     defaultEnsembleSize: 1
<     defaultAckQuorum: 1
<     defaultWriteQuorum: 1
---
>     defaultEnsembleSize: 3
>     defaultAckQuorum: 2
>     defaultWriteQuorum: 3
  1. Create the cluster by installing Pulsar with dev-values_large.yaml:

    $ helm install pulsar -f ~/dev-values_large.yaml --wait datastax-pulsar/pulsar

Generating data

  1. Generate data in your namespaces:

    $ bin/pulsar-admin namespaces set-retention public/default -t -1 -s 100M
    $ bin/pulsar-client produce test1 -n 500 -m "sdasdsdsadsadsad"
    $ bin/pulsar-client produce test2 -n 500 -m "sdasdsdsadsadsad"
    $ bin/pulsar-client produce test3 -n 500 -m "sdasdsdsadsadsad"
  2. Ensure all bookies have some ledgers assigned to them:

    $ ./bin/bookkeeper shell listledgers -meta | grep pulsar-bookkeeper-3

Downscaling bookies

  1. To downscale a bookie, change the replicaCount value in the YAML file to a number less than the current replicaCount:

     bookkeeper:
        replicaCount: 3
  2. Upgrade the Helm chart to use the new value:

    $ helm upgrade pulsar -f ~/dev-values_large.yaml --wait datastax-pulsar/pulsar

    Now, bookkeeper-3 will fail to start with org.apache.bookkeeper.bookie.BookieException$InvalidCookieException.

  3. Destroy the bookie’s PVC:

$ kubectl delete pvc pulsar-bookkeeper-ledgers-pulsar-bookkeeper-3

Scaling up your cluster

To scale up your cluster, change the replicaCount value in the YAML file to a number greater than the current replicaCount:

 bookkeeper:
    replicaCount: 5
  1. Upgrade the Helm chart to use the new value in the Pulsar cluster:

    $ helm upgrade pulsar -f ~/dev-values_large.yaml --wait datastax-pulsar/pulsar

Recovering from a lost Persistent Volume Claim (PVC)

  1. If a PVC goes down, track under-replicated ledgers from any live bookie:

    $ ./bin/bookkeeper shell listunderreplicated
  2. Double-check the bookie id of the failing bookie:

    $ ./bin/bookkeeper shell listbookies -a

    This command will return Cannot resolve pulsar-bookkeeper-3.pulsar-bookkeeper.default.svc.cluster.local:3181, bookie is unknown.

  3. Decommission the bookie:

    $ ./bin/bookkeeper shell decommissionbookie -bookieid pulsar-bookkeeper-3.pulsar-bookkeeper.default.svc.cluster.local:3181

    The decommissioning will take some time because it triggers the audit and auto-recovery and blocks until auto-recovery completes.

    This command will delete the cookie of the decommissioned bookie: pulsar-bookkeeper-3.pulsar-bookkeeper.default.svc.cluster.local:3181.

  4. Restart the failing bookie with Ctrl-K in Kubernetes, or wait for Kubernetes to restart the bookie again.

    The bookie will restart and rejoin the cluster. As expected with deleted PVC, data is lost.

Elasticsearch sink connector Troubleshooting

General Inquiries: +1 (650) 389-6000 info@datastax.com

© DataStax | Privacy policy | Terms of use

DataStax, Titan, and TitanDB are registered trademarks of DataStax, Inc. and its subsidiaries in the United States and/or other countries.

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.

Kubernetes is the registered trademark of the Linux Foundation.

landing_page landingpage