Upgrade Luna Streaming from 2.10 to 3.1

This guide provides instructions and recommendations for upgrading Luna Streaming from version 2.10 to 3.1.

Upgrades to Luna Streaming 3.1 should only be performed from Luna Streaming 2.10.

All Luna Streaming 2.10 components support the upgrade to 3.1.

Functional impacts

This section describes changes in Luna Streaming 3.1 that may impact how your deployment functions.

Default system topics

In Pulsar 3.1, system topics are now enabled by default.

Prometheus metrics changes

Prometheus metrics have been updated in Luna Streaming 3.1.

Prometheus Client version has changed from 0.5.0 to 0.16.0
Prometheus Metric type UNTYPED is renamed to UNKNOWN
Metrics have been renamed because OpenMetrics’s counter name needs a _total suffix

Renamed metrics
Luna Streaming 2.10	Luna Streaming 3.1
pulsar_expired_token_count	pulsar_expired_token_total
pulsar_authentication_success_count	pulsar_authentication_success_total
pulsar_authentication_failures_count	pulsar_authentication_failures_total
pulsar_source_received_total_1min	pulsar_source_received_1min_total
pulsar_source_written_total_1min	pulsar_source_written_1min_total
pulsar_source_source_total_1min	pulsar_source_source_exceptions_1min_total
pulsar_source_system_exceptions_total_1min	pulsar_source_system_exceptions_1min_total
pulsar_function_received_total_1min	pulsar_function_received_1min_total
pulsar_function_user_exceptions_total_1min	pulsar_function_user_exceptions_1min_total
pulsar_function_system_exceptions_total_1min	pulsar_function_system_exceptions_1min_total
pulsar_function_processed_successfully_total_1min	pulsar_function_processed_successfully_1min_total
pulsar_sink_received_total_1min	pulsar_sink_received_1min_total
pulsar_sink_written_total_1min	pulsar_sink_written_1min_total
pulsar_sink_sink_exceptions_total_1min	pulsar_sink_sink_exceptions_1min_total
pulsar_sink_system_exceptions_total_1min	pulsar_sink_system_exceptions_1min_total
pulsar_lb_unload_broker_count	pulsar_lb_unload_broker_total
pulsar_lb_unload_bundle_count	pulsar_lb_unload_bundle_total
pulsar_lb_bundles_split_count	pulsar_lb_bundles_split_total
pulsar_schema_del_ops_failed_count	pulsar_schema_del_ops_failed_total
pulsar_schema_get_ops_failed_count	pulsar_schema_get_ops_failed_total
pulsar_schema_put_ops_failed_count	pulsar_schema_put_ops_failed_total
pulsar_schema_compatible_count	pulsar_schema_compatible_total
pulsar_schema_incompatible_count	pulsar_schema_incompatible_total
pulsar_txn_committed_count	pulsar_txn_committed_total
pulsar_txn_aborted_count	pulsar_txn_aborted_total
pulsar_txn_created_count	pulsar_txn_created_total
pulsar_txn_timeout_count	pulsar_txn_timeout_total
pulsar_txn_append_log_count	pulsar_txn_append_log_total

The following PRs were merged to update metrics:

Other functional impacts

The following PRs were merged in Luna Streaming 3.1 that may impact your deployment’s functionality.

PR Link	Title	Functional Impact
#19180	Deprecate blocking AuthorizationService, AuthorizationProvider methods	This will affect the public API for the AuthorizationService and the AuthorizationProvider, which only impacts users that are running custom code inside the Pulsar Broker
#19182	Remove AuthorizationProvider methods deprecated in 2.7 and 2.9	Removing deprecated methods allowTenantOperationAsync, allowTenantOperation, allowNamespaceOperationAsync, allowNamespaceOperation, allowNamespacePolicyOperationAsync, allowNamespacePolicyOperation, allowTopicOperationAsync, allowTopicOperation. These methods could be used by third party extensions
#19197	Update AuthenticationProvider to simplify HTTP Authn	This changes the public API within the broker as some methods are marked as @Deprecated
#19295	OneStageAuth State: move authn out of constructor	This could break 3rd party plugins in the broker if they were relying on authentication to happen in the constructor. In order to make those implementations fail fast, this PR includes a change to throw an exception when the getAuthRole is called without first calling authenticateAsync or authenticate. That makes these changes semi-backwards compatible.
#19314	TokenAuthenticationState: authenticate token only once	In a sense, this breaks an implicit contract that the class had. However, because the getAuthRole() method will throw an exception if called incorrectly, it is likely that misuse of this class will result in a fail fast behavior.
#19455	Require authRole is proxyRole to set originalPrincipal	This change affects the binary protocol’s usage without changing the binary protocol itself. Upgrading existing proxies will not work if the proxyRoles is not correctly configured in the broker.conf.
#19486	Remove default 30s ackTimeout when setting DLQ policy on java consumer	Removed setting default ackTimeoutMillis in java ConsumerBuilder when a deadLetterPolicy is set. It has to be specified exclusively to use.

PR Link

Title

Functional Impact

#19180

Deprecate blocking AuthorizationService, AuthorizationProvider methods

This will affect the public API for the AuthorizationService and the AuthorizationProvider, which only impacts users that are running custom code inside the Pulsar Broker

#19182

Remove AuthorizationProvider methods deprecated in 2.7 and 2.9

Removing deprecated methods allowTenantOperationAsync, allowTenantOperation, allowNamespaceOperationAsync, allowNamespaceOperation, allowNamespacePolicyOperationAsync, allowNamespacePolicyOperation, allowTopicOperationAsync, allowTopicOperation. These methods could be used by third party extensions

#19197

Update AuthenticationProvider to simplify HTTP Authn

This changes the public API within the broker as some methods are marked as @Deprecated

#19295

OneStageAuth State: move authn out of constructor

This could break 3rd party plugins in the broker if they were relying on authentication to happen in the constructor. In order to make those implementations fail fast, this PR includes a change to throw an exception when the getAuthRole is called without first calling authenticateAsync or authenticate. That makes these changes semi-backwards compatible.

#19314

TokenAuthenticationState: authenticate token only once

In a sense, this breaks an implicit contract that the class had. However, because the getAuthRole() method will throw an exception if called incorrectly, it is likely that misuse of this class will result in a fail fast behavior.

#19455

Require authRole is proxyRole to set originalPrincipal

This change affects the binary protocol’s usage without changing the binary protocol itself. Upgrading existing proxies will not work if the proxyRoles is not correctly configured in the broker.conf.

#19486

Remove default 30s ackTimeout when setting DLQ policy on java consumer

Removed setting default ackTimeoutMillis in java ConsumerBuilder when a deadLetterPolicy is set. It has to be specified exclusively to use.

Configuration impacts

This section describes changes in Luna Streaming 3.1 that may impact your deployment’s configuration.

Configuration values removed in 3.1

PR #14506 removes managedLedgerNumWorkerThreads. The MetadataStore instance is now passed from the PulsarService directly to the ManagedLedgerFactory.
The Pulsar SQL conf/presto directory has been removed.
- If you’re upgrading Pulsar SQL from 2.11 or earlier, copy the Pulsar SQL config files from conf/presto to trino/conf.
- If you’re downgrading Pulsar SQL to 2.11 or earlier from newer versions, copy the Pulsar SQL config files from trino/conf to conf/presto.

Default values changed or deprecated in 3.1

The following default values in broker.conf and standalone.conf have changed or been deprecated in Luna Streaming 3.1.

`broker.conf` and `standalone.conf` values
Configuration	Luna Streaming 2.10 Default	Luna Streaming 3.1 Default
managedLedgerCacheEvictionFrequency`	`100.0`	`0`
`managedLedgerMaxUnackedRangesToPersistInZooKeeper`	`1000`	`-1`
`systemTopicEnabled`	`false`	`true`
`topicLevelPoliciesEnabled`	`false`	`true`
`supportedNamespaceBundleSplitAlgorithms`	`range_equally_divide`,`topic_count_equally_divide`,`specified_positions_divide`	`range_equally_divide`,`topic_count_equally_divide`,`specified_positions_divide`,`flow_or_qps_equally_divide`
`loadBalancerDirectMemoryResourceWeight`	`1.0`	`0`
`fileSystemProfilePath`	`../conf/filesystem_offload_core_site.xml`	`conf/filesystem_offload_core_site.xml`
`gcsManagedLedgerOffloadMaxBlockSizeInBytes`	`67108864`	`134217728`

Operational impacts

This section describes changes in Luna Streaming 3.1 that may impact how your deployment operates.

JDK 17 upgrade

Luna Streaming 3.1 uses JDK 17.

The Pulsar server module’s javac release version is 17.

Client and client-server shared modules remain at the target Java 8 release.

This modification is described in detail in PIP-156.

Python 2 support removed

Luna Streaming 3.1 removes Python 2 from build scripts.

Python 3 is used in the build image.

The build image is updated to use ubuntu:20.04, as there is no Python 3.7 support in the previous Ubuntu image.

Executable scripts have been updated to invoke python3 instead of python.

This modification is described in detail in PIP-155

Known issues

This section describes known issues encountered when upgrading to Luna Streaming 3.1.

Bookkeeper / RocksDB format

Downgrading to Luna Streaming 2.10 from Luna Streaming 3.1 is not supported for Bookies and ZooKeeper.

Pulsar 3.1 uses RocksDB 7.x, which writes in a format that is not compatible with RocksDB 6.x.

Luna Streaming 2.10 uses Bookkeeper 4.14, which uses RocksDB 6.x.

All other components such as Broker, Proxy, and Functions Worker can be downgraded at any time.

For more information, see Issue 22051.

Upgrade procedure

Luna Streaming can be deployed on bare metal, Docker, and Kubernetes.

This guide only addresses Kubernetes deployment.

For more information on upgrading bare metal and Docker Pulsar deployments, see the Pulsar documentation.

Upgrade Kubernetes deployment with KAAP Operator

Upgrade to Luna Streaming 3.1 on Kubernetes with the KAAP (Kubernetes Autoscaling for Apache Pulsar) operator.

For more information, see the KAAP documentation.

To prevent data loss, back up your existing Pulsar data and configuration files.
To save your current Helm release configuration, run the following command:
```
helm get values RELEASE-NAME > pulsar-backup-values.yaml
```
To update the DataStax Pulsar Helm chart repository, run the following command:
```
helm repo update
```

Open helm/kaap-stack/values.yaml, and then update the image tag to 3.1.0 (or the specific tag you wish to use).

kaap:
  enabled: true
  cluster:
    name: pulsar
    create: true
  spec:
    global:
      name: pulsar
      image:
        datastax/lunastreaming-all: 3.1_4.5

To modify other configurations, update values.yaml as needed. For example, to modify the broker’s namespace shedding and splitting configurations, update the following fields:

kaap:
  enabled: true
  cluster:
    name: pulsar
    create: true
  spec:
    global:
      name: pulsar
    broker:
      replicas: 2
      config:
        loadBalancerNamespaceBundleSplitConditionHitCountThreshold: 1
        loadBalancerSheddingConditionHitCountThreshold: 1e

To upgrade your existing Pulsar installation, run the following Helm command. This command assumes the default pulsar namespace. If you are using a different namespace, replace pulsar with your namespace. The --wait flag ensures that Helm waits until all pods are ready before completing the upgrade.
```
helm upgrade --namespace NAMESPACE --wait --debug --timeout 1200s \
--dependency-update pulsar KAAP-REPO-DIRECTORY/helm/kaap-stack \
--values PATH-TO-CURRENT-VALUES-FILE.yaml
```
Check the status of the pods to ensure they are running correctly:
```
kubectl get pods --namespace NAMESPACE
```
Check the logs for any issues:
```
kubectl logs POD-NAME -n NAMESPACE
```
After the upgrade, ensure all necessary configurations are in place and correct.

Upgrade Kubernetes deployment with Helm chart

The Helm chart for Luna Streaming is available in the Helm chart sources repository.

To prevent data loss, back up your existing Pulsar data and configuration files.
To save your current Helm release configuration, run the following command:
```
helm get values RELEASE-NAME > pulsar-backup-values.yaml
```
To update the DataStax Pulsar Helm chart repository, run the following command:
```
helm repo update
```

Open helm-chart-sources/pulsar/values.yaml and update the image tag to 3.1.0 (or the specific tag you wish to use).

image:
  broker:
    # If not using tiered storage, you can use the smaller pulsar image
    # for the broker
    repository: datastax/lunastreaming-all
    pullPolicy: IfNotPresent
    tag: 3.1_4.5
  brokerSts:
    # If not using tiered storage, you can use the smaller pulsar image
    # for the broker
    repository: apachepulsar/pulsar
    pullPolicy: IfNotPresent
    tag: latest
  function:
    repository: apachepulsar/pulsar
    pullPolicy: IfNotPresent
    tag: latest
  zookeeper:
    repository: apachepulsar/pulsar
    pullPolicy: IfNotPresent
    tag: latestupgr
  bookkeeper:
    repository: apachepulsar/pulsar
    pullPolicy: IfNotPresent
    tag: latest
  proxy:
    repository: apachepulsar/pulsar
    pullPolicy: IfNotPresent
    tag: latest
  bastion:
    repository: apachepulsar/pulsar
    pullPolicy: IfNotPresent
    tag: latest

Review and modify any other configuration parameters that may have changed between versions, such as resource limits, storage classes, and additional components. To modify other configurations, update values.yaml as needed. For example, to modify the broker’s replica count, update the following fields:
```
broker:
  component: broker
  replicaCount: 2
  configData:
    brokerDeduplicationEnabled: "false"
```
To upgrade your existing Pulsar installation, run the following Helm command. This command assumes the default pulsar namespace. If you are using a different namespace, replace pulsar with your namespace. The --wait flag ensures that Helm waits until all pods are ready before completing the upgrade.
```
helm upgrade --namespace NAMESPACE --wait --debug --timeout 1200s \
--dependency-update pulsar KAAP-REPO-DIRECTORY/helm/kaap-stack \
--values PATH-TO-CURRENT-VALUES-FILE.yaml
```
To check the status of the pods to ensure they are running correctly, run the following command:
```
kubectl get pods --namespace NAMESPACE
```
To check the logs for any issues, run the following command:
```
kubectl logs POD-NAME -n NAMESPACE
```
After the upgrade, ensure all necessary configurations are in place and correct.