Shared subscriptions in Pulsar

Subscriptions in Pulsar describe which consumers are consuming data from a topic and how they want to consume that data.

A shared subscription allows multiple consumers to consume messages from a single topic in a round-robin fashion. More consumers in a shared subscription can increase your Pulsar deployment’s rate of message consumption. However, there is a risk of losing message ordering guarantees and acknowledgment schemes.

This page explains how you can use Pulsar’s shared subscription model to manage your topic consumption.

Prerequisites

This example requires the following:

Apache Maven
Java OpenJDK 11
Astra Streaming access with at least one streaming tenant and one topic
A local clone of the DataStax Pulsar Subscription Example repository
In the pulsar-subscription-example repo, navigate to src/main/resources, and then edit application.properties to connect to your Astra Streaming cluster:
application.properties
```
service_url=BROKER_SERVICE_URL
namespace=default
tenant_name=my-tenant
authentication_token=ASTRA_DB_APPLICATION_TOKEN
topic_name=my-topic
```

Shared subscription example

To try out a Pulsar shared subscription, add .subscriptionType(SubscriptionType.Shared) to the pulsarConsumer in SimplePulsarConsumer.java:

SimplePulsarConsumer.java

pulsarConsumer = pulsarClient.newConsumer(Schema.JSON(DemoBean.class))
    .topic("persistent://"
        + conf.getTenantName() + "/"
        + conf.getNamespace() + "/"
        + conf.getTopicName())
    .startMessageIdInclusive()
    .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
    .subscriptionName("SimplePulsarConsumer")
    .subscriptionType(SubscriptionType.Shared)
    .subscribe();

In the pulsar-subscription-example project, run SimplePulsarConsumer.java to begin consuming messages.

The confirmation message and a cursor appear to indicate the consumer is ready:

Result

[main] INFO com.datastax.pulsar.Configuration - Configuration has been loaded successfully
...
[pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://<tenant_name>/<namespace>/in][SimplePulsarConsumer] Subscribed to topic on <service_url> -- consumer: 0

In a new terminal window, run SimplePulsarProducer.java to begin producing messages:

Result

[main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 59819331 sent
[main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 70129519 sent
[main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 31365142 sent
[main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 48206643 sent
[main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 51277375 sent

In the SimplePulsarConsumer terminal, the consumer begins receiving messages:

Result

[main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":59819331,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"}
[main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":31365142,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"}
[main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":51277375,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"}

In a new terminal window, run another instance of SimplePulsarConsumer.java.

The new consumer subscribes to the topic and consumes messages:

Result

[main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":70129519,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"}
[main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":48206643,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"}

Because this test uses shared subscriptions, you can attach multiple consumers to the topic. If you run this test with exclusive subscriptions, you can’t attach more than once subscriber to the exclusive topic.

To continue testing the shared subscription configuration, you can continue running new instances of SimplePulsarConsumer.java in new terminal windows. All the consumers subscribe to the topic and consume messages in a round-robin fashion.

Shared subscriptions in Pulsar

Prerequisites

Shared subscription example

See also

Was this helpful?

Give Feedback