Failover subscriptions in Pulsar

Subscriptions in Pulsar describe which consumers are consuming data from a topic and how they want to consume that data.

If an exclusive consumer fails, the topic backlog balloons with stale, unacknowledged messages. Pulsar solves this problem with failover subscriptions.
In failover subscriptions, Pulsar designates one primary consumer and multiple standby consumers. If the primary consumer disconnects, the standby consumers begin consuming the subsequent unacknowledged messages.

This document explains how to use Pulsar’s failover subscription model to manage your topic consumption.

Prerequisites

To run this example, you’ll need:

Apache Maven
Java OpenJDK 11
A configured Astra Streaming instance with at least one streaming tenant and one topic. See the Astra Streaming quick start for instructions.
A local clone of the DataStax Pulsar Subscription Example repository
Modify the src/main/resources/application.properties in the pulsar-subscription-example repo to connect to your Astra Streaming cluster, as below:
```
service_url={broker-service-url}
namespace=default
tenant_name=my-tenant
authentication_token={astra-auth-token}
topic_name=my-topic
```

Failover subscription example

To try out a Pulsar failover subscription, add .subscriptionType(SubscriptionType.Failover) to the pulsarConsumer in SimplePulsarConsumer.java.

pulsarConsumer = pulsarClient.newConsumer(Schema.JSON(DemoBean.class))
    .topic("persistent://"
        + conf.getTenantName() + "/"
        + conf.getNamespace() + "/"
        + conf.getTopicName())
    .startMessageIdInclusive()
    .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
    .subscriptionName("SimplePulsarConsumer")
    .subscriptionType(SubscriptionType.Failover)
    .subscribe();

Open pulsar-subscription-example in the IDE of your choice and run SimplePulsarConsumer.java to begin consuming messages.
This is the primary consumer.
The confirmation message and a cursor appear to indicate the consumer is ready.

[main] INFO com.datastax.pulsar.Configuration - Configuration has been loaded successfully
...
[pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://<tenant_name>/<namespace>/in][SimplePulsarConsumer] Subscribed to topic on <service_url> -- consumer: 0

In a new terminal window, run SimplePulsarProducer.java to begin producing messages.

[main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 50585599 sent
[main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 98055337 sent
[main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 36327100 sent
...
[main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 73260535 sent
[main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 42372149 sent

The primary consumer begins consuming messages.

[main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":50585599,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"}
[main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":98055337,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"}
[main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":36327100,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"}

Open a new terminal window and run SimplePulsarConsumer.java. This is the backup consumer. The backup consumer subscribes to the topic, but does not immediately begin consuming messages.

Return to the primary consumer and Ctrl+C to stop it. The backup consumer begins consuming right where the first consumer left off.

[main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":73260535,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"}
[main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":42372149,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"}

You can repeat this process with as many primary and backup consumers as you want- the next message is delivered to the subscribed failover consumers.

Failover subscription video

Follow along with this video from our Five Minutes About Pulsar series to see failover subscriptions in action.

What’s next?

For more on subscriptions, see: