Failover subscriptions in Pulsar

Subscriptions in Pulsar describe which consumers are consuming data from a topic and how they want to consume that data.

If an exclusive consumer fails, the topic backlog accumulates stale, unacknowledged messages. Pulsar solves this problem with failover subscriptions. In failover subscriptions, Pulsar designates one primary consumer and multiple standby consumers. If the primary consumer disconnects, the standby consumers begin consuming the subsequent unacknowledged messages.

This page explains how to use Pulsar’s failover subscription model to manage your topic consumption.

Prerequisites

This example requires the following:

Apache Maven
Java OpenJDK 11
Astra Streaming access with at least one streaming tenant and one topic
A local clone of the DataStax Pulsar Subscription Example repository
In the pulsar-subscription-example repo, navigate to src/main/resources, and then edit application.properties to connect to your Astra Streaming cluster:
application.properties
```
service_url=BROKER_SERVICE_URL
namespace=default
tenant_name=my-tenant
authentication_token=ASTRA_DB_APPLICATION_TOKEN
topic_name=my-topic
```

Failover subscription example

To try out a Pulsar failover subscription, add .subscriptionType(SubscriptionType.Failover) to the pulsarConsumer in SimplePulsarConsumer.java:

SimplePulsarConsumer.java

pulsarConsumer = pulsarClient.newConsumer(Schema.JSON(DemoBean.class))
    .topic("persistent://"
        + conf.getTenantName() + "/"
        + conf.getNamespace() + "/"
        + conf.getTopicName())
    .startMessageIdInclusive()
    .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
    .subscriptionName("SimplePulsarConsumer")
    .subscriptionType(SubscriptionType.Failover)
    .subscribe();

In the pulsar-subscription-example project, run SimplePulsarConsumer.java to begin consuming messages as the primary consumer.

The confirmation message and a cursor appear to indicate the consumer is ready:

Result

[main] INFO com.datastax.pulsar.Configuration - Configuration has been loaded successfully
...
[pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://<tenant_name>/<namespace>/in][SimplePulsarConsumer] Subscribed to topic on <service_url> -- consumer: 0

In a new terminal window, run SimplePulsarProducer.java to begin producing messages:

Result

[main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 50585599 sent
[main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 98055337 sent
[main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 36327100 sent
...
[main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 73260535 sent
[main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 42372149 sent

In the SimplePulsarConsumer terminal, the primary consumer begins consuming messages:

Result

[main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":50585599,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"}
[main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":98055337,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"}
[main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":36327100,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"}

In a new terminal window, run a new instance of SimplePulsarConsumer.java as a backup consumer. The backup consumer subscribes to the topic but does not immediately begin consuming messages.

In the primary SimplePulsarConsumer terminal, stop the process (Ctrl+C). In the second SimplePulsarConsumer terminal, the backup consumer begins consuming messages where the first consumer left off:

Result

[main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":73260535,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"}
[main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":42372149,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"}

You can configure as many backup consumers as you like. To test them, you can progressively end each SimplePulsarConsumer process, and then check that the next backup consumer has begun receiving messages.

Failover subscriptions in Pulsar

Prerequisites

Failover subscription example

See also

Was this helpful?

Give Feedback