Failover subscriptions in Pulsar
Subscriptions in Pulsar describe which consumers are consuming data from a topic and how they want to consume that data.
If an exclusive consumer fails, the topic backlog balloons with stale, unacknowledged messages. Pulsar solves this problem with failover subscriptions.
In failover subscriptions, Pulsar designates one primary consumer and multiple standby consumers. If the primary consumer disconnects, the standby consumers begin consuming the subsequent unacknowledged messages.
This document explains how to use Pulsar’s failover subscription model to manage your topic consumption.
Prerequisites
To run this example, you’ll need:
-
A configured Astra Streaming instance with at least one streaming tenant and one topic. See the Astra Streaming quick start for instructions.
-
A local clone of the DataStax Pulsar Subscription Example repository
-
Modify the
src/main/resources/application.propertiesin thepulsar-subscription-examplerepo to connect to your Astra Streaming cluster, as below:service_url={broker-service-url} namespace=default tenant_name=my-tenant authentication_token={astra-auth-token} topic_name=my-topic
Failover subscription example
To try out a Pulsar failover subscription, add .subscriptionType(SubscriptionType.Failover) to the pulsarConsumer in SimplePulsarConsumer.java.
pulsarConsumer = pulsarClient.newConsumer(Schema.JSON(DemoBean.class))
.topic("persistent://"
+ conf.getTenantName() + "/"
+ conf.getNamespace() + "/"
+ conf.getTopicName())
.startMessageIdInclusive()
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
.subscriptionName("SimplePulsarConsumer")
.subscriptionType(SubscriptionType.Failover)
.subscribe();-
Open
pulsar-subscription-examplein the IDE of your choice and runSimplePulsarConsumer.javato begin consuming messages.
This is the primary consumer.
The confirmation message and a cursor appear to indicate the consumer is ready.[main] INFO com.datastax.pulsar.Configuration - Configuration has been loaded successfully ... [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://<tenant_name>/<namespace>/in][SimplePulsarConsumer] Subscribed to topic on <service_url> -- consumer: 0 -
In a new terminal window, run
SimplePulsarProducer.javato begin producing messages.[main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 50585599 sent [main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 98055337 sent [main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 36327100 sent ... [main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 73260535 sent [main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 42372149 sent -
The primary consumer begins consuming messages.
[main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":50585599,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"} [main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":98055337,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"} [main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":36327100,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"} -
Open a new terminal window and run
SimplePulsarConsumer.java. This is the backup consumer. The backup consumer subscribes to the topic, but does not immediately begin consuming messages. -
Return to the primary consumer and
Ctrl+Cto stop it. The backup consumer begins consuming right where the first consumer left off.[main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":73260535,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"} [main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":42372149,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"} -
You can repeat this process with as many primary and backup consumers as you want- the next message is delivered to the subscribed failover consumers.
Failover subscription video
Follow along with this video from our Five Minutes About Pulsar series to see failover subscriptions in action.
What’s next?
For more on subscriptions, see:
