AMQP 0.9.1, the protocol RabbitMQ uses, employs the concepts of Exchanges, Queues, and Bindings to provide basic routing capabilities inside the message broker. These concepts are mapped to Pulsar topics and features.

Starlight for RabbitMQ acts as a proxy and uses the Pulsar binary protocol to communicate with the brokers. This method allows RabbitMQ applications to leverage Pulsar features like load balancing of the topics on the brokers, batching of messages, partitioning of topics, and load balancing of the data on the consumers.

On the publish side, an AMQP exchange is mapped to a topic. Depending on the type of exchange, the publish routing key may also be included in the topic name.

On the consumer side, Pulsar shared subscriptions are used to represent the AMQP Bindings from an Exchange to a Queue. When creating an AMQP Queue consumer, the proxy creates Pulsar consumers for all the Bindings of the Queue.

When you unbind the Queue, the Pulsar subscription isn’t deleted right away since the consumer may be lagging. Messages are still received from the subscription and filtered if their position is past the end of the binding. Starlight for RabbitMQ periodically checks if all messages from the closed binding have been acknowledged. Once the messages have been acknowledged, Starlight for RabbitMQ deletes the corresponding subscription.

The following capabilities have not been implemented:

Because RabbitMQ and Pulsar work differently, Starlight for RabbitMQ was designed to make the most of Pulsar’s scalability. This means that canceling an AMQP consumer will re-queue the messages that were received through it because it also closes the associated Pulsar consumers.

Starlight for RabbitMQ can be run in various deployments:

Starlight for RabbitMQ uses Apache ZooKeeper™ to store the AMQP entities metadata consistently. The existing ZooKeeper configuration store can be reused to store the Starlight for RabbitMQ metadata. Starlight for RabbitMQ employs the /starlight-rabbitmq prefix to write its entries in ZooKeeper.

Starlight for RabbitMQ supports connections using TLS/mTLS to ensure privacy and security of the communication. It also supports the PLAIN and EXTERNAL mechanisms used by RabbitMQ. Internally, it uses the same AuthenticationService as Pulsar and maps these mechanisms to existing Pulsar authentication modes. At the moment there is no support for authorization so an authenticated user has full access to all Virtual hosts. Starlight for RabbitMQ can connect to brokers that have TLS, authentication, and/or authorization enabled.

To perform its operations, Starlight for RabbitMQ proxy needs to use an admin role.

Multiple Starlight for RabbitMQ instances can be launched at the same time for scalability and high availability needs. The Starlight for RabbitMQ instances are stateless and can be started and stopped at will. They share their configuration in ZooKeeper so you can create, delete, bind, and unbind exchanges and queues on any Starlight for RabbitMQ instance. The configuration will be synchronized on the other Starlight for RabbitMQ instances.

Publishing messages can be done on any Starlight for RabbitMQ instance. On the receiving side, messages will be dispatched evenly to all connected AMQP consumers because the Pulsar subscriptions are shared.

Starlight for RabbitMQ offers support for multi-tenancy by mapping an AMQP Virtual host to a Pulsar tenant and namespace. The mapping is done as follows:

This means that AMQP vhosts must contain only characters that are accepted in Pulsar tenant and namespace names, such as a-zA-Z0-9_-=:..