Starlight for RabbitMQ™

Starlight for RabbitMQ™ acts as a proxy between your RabbitMQ application and Apache Pulsar™. It implements the AMQP 0.9.1 protocol used by RabbitMQ clients and translates AMQP frames and concepts to Pulsar ones. Starlight for RabbitMQ™ can be run as a standalone Java application, a Pulsar Pluggable Protocol Handler, or a Pulsar Proxy extension.

How Starlight for RabbitMQ™ works

AMQP 0.9.1, the protocol RabbitMQ uses, employs the concepts of Exchanges, Queues, and Bindings to provide basic routing capabilities inside the message broker. These concepts are mapped to Pulsar topics and features.

Starlight for RabbitMQ™ acts as a proxy and uses the Pulsar binary protocol to communicate with the brokers. This method allows RabbitMQ applications to leverage Pulsar features like load balancing of the topics on the brokers, batching of messages, partitioning of topics, and load balancing of the data on the consumers.

On the publish side, an AMQP exchange is mapped to a topic. Depending on the type of exchange, the publish routing key may also be included in the topic name.

On the consumer side, Pulsar shared subscriptions are used to represent the AMQP Bindings from an Exchange to a Queue. When creating an AMQP Queue consumer, the proxy creates Pulsar consumers for all the Bindings of the Queue.

When you unbind the Queue, the Pulsar subscription isn’t deleted right away since the consumer may be lagging. Messages are still received from the subscription and filtered if their position is past the end of the binding. Starlight for RabbitMQ™ periodically checks if all messages from the closed binding have been acknowledged. Once the messages have been acknowledged, Starlight for RabbitMQ™ deletes the corresponding subscription.

Limitations

The following capabilities have not been implemented:

  • Topic and headers exchanges

  • Exclusive consumers

  • Non durable exchanges and queues

  • Transient messages (all messages are persisted)

  • Authorization support

Because RabbitMQ and Pulsar work differently, Starlight for RabbitMQ™ was designed to make the most of Pulsar’s scalability. This means that canceling an AMQP consumer will re-queue the messages that were received through it because it also closes the associated Pulsar consumers.

Deployment options

Starlight for RabbitMQ™ can be run in various deployments:

  • Standalone Java application Running Starlight for RabbitMQ™ as a standalone Java application has the advantage of not impacting the resources used by the Pulsar cluster but on the other hand you need to deploy and manage the lifecycle of an additional runtime.

  • Pulsar Pluggable Protocol Handler Deploying Starlight for RabbitMQ™ in the broker as a Protocol Handler avoids having to deploy another runtime but there will be resources used by Starlight for RabbitMQ™ to handle the traffic that can have an impact on the load balancing algorithms used by Pulsar.

  • Pulsar Proxy extension Deploying Starlight for RabbitMQ™ in the Pulsar Proxy as a Proxy Extension avoids having to deploy another runtime if you already use the Pulsar Proxy and it will not have a big impact on the Pulsar Proxy functioning. Note that Proxy Extensions will be available in Pulsar v2.9+.

Consistent metadata store

Starlight for RabbitMQ™ uses Apache ZooKeeper™ to store the AMQP entities metadata consistently. The existing ZooKeeper configuration store can be reused to store the Starlight for RabbitMQ™ metadata. Starlight for RabbitMQ™ employs the /starlight-rabbitmq prefix to write its entries in ZooKeeper.

Security and authentication

Starlight for RabbitMQ™ supports connections using TLS/mTLS to ensure privacy and security of the communication. It also supports the PLAIN and EXTERNAL mechanisms used by RabbitMQ. Internally, it uses the same AuthenticationService as Pulsar and maps these mechanisms to existing Pulsar authentication modes. At the moment there is no support for authorization so an authenticated user has full access to all Virtual hosts. Starlight for RabbitMQ™ can connect to brokers that have TLS, authentication, and/or authorization enabled.

To perform its operations, Starlight for RabbitMQ™ proxy needs to use an admin role.

PLAIN authentication mechanism

The PLAIN mechanism is mapped to the AuthenticationProviderToken mode of authentication. The username is ignored and the password is used as the JSON Web Token (JWT).

EXTERNAL authentication mechanism

The EXTERNAL mechanism is mapped to the AuthenticationProviderTls mode of authentication. This is the equivalent of the rabbitmq-auth-mechanism-ssl plugin with ssl_cert_login_from parameter set to common_name.

Clustering

Multiple Starlight for RabbitMQ™ instances can be launched at the same time for scalability and high availability needs. The Starlight for RabbitMQ™ instances are stateless and can be started and stopped at will. They share their configuration in ZooKeeper so you can create, delete, bind, and unbind exchanges and queues on any Starlight for RabbitMQ™ instance. The configuration will be synchronized on the other Starlight for RabbitMQ™ instances.

Publishing messages can be done on any Starlight for RabbitMQ™ instance. On the receiving side, messages will be dispatched evenly to all connected AMQP consumers because the Pulsar subscriptions are shared.

Multi-tenancy

Starlight for RabbitMQ™ offers support for multi-tenancy by mapping an AMQP Virtual host to a Pulsar tenant and namespace. The mapping is done as follows:

  • AMQP vhost / is mapped to Pulsar namespace public/default.

  • AMQP vhost /<tenant> or <tenant> is mapped to Pulsar namespace public/<tenant>.

  • AMQP vhost /<tenant>/<namespace> or <tenant>/<namespace> is mapped to Pulsar namespace <tenant>/<namespace>.

This means that AMQP vhosts must contain only characters that are accepted in Pulsar tenant and namespace names, such as a-zA-Z0-9_-=:..