Starlight for RabbitMQ
Starlight for RabbitMQ acts as a proxy between your RabbitMQ application and Apache Pulsar™. It implements the AMQP 0.9.1 protocol used by RabbitMQ clients and translates AMQP frames and concepts to Pulsar ones. Starlight for RabbitMQ can be run as a standalone Java application, a Pulsar Pluggable Protocol Handler, or a Pulsar Proxy extension.
AMQP 0.9.1, the protocol RabbitMQ uses, employs the concepts of
Bindings to provide basic routing capabilities inside the message broker.
These concepts are mapped to Pulsar topics and features.
Starlight for RabbitMQ acts as a proxy and uses the Pulsar binary protocol to communicate with the brokers. This method allows RabbitMQ applications to leverage Pulsar features like load balancing of the topics on the brokers, batching of messages, partitioning of topics, and load balancing of the data on the consumers.
On the publish side, an AMQP exchange is mapped to a topic. Depending on the type of exchange, the publish routing key may also be included in the topic name.
On the consumer side, Pulsar shared subscriptions are used to represent the AMQP
Bindings from an
Exchange to a
When creating an AMQP Queue consumer, the proxy creates Pulsar consumers for all the
Bindings of the
When you unbind the
Queue, the Pulsar subscription isn’t deleted right away since the consumer may be lagging.
Messages are still received from the subscription and filtered if their position is past the end of the binding.
Starlight for RabbitMQ periodically checks if all messages from the closed binding have been acknowledged. Once the messages have been acknowledged, Starlight for RabbitMQ deletes the corresponding subscription.
The following capabilities have not been implemented:
Topic and headers exchanges
Non durable exchanges and queues
Transient messages (all messages are persisted)
Because RabbitMQ and Pulsar work differently, Starlight for RabbitMQ was designed to make the most of Pulsar’s scalability. This means that canceling an AMQP consumer will re-queue the messages that were received through it because it also closes the associated Pulsar consumers.
Starlight for RabbitMQ can be run in various deployments:
Standalone Java application Running Starlight for RabbitMQ as a standalone Java application has the advantage of not impacting the resources used by the Pulsar cluster but on the other hand you need to deploy and manage the lifecycle of an additional runtime.
Pulsar Pluggable Protocol Handler Deploying Starlight for RabbitMQ in the broker as a Protocol Handler avoids having to deploy another runtime but there will be resources used by Starlight for RabbitMQ to handle the traffic that can have an impact on the load balancing algorithms used by Pulsar.
Pulsar Proxy extension Deploying Starlight for RabbitMQ in the Pulsar Proxy as a Proxy Extension avoids having to deploy another runtime if you already use the Pulsar Proxy and it will not have a big impact on the Pulsar Proxy functioning. Note that Proxy Extensions will be available in Pulsar v2.9+.
Starlight for RabbitMQ uses Apache ZooKeeper™ to store the AMQP entities metadata consistently.
The existing ZooKeeper configuration store can be reused to store the Starlight for RabbitMQ metadata.
Starlight for RabbitMQ employs the
/starlight-rabbitmq prefix to write its entries in ZooKeeper.
Starlight for RabbitMQ supports connections using TLS/mTLS to ensure privacy and security of the communication.
It also supports the
EXTERNAL mechanisms used by RabbitMQ.
Internally, it uses the same AuthenticationService as Pulsar and maps these mechanisms to existing Pulsar authentication modes.
At the moment there is no support for authorization so an authenticated user has full access to all
Starlight for RabbitMQ can connect to brokers that have TLS, authentication, and/or authorization enabled.
To perform its operations, Starlight for RabbitMQ proxy needs to use an
PLAIN mechanism is mapped to the
AuthenticationProviderToken mode of authentication.
The username is ignored and the password is used as the JSON Web Token (JWT).
EXTERNAL mechanism is mapped to the
AuthenticationProviderTls mode of authentication.
This is the equivalent of the
rabbitmq-auth-mechanism-ssl plugin with
ssl_cert_login_from parameter set to
Multiple Starlight for RabbitMQ instances can be launched at the same time for scalability and high availability needs. The Starlight for RabbitMQ instances are stateless and can be started and stopped at will. They share their configuration in ZooKeeper so you can create, delete, bind, and unbind exchanges and queues on any Starlight for RabbitMQ instance. The configuration will be synchronized on the other Starlight for RabbitMQ instances.
Publishing messages can be done on any Starlight for RabbitMQ instance. On the receiving side, messages will be dispatched evenly to all connected AMQP consumers because the Pulsar subscriptions are shared.
Starlight for RabbitMQ offers support for multi-tenancy by mapping an AMQP Virtual host to a Pulsar tenant and namespace. The mapping is done as follows:
/is mapped to Pulsar namespace
<tenant>is mapped to Pulsar namespace
<tenant>/<namespace>is mapped to Pulsar namespace
This means that AMQP vhosts must contain only characters that are accepted in Pulsar tenant and namespace names, such as