A guide to deploying Luna Streaming with the Starlight for Kafka extension
Starlight for Kafka brings the native Apache Kafka protocol support to Apache Pulsar by introducing a Kafka protocol handler on Pulsar brokers. By adding the Starlight for Kafka protocol handler to your Pulsar cluster, you can migrate your existing Kafka applications and services to Pulsar without modifying the code.
You will need the following prerequisites in place to complete this guide:
Helm 3 CLI↗ (we used version 3.8.0)
Kafka CLI↗ (we used version 3.3.1)
Kubectl CLI↗ (we used version 1.23.4)
Enough access to a K8s cluster to create a namespace, deployments, and pods
Install Luna Streaming helm chart
Add the DataStax Helm chart repo to your Helm store.
helm repo add datastax-pulsar https://datastax.github.io/pulsar-helm-chart
Install the Helm chart using a minimalist values file.
This command creates a Helm release named "my-pulsar-cluster" using the DataStax Luna Helm chart, within the K8s namespace "datastax-pulsar". The minimal cluster creates only the essential components and has no ingress or load balanced services.
VALUES_URL="https://raw.githubusercontent.com/datastaxdevs/luna-streaming-examples/main/starlight-for-kafka/values.yaml" helm install \ --namespace datastax-pulsar \ --create-namespace \ --values $VALUES_URL \ --version 3.0.4 \ my-pulsar-cluster \ datastax-pulsar/pulsar
Wait for the broker pod to be in a running state. You might see a few restarts as your components start up.
kubectl -n datastax-pulsar wait --for=condition=Ready pod/pulsar-broker-0 --timeout=120s
Forward service port
We will need to interact with a few of the services in the K8s cluster. Let’s map a few ports to those services.
In a new terminal, port forward Pulsar’s admin service.
kubectl port-forward -n datastax-pulsar service/pulsar-broker 8080:8080
In a separate terminal window, port forward the Starlight for Kafka serivce.
kubectl port-forward -n datastax-pulsar service/pulsar-proxy 9092:9092
Have a look around
The Luna Streaming Helm Chart automatically creates a tenant named "public" and a namespace within that tenant named "default".
The Starlight for Kafka extension creates a few namespaces and topics to function correctly.
List the namespaces in the "public" tenant to see what was created.
~/apache-pulsar-2.10.1$ ./bin/pulsar-admin namespaces list public
The output should be similar to the following.
public/__kafka public/__kafka_producerid public/default
Notice the namespaces prefixed with "__kafka". These are used by the service for different functions. To learn more about Starlight for Kafka operations, see the S4K documentation.
Produce a message with the Kafka CLI
If you hadn’t noticed, we never opened the Pulsar binary port to accept new messages. Only the admin port and the Kafka port are open. To further show how native Starlight for Kafka is to Pulsar, we will use the Kafka CLI to produce and consume messages from Pulsar.
From within the Kafka directory, run the following command to start the shell.
~/kafka_2.13-3.3.1$ ./bin/kafka-console-producer.sh --topic quickstart --bootstrap-server localhost:9092
Type a message, press Enter to send it, then Ctrl-C to exit the producer shell.
This my first message
Wait a second! We never created a topic! And where did the "quickstart" topic come from?!
The default behavior of Starlight for Kafka is to create a new single partition, persistent topic when one is not present. You can configure this behavior and many other S4K parameters in the Helm chart↗. Learn more about the configuration values here.
Let’s have a look at the topic that was created. From your Pulsar home folder, run the following command.
~/apache-pulsar-2.10.1$ ./bin/pulsar-admin topics list public/default
The output will include the newly created topic.
Consume the new message with the Kafka CLI
Let’s use the Kafka CLI to consume the message we just produced. Start the consumer shell from the Kafka home folder with the following command.
~/kafka_2.13-3.3.1$ ./bin/kafka-console-consumer.sh --topic quickstart --from-beginning --bootstrap-server localhost:9092
The data of our new message will be output. Enter Ctrl-C to exit the shell.
This my first message
Pulsar and Starlight for Kafka are a powerful pair!
Kafka users and existing applications using Kafka can enjoy the many benefits of a Pulsar cluster, while never having to change tooling or libraries. Other folks that are more comfortable with Pulsar tooling and clients can also interact with the same topics. Together, new and legacy applications work together to create modern solutions.
Here are links to other guides and resource you might be interested in.