Using Starlight for Kafka with Luna Streaming

Starlight for Kafka brings the native Apache Kafka protocol support to Apache Pulsar by introducing a Kafka protocol handler on Pulsar brokers. By adding the Starlight for Kafka protocol handler to your Pulsar cluster, you can migrate your existing Kafka applications and services to Pulsar without modifying the code.

Prerequisites

You will need the following prerequisites in place to complete this guide:

Helm 3 CLI (we used version 3.8.0)
Kafka CLI (we used version 3.3.1)
Kubectl CLI (we used version 1.23.4)
Enough access to a K8s cluster to create a namespace, deployments, and pods

Install Luna Streaming helm chart

Add the DataStax Helm chart repo to your Helm store.

helm repo add datastax-pulsar https://datastax.github.io/pulsar-helm-chart

Install the Helm chart using a minimalist values file.
This command creates a Helm release named "my-pulsar-cluster" using the DataStax Luna Helm chart, within the K8s namespace "datastax-pulsar". The minimal cluster creates only the essential components and has no ingress or load balanced services.
```
VALUES_URL="https://raw.githubusercontent.com/datastaxdevs/luna-streaming-examples/main/starlight-for-kafka/values.yaml"
helm install \
  --namespace datastax-pulsar \
  --create-namespace \
  --values $VALUES_URL \
  --version 3.0.4 \
  my-pulsar-cluster \
  datastax-pulsar/pulsar
```
Wait for the broker pod to be in a running state. You might see a few restarts as your components start up.
```
kubectl -n datastax-pulsar wait --for=condition=Ready pod/pulsar-broker-0 --timeout=120s
```

Forward service port

We will need to interact with a few of the services in the K8s cluster. Let’s map a few ports to those services.

In a new terminal, port forward Pulsar’s admin service.

kubectl port-forward -n datastax-pulsar service/pulsar-broker 8080:8080

In a separate terminal window, port forward the Starlight for Kafka serivce.

kubectl port-forward -n datastax-pulsar service/pulsar-proxy 9092:9092

Have a look around

The Luna Streaming Helm Chart automatically creates a tenant named "public" and a namespace within that tenant named "default".

The Starlight for Kafka extension creates a few namespaces and topics to function correctly.

List the namespaces in the "public" tenant to see what was created.

~/apache-pulsar-3.1.3$ ./bin/pulsar-admin namespaces list public

The output should be similar to the following.

public/__kafka
public/__kafka_producerid
public/default

Notice the namespaces prefixed with "__kafka". These are used by the service for different functions. To learn more about Starlight for Kafka operations, see the S4K documentation.

Produce a message with the Kafka CLI

If you hadn’t noticed, we never opened the Pulsar binary port to accept new messages. Only the admin port and the Kafka port are open. To further show how native Starlight for Kafka is to Pulsar, we will use the Kafka CLI to produce and consume messages from Pulsar.

From within the Kafka directory, run the following command to start the shell.

~/kafka_2.13-3.3.1$ ./bin/kafka-console-producer.sh --topic quickstart --bootstrap-server localhost:9092

Type a message, press Enter to send it, then Ctrl-C to exit the producer shell.

This my first message

Wait a second! We never created a topic! And where did the "quickstart" topic come from?!

The default behavior of Starlight for Kafka is to create a new single partition, persistent topic when one is not present. You can configure this behavior and many other S4K parameters in the Helm chart. Learn more about the configuration values here.

Let’s have a look at the topic that was created. From your Pulsar home folder, run the following command.

~/apache-pulsar-3.1.3$ ./bin/pulsar-admin topics list public/default

The output will include the newly created topic.

persistent://public/default/quickstart-partition-0

Consume the new message with the Kafka CLI

Let’s use the Kafka CLI to consume the message we just produced. Start the consumer shell from the Kafka home folder with the following command.

~/kafka_2.13-3.3.1$ ./bin/kafka-console-consumer.sh --topic quickstart --from-beginning --bootstrap-server localhost:9092

The data of our new message will be output. Enter Ctrl-C to exit the shell.

This my first message

Summary

Pulsar and Starlight for Kafka are a powerful pair!
Kafka users and existing applications using Kafka can enjoy the many benefits of a Pulsar cluster, while never having to change tooling or libraries. Other folks that are more comfortable with Pulsar tooling and clients can also interact with the same topics. Together, new and legacy applications work together to create modern solutions.

Next steps

Here are links to other guides and resource you might be interested in.