Kafka Source Connector

The Kafka Source Connector pulls messages from Kafka topics and persists the messages to Pulsar topics.

This guide explains how to configure and use the Kafka Source Connector in DataStax Astra Streaming.

Adding a Kafka Source Connector in Datastax Astra Streaming

To create a Kafka Source Connector, visit the Sources tab in Astra Streaming and choose Create Source. This step opens the Create New Source page with a list of fields required to build your new Kafka Source Connector.

Basics

Advanced Settings

In your Advanced Settings, control the number of instances, auto-acknowledgement, and timeout. Processing Guarantee offers three options:

  • ATLEAST_ONCE: Each message sent to the source can be processed more than once.

  • ATMOST_ONCE: The message sent to the source is processed at most once. Therefore, there is a chance that the message is not processed.

  • EFFECTIVELY_ONCE: Each message sent to the source will have one output associated with it.

Advanced

Connect Topics

Choose which topic messages will be published from the connected source:

Connect Topics

Source-Specific Configuration

Group-ID is a unique string that identifies the consumer group that this consumer belongs to.

The bootstrap server host/port pairs must be in this format: hostname1:port1,hostname2:port2,…​

The Kafka topic defined in the source-specific configuration will receive messages from Pulsar:

Source Specific Configuration

Consumer Configuration Properties

Currently, only SASL_SSL is supported. Your username and password are required.

Consumer Configuration Properties

Review and Submit

Confirm your values are correct and select "Create". Confirm you’ve created your new Kafka Source Connector in the Sources tab.

Controlling Your Source

You can start, stop, and restart your source by selecting it in the Sources dashboard.

Source Controls

Monitoring Your Source

Sources produce logs to help you in debugging. To view your source’s logs, open your source in the Sources dashboard.

In the upper right corner of the source log are controls to Refresh, Copy to Clipboard, and Save your log.

Updating Your Source

To update your source, select your source in the Sources dashboard.

  1. Update your source’s settings. When you’re done, click Update.

  2. An Updates Submitted Successfully flag will appear to let you know your source has been updated.

Deleting Your Source

To delete a source, select the source to be deleted in the Sources dashboard.

  1. Click Delete.

  2. A popup will ask you to confirm deletion by entering the source’s name and clicking Delete.

  3. A Source-name Deleted Successfully! flag will appear to let you know you’ve deleted your source.

Next

For more information on configuring your Kafka source connector, visit Kafka Source Configuration.