Kafka Source Connector
The Kafka Source Connector pulls messages from Kafka topics and persists the messages to Pulsar topics.
This guide explains how to configure and use the Kafka Source Connector in DataStax Astra Streaming.
To create a Kafka Source Connector, visit the Sources tab in Astra Streaming and choose Create Source. This step opens the Create New Source page with a list of fields required to build your new Kafka Source Connector.
In your Advanced Settings, control the number of instances, auto-acknowledgement, and timeout. Processing Guarantee offers three options:
ATLEAST_ONCE: Each message sent to the source can be processed more than once.
ATMOST_ONCE: The message sent to the source is processed at most once. Therefore, there is a chance that the message is not processed.
EFFECTIVELY_ONCE: Each message sent to the source will have one output associated with it.
Group-ID is a unique string that identifies the consumer group that this consumer belongs to.
The bootstrap server
port pairs must be in this format:
The Kafka topic defined in the source-specific configuration will receive messages from Pulsar:
You can start, stop, and restart your source by selecting it in the Sources dashboard.
Sources produce logs to help you in debugging. To view your source’s logs, open your source in the Sources dashboard.
In the upper right corner of the source log are controls to Refresh, Copy to Clipboard, and Save your log.
To update your source, select your source in the Sources dashboard.
Update your source’s settings. When you’re done, click Update.
An Updates Submitted Successfully flag will appear to let you know your source has been updated.
To delete a source, select the source to be deleted in the Sources dashboard.
A popup will ask you to confirm deletion by entering the source’s name and clicking Delete.
A Source-name Deleted Successfully! flag will appear to let you know you’ve deleted your source.
For more information on configuring your Kafka source connector, visit Kafka Source Configuration.