Kafka Source Connector
The Kafka Source Connector pulls messages from Kafka topics and persists the messages to Pulsar topics.
This guide explains how to configure and use the Kafka Source Connector in DataStax Astra Streaming.
Adding a Kafka Source Connector in Datastax Astra Streaming
To create a Kafka Source Connector, visit the Sources tab in Astra Streaming and choose Create Source. This step opens the Create New Source page with a list of fields required to build your new Kafka Source Connector.
Advanced Settings
In your Advanced Settings, control the number of instances, auto-acknowledgement, and timeout. Processing Guarantee offers three options:
-
ATLEAST_ONCE: Each message sent to the source can be processed more than once.
-
ATMOST_ONCE: The message sent to the source is processed at most once. Therefore, there is a chance that the message is not processed.
-
EFFECTIVELY_ONCE: Each message sent to the source will have one output associated with it.
Source-Specific Configuration
Group-ID is a unique string that identifies the consumer group that this consumer belongs to.
The bootstrap server host/port pairs must be in this format: hostname1:port1,hostname2:port2,…
The Kafka topic defined in the source-specific configuration will receive messages from Pulsar:
Consumer Configuration Properties
Currently, only SASL_SSL is supported.
Your username and password are required.
Review and Submit
Confirm your values are correct and select "Create". Confirm you’ve created your new Kafka Source Connector in the Sources tab.
Controlling Your Source
You can start, stop, and restart your source by selecting it in the Sources dashboard.
Monitoring Your Source
Sources produce logs to help you in debugging. To view your source’s logs, open your source in the Sources dashboard.
In the upper right corner of the source log are controls to Refresh, Copy to Clipboard, and Save your log.
Updating Your Source
To update your source, select your source in the Sources dashboard.
-
Update your source’s settings. When you’re done, click Update.
-
An Updates Submitted Successfully flag will appear to let you know your source has been updated.
Deleting Your Source
To delete a source, select the source to be deleted in the Sources dashboard.
-
Click Delete.
-
A popup will ask you to confirm deletion by entering the source’s name and clicking Delete.
-
A Source-name Deleted Successfully! flag will appear to let you know you’ve deleted your source.
Next
For more information on configuring your Kafka source connector, visit Kafka Source Configuration.