Kafka Sink Connector
The Kafka Sink Connector is used to pull messages from Pulsar topics and persist the messages to a Kafka topic.
This guide explains how to configure and use the Kafka Sink Connector in DataStax Astra Streaming.
To create a Kafka Sink Connector, visit the Sinks tab in Astra Streaming and choose "Create Sink". This step opens the Create New Sink page with a list of fields required to build your new Kafka Sink Connector.
In your Advanced Settings, control the number of instances, auto-acknowledgement, and timeout. Processing Guarantee offers three options:
ATLEAST_ONCE: Each message sent to the sink can be processed more than once.
ATMOST_ONCE: The message sent to the sink is processed at most once. Therefore, there is a chance that the message is not processed.
EFFECTIVELY_ONCE: Each message sent to the sink will have one output associated with it.
Choose the topic messages will be published to from the connected sink:
The bootstrap server
port pairs must be in the format
The Kafka topic here will receive messages from Pulsar:
Acks is the number of acknowledgements that the producer requires before a request is complete.
Batch Size is the batch size that a Kafka producer attempts to batch records together before sending them to brokers.
Max Request Size is the maximum size of a Kafka request in bytes.
SASL_SSL is supported.
password are required.
Confirm your values are correct and select Create. Confirm you’ve created your new Kafka Sink Connector in the Sinks tab.
You can start, stop, and restart your sink by selecting it in the Sinks dashboard.
Sinks produce logs to help you in debugging. To view your sink’s logs, open your sink in the Sinks dashboard.
In the upper right corner of the sink log are controls to Refresh, Copy to Clipboard, and Save your log.
To update your sink, select your sink in the Sinks dashboard.
Update your sink’s settings. When you’re done, click Update.
An Updates Submitted Successfully flag will appear to let you know your sink has been updated.
To delete a sink, select the sink to be deleted in the Sinks dashboard.
A popup will ask you to confirm deletion by entering the sink’s name and clicking Delete.
A Sink-name Deleted Successfully! flag will appear to let you know you’ve deleted your sink.