Kafka Sink Connector
The Kafka Sink Connector is used to pull messages from Pulsar topics and persist the messages to a Kafka topic.
This guide explains how to configure and use the Kafka Sink Connector in DataStax Astra Streaming.
Adding a Kafka Sink Connector in Astra Streaming
To create a Kafka Sink Connector, visit the Sinks tab in Astra Streaming and choose "Create Sink". This step opens the Create New Sink page with a list of fields required to build your new Kafka Sink Connector.
Advanced Settings
In your Advanced Settings, control the number of instances, auto-acknowledgement, and timeout. Processing Guarantee offers three options:
-
ATLEAST_ONCE: Each message sent to the sink can be processed more than once.
-
ATMOST_ONCE: The message sent to the sink is processed at most once. Therefore, there is a chance that the message is not processed.
-
EFFECTIVELY_ONCE: Each message sent to the sink will have one output associated with it.
Sink-Specific Configuration
The bootstrap server host/port pairs must be in the format hostname1:port1,hostname2:port2,…
The Kafka topic here will receive messages from Pulsar:
Acks is the number of acknowledgements that the producer requires before a request is complete.
Batch Size is the batch size that a Kafka producer attempts to batch records together before sending them to brokers.
Max Request Size is the maximum size of a Kafka request in bytes.
Producer Configuration Properties
Currently, only SASL_SSL is supported.
Your username and password are required.
Review and Submit
Confirm your values are correct and select Create. Confirm you’ve created your new Kafka Sink Connector in the Sinks tab.
Controlling Your Sink
You can start, stop, and restart your sink by selecting it in the Sinks dashboard.
Monitoring Your Sink
Sinks produce logs to help you in debugging. To view your sink’s logs, open your sink in the Sinks dashboard.
In the upper right corner of the sink log are controls to Refresh, Copy to Clipboard, and Save your log.
Updating Your Sink
To update your sink, select your sink in the Sinks dashboard.
-
Update your sink’s settings. When you’re done, click Update.
-
An Updates Submitted Successfully flag will appear to let you know your sink has been updated.
Deleting Your Sink
To delete a sink, select the sink to be deleted in the Sinks dashboard.
-
Click Delete.
-
A popup will ask you to confirm deletion by entering the sink’s name and clicking Delete.
-
A Sink-name Deleted Successfully! flag will appear to let you know you’ve deleted your sink.
Next
For more information on configuring the Kafka Sink Connector, visit Kafka Sink Configuration.