Kafka Sink Connector

The Kafka Sink Connector is used to pull messages from Pulsar topics and persist the messages to a Kafka topic.

This guide explains how to configure and use the Kafka Sink Connector in Datastax Astra Streaming.

Adding a Kafka Sink Connector in Astra Streaming

To create a Kafka Sink Connector, visit the Sinks tab in Astra Streaming and choose "Create Sink". This step opens the Create New Sink page with a list of fields required to build your new Kafka Sink Connector.

Basics

Advanced Settings

In your Advanced Settings, control the number of instances, auto-acknowledgement, and timeout. Processing Guarantee offers three options:

  • ATLEAST_ONCE: Each message sent to the sink can be processed more than once.

  • ATMOST_ONCE: The message sent to the sink is processed at most once. Therefore, there is a chance that the message is not processed.

  • EFFECTIVELY_ONCE: Each message sent to the sink will have one output associated with it.

Advanced

Connect Topics

Choose the topic messages will be published to from the connected sink:

Connect Topics

Sink-Specific Configuration

The bootstrap server host/port pairs must be in the format hostname1:port1,hostname2:port2,…​

The Kafka topic here will receive messages from Pulsar:

Sink Specific Configuration

Acks is the number of acknowledgements that the producer requires before a request is complete.

Batch Size is the batch size that a Kafka producer attempts to batch records together before sending them to brokers.

Max Request Size is the maximum size of a Kafka request in bytes.

Producer Configuration Properties

Currently, only SASL_SSL is supported. Your username and password are required.

Consumer Configuration Properties

Review and Submit

Confirm your values are correct and select Create. Confirm you’ve created your new Kafka Sink Connector in the Sinks tab.

Controlling Your Sink

You can start, stop, and restart your sink by selecting it in the Sinks dashboard.

Sink Controls

Monitoring Your Sink

Sinks produce logs to help you in debugging. To view your sink’s logs, open your sink in the Sinks dashboard.

In the upper right corner of the sink log are controls to Refresh, Copy to Clipboard, and Save your log.

Updating Your Sink

To update your sink, select your sink in the Sinks dashboard.

  1. Update your sink’s settings. When you’re done, click Update.

  2. An Updates Submitted Successfully flag will appear to let you know your sink has been updated.

Deleting Your Sink

To delete a sink, select the sink to be deleted in the Sinks dashboard.

  1. Click Delete.

  2. A popup will ask you to confirm deletion by entering the sink’s name and clicking Delete.

  3. A Sink-name Deleted Successfully! flag will appear to let you know you’ve deleted your sink.

Next

For more information on configuring the Kafka Sink Connector, visit Kafka Sink Configuration.