Installing the DataStax Kafka Connector 1.0

Install on Linux-based platform using a binary tarball.

Install the connector from the DataStax distribution binary tarball using an account that has write access to the Kafka configuration directory.

dse-sink-standalone.properties.sample

The dse-sink-standalone.properties.sample file is located in the conf directory of the DataStax Apache Kafka Connector distribution package.

dse-sink-distributed.json.sample

  • The dse-sink-distributed.json.sample file is located in the conf directory of the DataStax Apache Kafka Connector distribution package.

Support Apache Kafka versions

Install the DataStax Apache Kafka™ Connector on any of following versions:
  • Confluent 3.2 and higher
  • Apache Kafka 0.10.2 and higher

Stream data using the connector to DataStax Enterprise (DSE) 5.0 and higher.

Prerequisites

Sign up for an account on DataStax Academy and configure a Download Key.

Procedure

Perform the following steps on a Kafka Connect node

  1. Using your DataStax Academy credentials, download the tarball from Download Kafka Connector.
  2. Extract the files:
    tar zxf kafka-connect-dse-version_number.tar.gz
    where version_number is the connector version number.
    The following files are unpacked into a directory such as kafka-connect-dse-version_number.
    LICENSE.txt
    README.md
    THIRD-PARTY.txt
    conf/dse-sink-distributed.json.sample
    conf/dse-sink-standalone.properties.sample
    kafka-connect-dse-version_number.jar
    where version_number is the DSE connector version number.
  3. Configured the DataStax connector JAR using one of the following methods:
    • Move the DataStax connector JAR to Kafka plugins directory:
      mv installation_location/kafka-connect-dse-version_number.jar kafka_plugins_dir
    • Configure the path to the JAR:
      • Apache Kafka 0.11.x and higher - Specify the JAR location in the plugin.path parameter in the connect-standalone.properties or connect-distributed.properties file that is passed to the worker start-up scripts. For example:
        plugin.path=install_location/kafka-connect-dse-1.0.0-alpha2.jar
      • Apache Kafka versions earlier than 0.11 - To properly load the classes, add the JAR path to the worker CLASSPATH. For example:
        export CLASSPATH=$CLASSPATH:install_location/kafka-connect-dse-1.0.0-alpha2.jar 
        Note: For Confluent, reference the compatibility table.
  4. Copy the sample configuration file from kafka-connect-dse-version_number/conf/ to the Kafka configuration directory, which is typically the config or etc directory.
    DataStax provides the following sample files in the conf directory of the connector distribution package:
    • dse-sink-standalone.properties.sample for standalone mode. It is a Java properties file that contains all settings with descriptions. Settings with a default value are commented out.
    • dse-sink-distributed.json.sample for distributed mode. This file is in JSON format and contains all settings, which are enumerated and active. To use the default values, remove settings from the configuration file. JSON does not support comments.
  5. Rename the sample file to dse-sink.properties or dse-sink.json.
  6. Update the settings as necessary. See the DataStax Apache Kafka Connector configuration parameter reference.
  7. Ensure that the user running Kafka has permission to access the configuration and JAR files.