Starlight for Kafka security

This page covers enabling authentication features for Starlight for Kafka.

Starlight for Kafka supports SASL/PLAIN authentication, JWT authorization, and SSL encryption.

Starlight for Kafka is installed with no authentication, authorization, or encryption, but these can be enabled to improve security.

Currently, these features are only available in Java.

Authentication

Starlight for Kafka authentication uses the Kafka SASL/PLAIN mechanisms and Pulsar token-based mechanisms.

Enabling authentication for Starlight for Kafka requires enabling authentication for the following components:

  • Pulsar brokers

  • Starlight for Kafka (some configurations of S4K rely on the configurations of Pulsar brokers)

  • Kafka clients

Currently, Starlight for Kafka only supports the PLAIN SASL mechanism, but there are more to come.

Enable PLAIN authentication

To enable authentication with the PLAIN SASL mechanism, follow the steps below.

  1. Enable authentication on the Pulsar broker. For the PLAIN mechanism, Kafka authentication is forwarded to the JWT authentication of Pulsar, so you need to configure the JWT authentication by setting the following properties in conf/broker.conf.

    Modify conf/standalone.conf if you’re using a standalone Pulsar deployment.

    1. Enable authentication for the Pulsar broker.

      authenticationEnabled=true
      authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken
    2. Enable authentication between the Pulsar broker and Starlight for Kafka.

      brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationToken
      brokerClientAuthenticationParameters=token:<token-of-super-user-role>
      superUserRoles=<super-user-roles>
    3. Specify the key. For more information, see Enable Token Authentication on Brokers.

      • If using a secret key, set the property as below.

        tokenSecretKey=file:///path/to/secret.key
      • To pass the key inline, set the property as below.

        tokenSecretKey=data:;base64,FLFyW0oLJ2Fi22KKCm21J18mbAdztfSHN/lAT5ucEKU=
      • If using a public/private key, set the property as below.

        tokenPublicKey=file:///path/to/public.key
  2. Enable authentication on Starlight for Kafka. Set the following property in the conf/broker.conf:

    saslAllowedMechanisms=PLAIN
  3. Enable authentication on the Kafka client. To forward your credentials, SASL-PLAIN is used on the Kafka client side. To enable SASL-PLAIN, set the following properties through Kafka JAAS.

Property Description Example Value

username

username

The username of Kafka JAAS is tenant/namespace or tenant, where Kafka’s topics are stored in Pulsar.

password

password must be your token authentication parameters from Pulsar.
The token can be created by Pulsar token tools. The role is the subject for the token. It is embedded in the created token and the broker can get role by parsing this token.

token:xxx

security.protocol=SASL_PLAINTEXT
# or security.protocol=SASL_SSL if SSL connection is used
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule \
required username="public/default" password="token:xxx";

For more on passing credentials with JAAS, see Configuring SASL/PLAIN.

Authorization

To enable authorization on Starlight for Kafka, ensure that authentication is enabled first.

For more on authorization in Pulsar, see Pulsar Authorization.

  1. Enable authorization and assign superusers for the Pulsar broker.

    authorizationEnabled=true
  2. Generate JWT tokens. A token is the credential associated with a user. The association is done through the principal or role. In the case of JWT tokens, this field is typically referred as subject, though they are exactly the same concept.

    Use this command to require the generated token to have a subject field set:

    bin/pulsar tokens create --secret-key file:///path/to/secret.key \
     --subject <user-role>

    This command will print the token string on stdout.

  3. Grant permission to a specific role. The token itself does not have any permissions associated with it. The authorization engine determines whether the token should have permissions or not. Once you have created the token, you can grant permission for this token.
    For example, to grant a role to the user-role created above:

    bin/pulsar-admin --auth-plugin "org.apache.pulsar.client.impl.auth.AuthenticationToken" --auth-params "token:<token-of-super-user-role>" \
        namespaces grant-permission <tenant>/<namespace> \
        --role <user-role> \
        --actions produce,consume

SSL encryption

This section explains connecting and configuring Starlight for Kafka with SSL. There are two methods available:

  • (Recommended) Use the TLS certificates already configured for the Pulsar TLS endpoint to also connect the Kafka client. This is a new feature of Starlight for Kafka designed to make TLS connection fast and painless. Proceed to Create SSL connection with Pulsar.

  • Create dedicated TLS certificates and modify the Pulsar broker and Kafka client. Instructions for this method are here.

Create SSL configuration with Pulsar

Starlight for Kafka features fast, painless TLS configuration between Pulsar and Kafka. Instead of creating different certificates for the Pulsar broker and Kafka client, simply modify broker.conf to pick up the existing Pulsar TLS certifications and configurations.

  1. Configure TLS on the Pulsar broker. For more, see Create TLS certificates.

  2. Expose TLS endpoints by adding the following configurations to broker.conf:

    kopTlsEnabledWithBroker=true
    kafkaListeners=PLAINTEXT://127.0.0.1:9092, SSL://PRIVATE-HOSTNAME-OF-THE-BROKER:9093
    kafkaAdvertisedListeners=PLAINTEXT://127.0.0.1:9092, SSL://PUBLIC-HOSTNAME-OF-THE-BROKER:9093

    These settings will enable TLS on the broker, and Starlight for Kafka will pick up the TLS certificates automatically.

  3. Proceed to Test Starlight for Kafka to test your configuration.

Create SSL configuration manually

The following example shows how to manually configure Starlight for Kafka with SSL.

Starlight for Kafka supports PLAINTEXT and SSL configuration types for Kafka listeners. SSL listeners are added to the comma-separated list of URIs in kafkaListeners, as below.

kafkalisteners=PLAINTEXT://localhost:9092,SSL://localhost:9093

For more on generating SSL keys for Kafka brokers, see Kafka SSL.

  1. Create SSL related keys. This example creates the related CA and JKS files.

    # Input a password, for example "server-keystore".
    keytool -keystore server.keystore.jks -alias localhost -validity 365 -keyalg RSA -genkey
    # Input a password, for example "server".
    openssl req -new -x509 -keyout ca-key -out ca-cert -days 365
    # Input a password, for example "server-truststore"
    keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert
    # Input a password, for example "client-truststore"
    keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert
    # Input the password of server.keystore.jks: "server-keystore"
    keytool -keystore server.keystore.jks -alias localhost -certreq -file cert-file
    # The password followed by `-passin pass:` is the password of ca-cert: "server"
    openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:server
    # You must input the password of server.keystore.jks: "server-keystore"
    keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert
    # You must input the password of server.keystore.jks: "server-keystore"
    keytool -keystore server.keystore.jks -alias localhost -import -file cert-signed

    In the above example, we have input four passwords:

    • server-keystore for server.keystore.jks

    • server for ca-cert and ca-key

    • server-truststore for server.truststore.jks

    • client-truststore for client.truststore.jks

  2. Configure the Pulsar broker. In conf/broker.conf, add the related configurations using the JKS configurations created in the previous step.

    listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093
    
    # You need to use the full path of server.keystore.jks
    kopSslKeystoreLocation=server.keystore.jks
    kopSslKeystorePassword=server-keystore
    kopSslKeyPassword=server-keystore
    # You need to use the full path of server.truststore.jks
    kopSslTruststoreLocation=server.truststore.jks
    kopSslTruststorePassword=server-truststore
  3. Configure the Kafka client. Create a file named client-ssl.properties in kafka/config with the following configuration:

    security.protocol=SSL
    # Include the full path of client.truststore.jks
    ssl.truststore.location=client.truststore.jks
    ssl.truststore.password=client-truststore
    # The identification algorithm must be empty
    ssl.endpoint.identification.algorithm=
  4. Verify the console-producer and the console-consumer send messages when started with client-ssl.properties:

kafka-console-producer.sh --broker-list localhost:9093 --topic test --producer.config client-ssl.properties
kafka-console-consumer.sh --bootstrap-server localhost:9093 --topic test --consumer.config client-ssl.properties

For more on configuring Kafka, see Configure Kafka client.

Was this helpful?

Give Feedback

How can we improve the documentation?

© 2024 DataStax | Privacy policy | Terms of use

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: +1 (650) 389-6000, info@datastax.com