Starlight for Kafka security
This page covers enabling authentication features for Starlight for Kafka.
Starlight for Kafka supports SASL/PLAIN authentication, JWT authorization, and SSL encryption.
Starlight for Kafka is installed with no authentication, authorization, or encryption, but these can be enabled to improve security.
Currently, these features are only available in Java.
Authentication
Starlight for Kafka authentication uses the Kafka SASL/PLAIN mechanisms and Pulsar token-based mechanisms.
Enabling authentication for Starlight for Kafka requires enabling authentication for the following components:
-
Pulsar brokers
-
Starlight for Kafka (some configurations of S4K rely on the configurations of Pulsar brokers)
-
Kafka clients
Currently, Starlight for Kafka only supports the PLAIN
SASL mechanism, but there are more to come.
Enable PLAIN
authentication
To enable authentication with the PLAIN
SASL mechanism, follow the steps below.
-
Enable authentication on the Pulsar broker. For the
PLAIN
mechanism, Kafka authentication is forwarded to the JWT authentication of Pulsar, so you need to configure the JWT authentication by setting the following properties inconf/broker.conf
.Modify
conf/standalone.conf
if you’re using a standalone Pulsar deployment.-
Enable authentication for the Pulsar broker.
authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken
-
Enable authentication between the Pulsar broker and Starlight for Kafka.
brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationToken brokerClientAuthenticationParameters=token:<token-of-super-user-role> superUserRoles=<super-user-roles>
-
Specify the key. For more information, see Enable Token Authentication on Brokers.
-
If using a secret key, set the property as below.
tokenSecretKey=file:///path/to/secret.key
-
To pass the key inline, set the property as below.
tokenSecretKey=data:;base64,FLFyW0oLJ2Fi22KKCm21J18mbAdztfSHN/lAT5ucEKU=
-
If using a public/private key, set the property as below.
tokenPublicKey=file:///path/to/public.key
-
-
-
Enable authentication on Starlight for Kafka. Set the following property in the
conf/broker.conf
:saslAllowedMechanisms=PLAIN
-
Enable authentication on the Kafka client. To forward your credentials,
SASL-PLAIN
is used on the Kafka client side. To enableSASL-PLAIN
, set the following properties through Kafka JAAS.
Property | Description | Example Value |
---|---|---|
|
|
The |
|
|
|
security.protocol=SASL_PLAINTEXT
# or security.protocol=SASL_SSL if SSL connection is used
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule \
required username="public/default" password="token:xxx";
For more on passing credentials with JAAS, see Configuring SASL/PLAIN.
Authorization
To enable authorization on Starlight for Kafka, ensure that authentication is enabled first.
For more on authorization in Pulsar, see Pulsar Authorization.
-
Enable authorization and assign superusers for the Pulsar broker.
authorizationEnabled=true
-
Generate JWT tokens. A token is the credential associated with a user. The association is done through the
principal
orrole
. In the case of JWT tokens, this field is typically referred assubject
, though they are exactly the same concept.Use this command to require the generated token to have a
subject
field set:bin/pulsar tokens create --secret-key file:///path/to/secret.key \ --subject <user-role>
This command will print the token string on
stdout
. -
Grant permission to a specific role. The token itself does not have any permissions associated with it. The authorization engine determines whether the token should have permissions or not. Once you have created the token, you can grant permission for this token.
For example, to grant a role to theuser-role
created above:bin/pulsar-admin --auth-plugin "org.apache.pulsar.client.impl.auth.AuthenticationToken" --auth-params "token:<token-of-super-user-role>" \ namespaces grant-permission <tenant>/<namespace> \ --role <user-role> \ --actions produce,consume
SSL encryption
This section explains connecting and configuring Starlight for Kafka with SSL. There are two methods available:
-
(Recommended) Use the TLS certificates already configured for the Pulsar TLS endpoint to also connect the Kafka client. This is a new feature of Starlight for Kafka designed to make TLS connection fast and painless. Proceed to Create SSL connection with Pulsar.
-
Create dedicated TLS certificates and modify the Pulsar broker and Kafka client. Instructions for this method are here.
Create SSL configuration with Pulsar
Starlight for Kafka features fast, painless TLS configuration between Pulsar and Kafka. Instead of creating different certificates for the Pulsar broker and Kafka client, simply modify broker.conf
to pick up the existing Pulsar TLS certifications and configurations.
-
Configure TLS on the Pulsar broker. For more, see Create TLS certificates.
-
Expose TLS endpoints by adding the following configurations to
broker.conf
:kopTlsEnabledWithBroker=true kafkaListeners=PLAINTEXT://127.0.0.1:9092, SSL://PRIVATE-HOSTNAME-OF-THE-BROKER:9093 kafkaAdvertisedListeners=PLAINTEXT://127.0.0.1:9092, SSL://PUBLIC-HOSTNAME-OF-THE-BROKER:9093
These settings will enable TLS on the broker, and Starlight for Kafka will pick up the TLS certificates automatically.
-
Proceed to Test Starlight for Kafka to test your configuration.
Create SSL configuration manually
The following example shows how to manually configure Starlight for Kafka with SSL.
Starlight for Kafka supports PLAINTEXT
and SSL
configuration types for Kafka listeners. SSL listeners are added to the comma-separated list of URIs in kafkaListeners
, as below.
kafkalisteners=PLAINTEXT://localhost:9092,SSL://localhost:9093
For more on generating SSL keys for Kafka brokers, see Kafka SSL.
-
Create SSL related keys. This example creates the related CA and JKS files.
# Input a password, for example "server-keystore". keytool -keystore server.keystore.jks -alias localhost -validity 365 -keyalg RSA -genkey # Input a password, for example "server". openssl req -new -x509 -keyout ca-key -out ca-cert -days 365 # Input a password, for example "server-truststore" keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert # Input a password, for example "client-truststore" keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert # Input the password of server.keystore.jks: "server-keystore" keytool -keystore server.keystore.jks -alias localhost -certreq -file cert-file # The password followed by `-passin pass:` is the password of ca-cert: "server" openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:server # You must input the password of server.keystore.jks: "server-keystore" keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert # You must input the password of server.keystore.jks: "server-keystore" keytool -keystore server.keystore.jks -alias localhost -import -file cert-signed
In the above example, we have input four passwords:
-
server-keystore
forserver.keystore.jks
-
server
forca-cert
andca-key
-
server-truststore
forserver.truststore.jks
-
client-truststore
forclient.truststore.jks
-
-
Configure the Pulsar broker. In
conf/broker.conf
, add the related configurations using the JKS configurations created in the previous step.listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093 # You need to use the full path of server.keystore.jks kopSslKeystoreLocation=server.keystore.jks kopSslKeystorePassword=server-keystore kopSslKeyPassword=server-keystore # You need to use the full path of server.truststore.jks kopSslTruststoreLocation=server.truststore.jks kopSslTruststorePassword=server-truststore
-
Configure the Kafka client. Create a file named
client-ssl.properties
inkafka/config
with the following configuration:security.protocol=SSL # Include the full path of client.truststore.jks ssl.truststore.location=client.truststore.jks ssl.truststore.password=client-truststore # The identification algorithm must be empty ssl.endpoint.identification.algorithm=
-
Verify the console-producer and the console-consumer send messages when started with
client-ssl.properties
:
kafka-console-producer.sh --broker-list localhost:9093 --topic test --producer.config client-ssl.properties
kafka-console-consumer.sh --bootstrap-server localhost:9093 --topic test --consumer.config client-ssl.properties
For more on configuring Kafka, see Configure Kafka client.
What’s next?
For more on Starlight for Kafka, see: