Luna Streaming Authentication

The Helm chart can enable token-based authentication for your Pulsar cluster. For more, see Pulsar token authentication.

For authentication to work, the token-generation keys need to be stored in Kubernetes secrets along with superuser default tokens.

The Helm chart includes tooling to automatically create the necessary secrets, or you can do this manually.

Automatically generating secrets for Pulsar token authentication

Use the following settings in your values.yaml file to enable automatic generation of the secrets and enable token-based authentication:

enableTokenAuth: yes
autoRecovery:
  enableProvisionContainer: yes

When enableProvisionContainer is enabled, Pulsar will check if the required secrets exist. If they don’t exist, it will generate new token keys and use those keys to generate the default set of tokens.

The name of the key secrets are:

  • token-private-key

  • token-public-key

Using these keys will generate tokens for each role listed in superUserRoles in the values.yaml file. Based on the default settings, the following secrets will be created to store the tokens:

  • token-superuser

  • token-admin

  • token-proxy

  • token-websocket

Manually generating secrets for Pulsar token authentication

A number of values need to be stored in secrets prior to enabling token-based authentication.

  1. Generate a key-pair for signing the tokens using the Pulsar tokens command:

    bin/pulsar tokens create-key-pair --output-private-key my-private.key --output-public-key my-public.key

    The names of the files used in this section match the default values in the Helm chart’s values.yaml file. If you used different names, then you will have to update the corresponding values.

  2. Store those keys as secrets.

    kubectl create secret generic token-private-key \
 --from-file=my-private.key \
 --namespace pulsar
    kubectl create secret generic token-public-key \
 --from-file=my-public.key \
 --namespace pulsar

  3. Using those keys, generate tokens for the following subjects (roles):

    • admin

    • superuser

    • proxy

    • websocket (only required if using the standalone WebSocket proxy)

      bin/pulsar tokens create --private-key file:///pulsar/token-private-key/my-private.key --subject <subject>

  4. Add each token as a secret:

    kubectl create secret generic token-<subject> \
 --from-file=<subject>.jwt \
 --namespace pulsar

  5. Enable token-based authentication with this setting in values.yaml:

    enableTokenAuth: yes

TLS

Automatically generating certificates using cert-manager

To automatically generate certificates using cert-manager, see Cert-Manager Documentation.

Manually configuring certificate secrets for TLS

To use TLS, you must first create a certificate and store it in the secret defined by tlsSecretName.

Create the certificate:

kubectl create secret tls <tlsSecretName> --key <keyFile> --cert <certFile>

The resulting secret will be of type kubernetes.io/tls. The key should not be in PKCS 8 format, even though that is the format used by Pulsar. The kubernetes.io/tls format will be converted by the chart to PKCS 8.

If you have a self-signed certificate, manually specify the certificate information directly in values:

# secrets:
  # key: |
  # certificate: |
  # caCertificate: |

Once you have created the secrets that store the certificate info (or manually specified it in values), enable TLS in the values:

enableTls: yes

Token Authentication via Keycloak Integration

DataStax created the Pulsar OpenID Connect Authentication Plugin to provide a more dynamic authentication option for Pulsar. This plugin integrates with any OpenID Connect-compliant identity provider to dynamically retrieve public keys for token validation. This dynamic public key retrieval enables support for key rotation and multiple authentication/identity providers by configuring multiple allowed token issuers. It also means that token secret keys will not be stored in Kubernetes secrets.

In order to simplify deployment for Pulsar cluster components, the plugin provides the option to use Keycloak in conjunction with Pulsar’s basic token based authentication. For more, see Pulsar OpenID Connect Authentication Plugin.

See the example Keycloak Helm chart for deploying a working cluster that integrates with Keycloak. By default, the Helm chart creates a Pulsar realm within Keycloak and sets up the client used by the Pulsar Admin Console as well as a sample client and some sample groups. The configuration for the broker side auth plugin should be placed in the .Values.<component>.configData maps.

Configuring Keycloak for Token Generation

  1. Deploy the cluster:

    $ helm install test --values ../../examples/dev-values-keycloak-auth.yaml

  2. Ensure your deployment name matches the working cluster’s name. The name of the deployment is very important for a working cluster. The values file assumes that the cluster’s name is test.

  3. Once the cluster is operational, port forward to Keycloak:

    $ kubectl port-forward test-keycloak-0 8080:8080

  4. Retrieve the configured username and password for the admin user in Keycloak. The dev-values-keycloak-auth.yaml file configures them here:

    keycloak:
  auth:
    adminUser: "admin"
    adminPassword: "F3LVqnxqMmkCQkvyPdJiwXodqQncK@"

  5. Navigate to localhost:8080 in a browser and view the Pulsar realm in the Keycloak UI. Note that the realm name must match the configured realm name (.Values.keycloak.realm) for the OpenID Connect plugin to work properly.

The OpenID Connect plugin uses the sub (subject) claim from the JWT as the role used for authorization within Pulsar. To get Keycloak to generate the JWT for a client with the right sub, create a special "mapper" that is a "Hardcoded claim" mapping claim name sub to a claim value that is the desired role, like superuser. The default config installed by this helm chart provides examples of how to add custom mapper protocols to clients.

Retrieving and using a token from Keycloak with Pulsar Admin CLI

  1. After creating your realm and client, retrieve a token with the Pulsar Admin CLI. To generate a token that will have an allowed issuer, you should exec into a bastion pod in the k8s cluster. Exec’ing into a bastion host will give you immediate access to a pulsar-admin cli tool that you can use to verify that you have access.

    kubectl -n default exec $(kubectl get pods --namespace default -l "app=pulsar,component=bastion" -o jsonpath="{.items[0].metadata.name}") -it -- bash

  2. Run the following from a bastion pod to generate an allowed issuer token.

    pulsar@pulsar-bastion-85c9b777f6-gt9ct:/pulsar$ curl -d "client_id=test-client" \
       -d "client_secret=19d9f4a2-65fb-4695-873c-d0c1d6bdadad" \
       -d "grant_type=client_credentials" \
       "http://test-keycloak/auth/realms/pulsar/protocol/openid-connect/token"
{"access_token":"eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJDY3c3ZXcwQ0hKMThfbWpCQzYxb2xOSU1wT0d3TkEyd1ZFbHBZLUdzb2tvIn0.eyJleHAiOjE2MjY5NzUwNzIsImlhdCI6MTYyNjk3NDQ3MiwianRpIjoiYTExZmFkY2YtYTJkZi00NmNkLTk0OWEtNDdkNzdmNDYxMDMxIiwiaXNzIjoiaHR0cDovL3Rlc3Qta2V5Y2xvYWsvYXV0aC9yZWFsbXMvcHVsc2FyIiwiYXVkIjoiYWNjb3VudCIsInN1YiI6ImQwN2UxOGIxLTE4YzQtNDZhMC1hNGU0LWE3YTZjNmRiMjFkYyIsInR5cCI6IkJlYXJlciIsImF6cCI6InRlc3QtY2xpZW50IiwiYWNyIjoiMSIsInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJvZmZsaW5lX2FjY2VzcyIsImRlZmF1bHQtcm9sZXMtcHVsc2FyIiwidW1hX2F1dGhvcml6YXRpb24iXX0sInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6ImVtYWlsIHByb2ZpbGUiLCJzdWIiOiJzdXBlcnVzZXIiLCJjbGllbnRIb3N0IjoiMTcyLjE3LjAuMSIsImNsaWVudElkIjoidGVzdC1jbGllbnQiLCJlbWFpbF92ZXJpZmllZCI6ZmFsc2UsInByZWZlcnJlZF91c2VybmFtZSI6InNlcnZpY2UtYWNjb3VudC10ZXN0LWNsaWVudCIsImNsaWVudEFkZHJlc3MiOiIxNzIuMTcuMC4xIn0.FckQLOD64ZTKmx2uutP75QBpZAqHaqWyEE6jRUXvbSzsiXTAQyz-30zKsUSEjOMJp97NlTy3NZECVo_GdZ7oPcneFdglmFY62btWj-5s6ELcazj-AGQhyt0muGD4VP71xjpjCUpVxhyBIQlltGZLu7Rgw4trfh3LS8YjaY74vGg_BjOzZ8VI4S352lyGOULou7_dRbaeKhv43OfU7e_Y_ro_m_9UaDARypcj3uqSllhZdifA4YbHyaBCCu5eH19GCLtFm3I00PvWkOy3iTyOkkTcayqJ-Vlraf95qCZFN-sooIIU6o8L-wS-Zr7EvkoDJ-II9q49WHJJLIIvnCE2ug","expires_in":600,"refresh_expires_in":0,"token_type":"Bearer","not-before-policy":0,"scope":"email profile"}

  3. Copy the access_token contents and use it here:

    pulsar@pulsar-bastion-85c9b777f6-gt9ct:/pulsar$ bin/pulsar-admin --auth-params "token:eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJDY3c3ZXcwQ0hKMThfbWpCQzYxb2xOSU1wT0d3TkEyd1ZFbHBZLUdzb2tvIn0.eyJleHAiOjE2MjY5NzUwNzIsImlhdCI6MTYyNjk3NDQ3MiwianRpIjoiYTExZmFkY2YtYTJkZi00NmNkLTk0OWEtNDdkNzdmNDYxMDMxIiwiaXNzIjoiaHR0cDovL3Rlc3Qta2V5Y2xvYWsvYXV0aC9yZWFsbXMvcHVsc2FyIiwiYXVkIjoiYWNjb3VudCIsInN1YiI6ImQwN2UxOGIxLTE4YzQtNDZhMC1hNGU0LWE3YTZjNmRiMjFkYyIsInR5cCI6IkJlYXJlciIsImF6cCI6InRlc3QtY2xpZW50IiwiYWNyIjoiMSIsInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJvZmZsaW5lX2FjY2VzcyIsImRlZmF1bHQtcm9sZXMtcHVsc2FyIiwidW1hX2F1dGhvcml6YXRpb24iXX0sInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6ImVtYWlsIHByb2ZpbGUiLCJzdWIiOiJzdXBlcnVzZXIiLCJjbGllbnRIb3N0IjoiMTcyLjE3LjAuMSIsImNsaWVudElkIjoidGVzdC1jbGllbnQiLCJlbWFpbF92ZXJpZmllZCI6ZmFsc2UsInByZWZlcnJlZF91c2VybmFtZSI6InNlcnZpY2UtYWNjb3VudC10ZXN0LWNsaWVudCIsImNsaWVudEFkZHJlc3MiOiIxNzIuMTcuMC4xIn0.FckQLOD64ZTKmx2uutP75QBpZAqHaqWyEE6jRUXvbSzsiXTAQyz-30zKsUSEjOMJp97NlTy3NZECVo_GdZ7oPcneFdglmFY62btWj-5s6ELcazj-AGQhyt0muGD4VP71xjpjCUpVxhyBIQlltGZLu7Rgw4trfh3LS8YjaY74vGg_BjOzZ8VI4S352lyGOULou7_dRbaeKhv43OfU7e_Y_ro_m_9UaDARypcj3uqSllhZdifA4YbHyaBCCu5eH19GCLtFm3I00PvWkOy3iTyOkkTcayqJ-Vlraf95qCZFN-sooIIU6o8L-wS-Zr7EvkoDJ-II9q49WHJJLIIvnCE2ug" \
      tenants list
"public"
"pulsar"

You’re now using Keycloak tokens with pulsar-admin CLI.

Alternative method

An alternative method for retrieving and using a Keycloak token from the bastion pod is to create a file with the Keycloak client credentials first.

  1. Retrieve the client credentials from Keycloak as above.

  2. Create a creds.json file and enter your retrieved credentials in this format:

    {
 "client_id": "pulsar-admin-example-client",
 "client_secret": "<client-secret-key>",
 "grant_type": "client_credentials"
}

  3. In the bastion pod, issue the command to use the Keycloak token:

    pulsar@pulsar-broker-79b87f786d-tjvm7:/pulsar$ bin/pulsar-admin \
--auth-plugin "org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2"
--auth-params '{"privateKey":"file:///pulsar/creds.json","issuerUrl":"http://test-keycloak:8081/auth/realms/pulsar","audience":"I dont matter"}'
--tenants list
public
pulsar

You’re now using Keycloak tokens with pulsar-admin CLI.

What’s next?

To connect with the Pulsar Admin console and start sending and consuming messages, see Admin Console.

Was this helpful?

Edit this page

Give Feedback

How can we improve the documentation?

© 2024 DataStax | Privacy policy | Terms of use

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: +1 (650) 389-6000, info@datastax.com