Connect the DataStax Apache Pulsar™ connector

To connect the DataStax Apache Pulsar™ connector to your database cluster, you must configure authentication, encryption, and connection parameters.

Set connection parameters in the configuration file

Use the connection parameters to configure the connection settings from Apache Pulsar to your database cluster.

 connectionPoolLocalSize: 4
 contactPoints: [dse_host_list]
 loadBalancing.localDc: datacenter_name
 port: 9042
 maxConcurrentRequests: 500
 maxNumberOfRecordsInBatch: 32
 queryExecutionTimeout: 30
 jmx: true
 compression: None

connectionPoolLocalSize: Number of connections that driver maintains within a connection pool to each node in the local datacenter.

Default: 4
contactPoints: Either localhost or a comma-separated list of host names or IP addresses in square brackets. If you provide a list of hosts, you must also set loadBalancing.localDc.

Default: localhost
loadBalancing.localDc: The case-sensitive datacenter name for the driver to use for load balancing.

When connecting to Astra with cloud.secureConnectBundle, leave this property unset.

Default: ""

port: DSE native transport port. See native_transport_port.

Default: 9042
maxConcurrentRequests: Maximum number of requests to send to DSE at the same time.

Default: 500
maxNumberOfRecordsInBatch: Number of records to include in a write request to the database table.

Default: 32
jmx: Whether to enable metrics reporting using Java Management Extensions (JMX).

Default: true
compression: Compression algorithm to use when issuing requests to the database server. Valid values are None, Snappy, and LZ4, which is applied per connector instance.

Default: None
queryExecutionTimeout: CQL statement execution timeout, in seconds.

Default: 30

Pass connection parameters directly to the Java driver

In your DataStax Pulsar connector configuration file, you can pass some connection parameters directly to the embedded Cassandra Java driver by using the datastax-java-driver prefix. For example:

datastax-java-driver.basic.request.consistency=ALL

Available Java driver parameters

If you define both base and java-driver parameters in your configuration file, the base parameters take precedence over their equivalent java-driver parameters. If you don’t specify either, then the default value for the base Pulsar connector parameter is used.

The following list maps DataStax Pulsar connector parameters to their functionally equivalent Java driver parameters:

contactPoints: datastax-java-driver.basic.contact-points

contactPoints automatically appends the port to every host name or IP provided by this setting.

If you use datastax-java-driver.basic.contact-points, you must provide the fully qualified contact points (host:port). However, this makes datastax-java-driver.basic.contact-points more flexible because you can specify a different port for each host. For example: datastax-java-driver.basic.contact-points = 127.0.0.1:9042, 127.0.0.2:90
loadBalancing.localDc: datastax-java-driver.basic.load-balancing-policy.local-datacenter
cloud.secureConnectBundle: datastax-java-driver.basic.cloud.secure-connect-bundle
queryExecutionTimeout: datastax-java-driver.basic.request.timeout
connectionPoolLocalSize: datastax-java-driver.advanced.connection.pool.local.size
compression: datastax-java-driver.advanced.protocol.compression
metricsHighestLatency: datastax-java-driver.advanced.metrics.session.cql-requests.highest-latency

You can also specify these additional Java driver parameters in your configuration file:

datastax-java-driver.advanced.ssl-engine-factory.cipher-suites
datastax-java-driver.advanced.metrics.node.enabled
datastax-java-driver.advanced.metadata.schema.refreshed-keyspaces
datastax-java-driver.advanced.metrics.session.enabled

Conversion of type List to TypeSafe Config

Properties that are of type List are converted by the Java driver to the TypeSafe Config format. This includes the following properties:

datastax-java-driver.advanced.ssl-engine-factory.cipher-suites
datastax-java-driver.advanced.metrics.node.enabled
datastax-java-driver.advanced.metadata.schema.refreshed-keyspaces
datastax-java-driver.advanced.metrics.session.enabled
datastax-java-driver.basic.contact-points

The conversion splits the comma-separated values provided in those setting to create indexed properties. For example:

# Original comma-separated values
datastax-java-driver.advanced.metrics.session.enabled=a,b

# Converted indexed properties
datastax-java-driver.advanced.metrics.session.enabled.0=a
datastax-java-driver.advanced.metrics.session.enabled.1=b

Authentication and encryption

You can authenticate the DataStax Pulsar connector session using internal, LDAP, Kerberos, or application token authentication, depending on the methods supported by your cluster. The connector also supports SSL-encrypted connections.

DSE
Astra

When DSE Unified Authentication is enabled, you must provide internal/LDAP authentication credentials or the Kerberos keytab filepath in the authentication parameters. The connector supports DSE role management with internal and LDAP schemes.

If the cluster has client encryption enabled, configure the SSL keys and certificates in the Pulsar connector. SSL encryption settings are configured in the DSE client-to-node encryption options.

For information about authentication configuration for DSE clusters, see the documentation for your version of DSE.

To connect to Astra, provide the application token credentials in the authentication parameters, and use the Secure Connect Bundle (SCB) to establish an SSL-encrypted connection.

Authentication parameters

If the cluster has internal or LDAP authentication enabled, configure the username and password settings for the Pulsar connector. The authenticated role must have read/write access to the tables that will receive data from the Pulsar connector.

In your connector’s configuration YAML file, set the relevant auth parameters for auth.provider, auth.username, and auth.password:

auth:
  provider: DSE
  username: login_role
  password: password

provider

Select the type of authentication provider configured for the cluster:

None (default): No authentication
DSE: Internal or LDAP authentication
GSSAPI (Kerberos): SASL authentication to DSE clusters using the GSSAPI mechanism. The path to the Kerberos configuration file (krb5.conf) must be set in the java.security.krb5.conf system property at startup.

username

For DSE, provide the login role name or LDAP username. If authorization is enabled, this login role must have a minimum of modify privileges on the tables that receive data from the DataStax Pulsar connector.

For Astra, enter the literal string token or provide the clientId from an Astra application token. The application token’s role must have read/write privileges on the tables that receive data from the DataStax Pulsar connector.

password

For DSE, provide the password for the login role or LDAP username.

For Astra, provide one of the following, depending on the value you entered for username:

If username: token, provide a secure reference to your actual Astra application token, which is prefixed by AstraCS:.
If username: <clientId>, provide the corresponding clientSecret from the application token.

SSL-encrypted connections

DSE
Astra

For DSE clusters with client-to-node encryption enabled, configure the ssl: properties for the Pulsar connector:

ssl:
  provider:
  cipherSuites:
  hostnameValidation: true
  keystore:
    password:
    path:
  truststore:
    password:
    path:
  openssl:
    keyCertChain:
    privateKey:

provider

SSL provider to use:

None (default)
JDK
OpenSSL

cipherSuites

Cipher suites to enable

Default: unset (no cipher suites)

hostnameValidation

Whether to validate node hostnames when using SSL

Default: true (enabled)

keystore

password: Keystore password
path: Path to the keystore file

truststore

password: Truststore password
path: Path to the truststore file

openssl

Only applicable when using OpenSSL

keyCertChain: Path to the SSL certificate file
privateKey: Path to the private key file

In your Pulsar configuration, you can provide paths to the keystore, truststore, private key, and certificate files, or you can provide the files as base64-encoded strings:

Encode the file:
```
base64 -i trust-store-key
```
Set the resulting Base64-encoded string in the appropriate parameter:

  path: base64:UEsDBBQACAAIADmJJ...

The Pulsar function framework handles deployment to additional Pulsar machines.

When connecting to Astra, use your database’s Secure Connect Bundle (SCB) zip file to establish an SSL-encrypted connection. The SCB also sets the local datacenter for the connection.

Unencrypted string
Base64-encoded string

Provide the full path to the SCB zip file directly in cloud.secureConnectBundle:

cloud.secureConnectBundle=/path/to/sbc.zip

You can provide the literal path to your SCB or encode the zip file using the standard base64 tool:

base64 -i scb.zip

Then, set the resulting base64-encoded string directly in the Pulsar configuration file:

cloud.secureConnectBundle=base64:UEsDBBQACAAI....