Configure OpsCenter to use DSE Kerberos authentication

OpsCenter can use Kerberos to authenticate to DataStax Enterprise (DSE) clusters where Kerberos authentication is enabled.

To configure OpsCenter for Kerberos authentication, you must enable Kerberos authentication on the DSE cluster, create Kerberos principals, CQL roles, and keytab files, and then define the cluster’s Kerberos configuration in OpsCenter.

OpsCenter supports only one Kerberos configuration for each cluster.

For database clusters that use a separate cluster to store OpsCenter metrics and other data, the same Kerberos configuration and principals must be applied to both clusters.

Kerberos principal format

Understanding the format of Kerberos principals is crucial for successfully configuring OpsCenter to use Kerberos authentication.

A Kerberos principal is an identity, such as for a user or service account. Each principal is comprised of a primary, an optional instance, and a realm:

Realm

The last part of the principal, but important for understanding the primary and instance. Each principal must be fully qualified with the name of a realm, which is similar to a domain. The realm is prefixed with @ to separate it from the primary and instance. It is typically written in all capital letters. In the following examples, the realm is EXAMPLE.COM.

Primary

The first part of the principal. Names a specific identity within the realm, such as a user or service. For example user123.

A principal that represents a user typically consists of a primary and a realm, such as user123@EXAMPLE.COM.

If the primary represents a service, the principal can be referred to as a Kerberos service principal, and the instance component specifies the host where the service runs.

Instance

Optional component that defines the hostname or IP address where a primary service runs. The instance is prefixed with / to separate it from the primary. For example:

service456/server.example.com@EXAMPLE.COM: A principal for the primary service456 that runs on the host server.example.com in the realm EXAMPLE.COM.
cassandra/192.168.1.102@EXAMPLE.COM: A principal for the primary cassandra that runs on the host with IP address 192.168.1.102 in the realm EXAMPLE.COM.

Commands to create Kerberos principals

The way you create Kerberos principals depends on the type of Kerberos Key Distribution Center (KDC) server that you use. The following common commands and the examples in this guide are based on MIT Kerberos.

kadmin

Launch the an interactive session of the Kerberos admin shell (kadmin CLI). To create principals, you must start kadmin with an admin user that has add privileges. For example:

kadmin -p user_name/admin

After starting a kadmin session, enter subsequent commands at the kadmin: prompt. The examples for the following commands show the kadmin: prompt for clarity.

addprinc

Create a Kerberos principal:

kadmin: addprinc -randkey service/fqdn@REALM

-randkey: Set the principal’s key to a random value.
service: The service or user for the principal, such as opscenterd or HTTP. This is the primary portion of the Kerberos principal.
fqdn: The Fully Qualified Domain Name of the host where the service runs, such as node1.example.com. This is the instance portion of the Kerberos principal.
REALM: Optional realm portion of the Kerberos principal, such as EXAMPLE.COM. If you don’t specify a realm, kadmin uses your default realm to create the complete principal name, such as opscenterd/node1.example.com@EXAMPLE.COM.

For more information about the principal format, see Kerberos principal format.

ktadd

Add a service principal to a keytab file:

kadmin: ktadd -k /tmp/krb5_opsc.keytab service/fqdn@REALM

In the keytab file created with ktadd, the format of the service principal must match the principal created with addprinc. For example, if the service principal is service/HOST@EXAMPLE.COM, then the principal in the keytab file must use the same format. Like addprinc, if you don’t specify the realm, kadmin uses your default realm.

Enable Kerberos authentication in DSE

Configure Kerberos authentication on the DSE cluster:

Create Kerberos principles

On your Kerberos Key Distribution Center (KDC) host, create principals for the OpsCenter and DataStax Agent services:

Use the kinit command to obtain and cache an initial ticket-granting ticket for the admin user.

In the following example, dse is the name of the DSE cluster.
```
kinit cassandra/admin@dse
```
Enter the password for the admin user when prompted:
```
Password for cassandra/admin@dse:
```
Start a kadmin CLI session, and enter the password for the admin user when prompted:
```
kadmin -p cassandra/admin@dse
```
Make sure the kadmin: prompt appears.
```
kadmin:
```
In the following steps, kadmin: prompt is included for clarity. If you copy these examples, drop the kadmin: prompt, and edit the example values for your environment.
Create service principals for the OpsCenter daemon (opscenterd) and for HTTP communication.

If you are configuring OpsCenter for high availability, you must create principals for the OpsCenter primary instance and backup instances.

The service principal format for the OpsCenter daemon is service/fqdn@REALM:
- service: The service name for the principal. In the following example, the daemon service is named opscenterd, but this can be any valid name.
- fqdn: The Fully Qualified Domain Name of the OpsCenter node, such as node1.example.com.
- REALM: Optional Kerberos realm, such as EXAMPLE.COM. Separated from the FQDN with @. If not specified, kadmin uses your default Kerberos realm.
```
# Create service principal for the OpsCenter daemon
kadmin: addprinc -randkey opscenterd/node1.example.com

# Create service principal for HTTP communication on the OpsCenter daemon node
kadmin: addprinc -randkey HTTP/node1.example.com
```
Use the same pattern as the previous commands to create service principals for DataStax Agent nodes and HTTP communication on agent nodes.

You can create separate principals for each DataStax Agent, or create two principals that are shared by all agents.

In the service principal format (service/fqdn@REALM), use the FQDN for the agent node. The following example, the agent service name is dx-agent, but this can be any valid name.
```
# Create service principal for an agent node
kadmin: addprinc -randkey dx-agent/node2.example.com

# Create service principal for HTTP communication on an agent node
kadmin: addprinc -randkey HTTP/node2.example.com

# Repeat for all agent nodes, if needed.
```
Verify that the principals were created by running listprincs in the kadmin session:
```
kadmin: listprincs
```
In the following example output, the service names are HTTP, opscenterd, and dx-agent, the FQDNs are node*.example.com and admin, and the realm is EXAMPLE.COM:
```
HTTP/node1.example.com@EXAMPLE.COM
HTTP/node2.example.com@EXAMPLE.COM
opscenterd/node1.example.com@EXAMPLE.COM
dx-agent/node2.example.com@EXAMPLE.COM
kadmin/admin@EXAMPLE.COM
```

Create CQL roles for Kerberos principals

On your DSE cluster, create CQL roles that map to the Kerberos principals:

Start the CQL shell.
Create a role with LOGIN enabled for the opscenterd and DataStax Agent node service principals.

The names of CQL roles for Kerberos principals must include the entire Kerberos principal as returned by kadmin listprincs. Because the OpsCenter daemon and DataStax Agent are services, each principal includes a primary, instance, and realm.

Wrap role names in double quotes (") to preserve capitalization and special characters.
```
// Create role for the opscenterd principal:
CREATE ROLE "opscenterd/node1.example.com@EXAMPLE.COM" WITH LOGIN = true;

// Create roles for agent principals:
CREATE ROLE "dx-agent/node1.example.com@EXAMPLE.COM" WITH LOGIN = true;
CREATE ROLE "dx-agent/node2.example.com@EXAMPLE.COM" WITH LOGIN = true;
CREATE ROLE "dx-agent/node3.example.com@EXAMPLE.COM" WITH LOGIN = true;
```
For more information, see Add roles for Kerberos principals and CREATE ROLE.
To confirm that the roles were created, run LIST ROLES in cqlsh.
Run EXIT to close the cqlsh session.

Create keytab files

Create keytab files on your Kerberos Key Distribution Center (KDC) host, and then copy the files to the OpsCenter and DataStax Agent nodes:

Use the kadmin CLI to create keytab files that map to your opscenterd node principal and all DataStax Agent node principals.

If you created one principal for all agents, all agents can share the same keytab file. If you created separate principals for each agent, create one keytab file for each agent.

For example:

# Create keytab file for opscenterd principal
kadmin: ktadd -k /tmp/krb5_opsc.keytab opscenterd/node1.example.com

# Create keytab files for agent principals
# If multiple agent principles, use same file name for all agents
kadmin: ktadd -k /tmp/krb5_agent.keytab dx-agent/node1.example.com
kadmin: ktadd -k /tmp/krb5_agent.keytab dx-agent/node2.example.com
kadmin: ktadd -k /tmp/krb5_agent.keytab dx-agent/node3.example.com

Keytab files store information about the Kerberos principals. To obtain principal information outside of the kadmin CLI, you can use the klist command. In the following example, klist requests the ticket from the krb5_opsc.keytab file:

klist -kt krb5_opsc.keytab

Keytab name: FILE:krb5_opsc.keytab
KVNO Timestamp           Principal
---- ------------------- ----------------------------------------
2    01/26/2018 18:16:18 opscenterd/node1.example.com@EXAMPLE.COM

After creating the keytab files, exit the kadmin session:
```
kadmin: quit
```
On the OpsCenter node and each node where a DataStax Agent is running, create a directory to store the keytab file.

The recommended directory path on all nodes is /etc/opscenter/security.
```
mkdir /etc/opscenter/security
```

Copy each generated keytab file to the appropriate location.

The following examples use scp. In these commands, opscenterd@node1.example.com and dx-agent@node1.example.com are the user name and host for each node, not the service principal name.

# Copy opscenterd keytab file to the opscenterd node
scp /tmp/krb5_opsc.keytab opscenterd@node1.example.com:/etc/opscenter/security

# Copy agent keytab files to the nodes where the agents are running
scp /tmp/krb5_agent.keytab dx-agent@node1.example.com:/etc/opscenter/security
scp /tmp/krb5_agent.keytab dx-agent@node2.example.com:/etc/opscenter/security
scp /tmp/krb5_agent.keytab dx-agent@node3.example.com:/etc/opscenter/security

On the OpsCenter and DataStax Agent nodes, set the admin user as the owner of the keytab files and directories.

For example, if the admin user is cassandra and the keytab directory path is /etc/opscenter/security:

# Set ownership of the opscenterd keytab file and directory
sudo chown cassandra /etc/opscenter/security /etc/opscenter/security/krb5_opsc.keytab

# Set ownership of the agent keytab file and directory
# Run on each agent node
sudo chown cassandra /etc/opscenter/security /etc/opscenter/security/krb5_agent.keytab

Configure Kerberos authentication for a DSE cluster in OpsCenter

After you create the principals, roles, and keytab files, you must configure the Kerberos authentication settings for the cluster in OpsCenter. This allows the OpsCenter daemon and agents to authenticate with the DSE cluster using Kerberos credentials.

Add a DSE cluster to OpsCenter or edit cluster connection settings in OpsCenter.

If you prefer to modify the configuration files directly, see Kerberos configuration options.
In the cluster connection options, select DSE security (kerberos) is enabled on my cluster, and then configure the Kerberos fields as explained in the following steps.
For Service Name, enter the primary portion of the service_principal value in the kerberos_options section of dse.yaml.

For example, if dse.yaml contains service_principal: dse/host@EXAMPLE.COM, then set Service Name to dse.
For Opscenterd Client Principal, enter the opscenterd principal, such as opscenterd/opscenterd.example.com@EXAMPLE.COM.
For Opscenterd Keytab Location, enter the path to the keytab file on the opscenterd node, such as /etc/opscenter/security/krb5_opsc.keytab.

For DataStax Agent Client Principal, enter the DataStax Agent principal, such as dx-agent/node1.example.com@EXAMPLE.COM.

If you created separate principals for each agent, the value of DataStax Agent Client Principal is a placeholder. You must set the actual principal in the kerberos_client_principal property in each agent’s address.yaml file. For example, kerberos_client_principal: dx-agent/node1.example.com@EXAMPLE.COM.

For DataStax Agent Keytab Location, enter the path to the keytab file on the agent nodes, such as /etc/opscenter/security/krb5_agent.keytab.
If this is a new cluster, complete any other fields that are required to add your DSE cluster to OpsCenter.
If you created separate principals for each agent, restart all agents after adding the cluster to OpsCenter.

Kerberos configuration options

For clusters managed with OpsCenter, the OpsCenter Kerberos options are written to the [kerberos] section of the cluster-specific configuration files, CLUSTER_NAME.conf. For example the Opscenterd Keytab Location and DataStax Agent Keytab Location fields are written to the opscenterd_keytab_location and agent_keytab_location options respectively. For all options, see Cassandra connection properties: Kerberos sections.

If you created separate principals for each agent, you must configure the agent-specific Kerberos authentication settings in each agent’s address.yaml file. For all options, see DataStax Agent configuration: Agent Kerberos options.