Preparing DSE nodes for Kerberos

Example instructions to install the Kerberos client libraries on DSE nodes, verify DNS entry, system time settings, and set up a service principal.

Use these instructions as guidelines for installing the Kerberos client libraries on DSE nodes, verifying DNS entry, and system time settings. Each node in your cluster requires DNS to be working properly, NTP to be enabled and the system time synchronized, and the Kerberos client libraries installed.

Note: Do not upgrade DataStax Enterprise and set up Kerberos at the same time; see General restrictions and limitations during the upgrade process.

Prerequisites

Complete the following prerequisites:
  • Each node has a DNS entry that resolves the hostname to the correct IP address
  • Each node uses NTP for the system time
  • All KDS requirements have been met, see Kerberos guidelines.
  • If using Oracle Java 8, DataStax recommends using the latest version, however the minimum version is 1.8.0_151
  • Each node has the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files enabled. Refer to Enabling JCE Unlimited.
    Note: If you are not using the JCE Unlimited Strength Jurisdiction Policy, make sure that your ticket granting principal does not use AES-256. Starting in JDK 8u161, JCE Unlimited is enabled by default. Refer to the Release Notes for JDK 8u161.

Procedure

  1. Verify the DNS resolves hostname and NTP settings on each node:
    1. For Kerberos the hostname is used for the Service Principal name, therefore the hostname must resolve to the correct IP address. To verify:
      nslookup $(hostname --fqdn) && hostname --fqdn && hostname -i
      Server:		10.200.1.10
      Address:	10.200.1.10#53
      
      Name:	node.example.com
      Address: 10.200.182.183
      
      node.example.com
      10.200.182.183
    2. Kerberos authentication is sensitive to system time, manually set system clocks may cause issues. On each node, confirm that NTP is configured and running:
      ntpq -p
      
      remote           refid            st t when poll reach   delay   offset  jitter
      ==============================================================================
      *li506-17.member 209.51.161.238   2  u 331 1024  377   80.289    1.384   1.842
      -tock.eoni.com   216.228.192.69   2  u 410 1024  377   53.812    1.706  34.692
      +time01.muskegon 64.113.32.5      2  u 402 1024  377   59.378   -1.635   1.840
      -time-a.nist.gov .ACTS.           1  u 746 1024  151  132.832   26.931  55.018
      +golem.canonical 131.188.3.220    2  u 994 1024  377  144.080   -1.732  20.072
  2. On each node, install Kerberos:
    • RHEL-based systems:
      sudo yum install krb5-workstation krb5-libs krb5-pkinit-openssl
    • Debian-based systems:
      sudo apt-get install krb5-user krb5-config krb5-pkinit
  3. Your the krb5.conf for your REALM in the /etc directory on each DataStax Enterprise node.
    The krb5.conf file contains Realm configuration required by Kerberos, see MIT Kerberos krb5.conf documentation. DataStax recommends not using DNS lookup for KDC, and REALM entries. Relying on DNS may negative impact performance and functionality. Verify that the libdefaults section contains the following entries.
    [libdefaults]
    dns_lookup_kdc = false
    dns_lookup_realm = false
    
  4. On the Key Distribution Center (KDC) server, create a Service Principal and keytab for each node:
    1. Add a DSE Service Principal and HTTP Principal for each node using the kadmin command:
      kadmin -p user_name/admin
      addprinc -randkey service_name/FQDN
      addprinc -randkey HTTP/FQDN
      quit
      Parameter Description
      kadmin Launch Kerberos admin shell with an administrator account that has add privileges.
      addprinc Creates a new service principal for each DSE node.
      dse Service name is dse.
      FQDN The fully qualified domain name of the DSE node.
      -randkey Sets the key of the principal to a random value.
      Tip: For more details on using kadmin, see the kadmin help.
      Example:
      kadmin -p kadmin/admin
      kadmin: addprinc -randkey dse/node1.example.com
      kadmin: addprinc -randkey HTTP/node1.example.com
      kadmin: addprinc -randkey dse/node2.example.com
      kadmin: addprinc -randkey HTTP/node2.example.com
    2. Optional: Verify that the principals have been added by running the listprincs command within kadmin:
      kadmin: listprincs
      HTTP/node1.example.com@EXAMPLE.COM
      HTTP/node2.example.com@EXAMPLE.COM
      dse/node1.example.com@EXAMPLE.COM
      dse/node2.example.com@EXAMPLE.COM
      kadmin/admin@EXAMPLE.COM
      where node*.example.com is the FQDN and EXAMPLE.COM is your Kerberos realm, which must be all uppercase.
    3. Create a keytab file for each node and add the principals keys for each node:
      kadmin: ktadd -k keytabfilename dse/FQDN
      kadmin: ktadd -k keytabfilename HTTP/FQDN

      where ktadd -k creates or appends a key for the DSE service and HTTP principals.

      Example:
      kadmin: ktadd -k /tmp/node1.keytab dse/node1.example.com
      kadmin: ktadd -k /tmp/node1.keytab HTTP/node1.example.com
      kadmin: ktadd -k /tmp/node2.keytab dse/node2.example.com
      kadmin: ktadd -k /tmp/node2.keytab HTTP/node2.example.com
    4. Optional: Use the klist command to view your principals in each keytab file:
      For example:
      sudo klist -e -kt /tmp/node1.keytab
      
      Keytab name: FILE:/tmp/node1.keytab
      KVNO Timestamp        Principal
      ---- ---------------- ----------------------------------------------
      2    14/02/16 22:03   HTTP/node1FQDN@YOUR_REALM (des3-cbc-sha1)
      2    14/02/16 22:03   HTTP/node1FQDN@YOUR_REALM (arcfour-hmac)
      2    14/02/16 22:03   HTTP/node1FQDN@YOUR_REALM (des-hmac-sha1)
      2    14/02/16 22:03   HTTP/node1FQDN@YOUR_REALM (des-cbc-md5)
      2    14/02/16 22:03   dse/node1FQDN@YOUR_REALM (des3-cbc-sha1)
      2    14/02/16 22:03   dse/node1FQDN@YOUR_REALM (arcfour-hmac)
      2    14/02/16 22:03   dse/node1FQDN@YOUR_REALM (des-hmac-sha1)
      2    14/02/16 22:03   dse/node1FQDN@YOUR_REALM (des-cbc-md5)
      where: -e displays the encryption type and -kt displays the keytab file and its timestamp.
    5. Distribute keytab files from the KDC server to the nodes, to ease DSE Kerberos configuration ensure the files have the same name on each node:
      scp /tmp/node1.keytab node_admin@node_hostname:/etc/dse/dse.keytab
    6. Change the permissions on dse.keytab so that only the dse_service_account user can read and write to the keytab file:
      sudo chown dse:dse /etc/dse/dse.keytab && sudo chmod 600 /etc/dse/dse.keytab