BYOK GCP with Astra DB console

Encryption is a widely accepted mechanism to secure data against breaches. By default, DataStax Astra DB encrypts data, and providers such as Google Cloud offer encryption solutions. However, you may want to further limit data access, because cloud providers have access to the keys and ultimately to the data.

To address this security concern, DataStax Astra DB allows you to associate a Customer Managed Encryption Key (one per region) that you defined in Google Cloud console with a Customer Key that you create in Astra DB.

In this topic, we’ll use these terms:

  • Bring Your Own Key (BYOK) refers to the overall capability.

  • Customer Managed Encryption Key (CMEK) refers to a particular key type in Google Cloud Key Management Service (KMS).

  • Customer Key refers to the corresponding key association in Astra DB.

This BYOK feature:

  • Is available for serverless Astra DB databases with these cloud providers:

  • Is not available in the Astra DB Free Tier.

  • Free Tier users: instead of submitting a Support ticket, click the Chat icon in the Astra DB console’s lower-right corner. Ask the DataStax onboarding representative about options to upgrade your organization and use BYOK.

Introduction

Data encryption is defined as a process that transforms data into an encoded format. Once encoded, the data is incomprehensible without being decrypted. Data encryption is essential for organizations in all industries because it protects data from unauthorized access. When thinking of data encryption, two main scenarios are often considered:

  • Data at rest

    Encrypting data while it is stored in the file storage in use.

  • Data in transit

    Encrypting data while it travels through private or public networks.

BYOK allows customers to manage encryption for data at rest.

Benefits

With BYOK, you can take full control of the encryption keys when storing data in the cloud. Google Cloud Key Management Service (KMS) provides protection against data breaches by alerting you when tampering occurs. In KMS, you can configure specific policies to adhere to compliance guidelines, such as auditing, key rotation, and access.

Setting up a corresponding Customer Key for a Google Cloud based Astra DB database separates the:

  • The encrypted lock

  • The key that encrypts/decrypts data

This separation of lock and key is considered a best practice to secure data via encryption.

After setting up a Customer Managed Encryption Key (CMEK) in your Google Cloud project, you can use either the Astra DB console or the DataStax DevOps API, to associate the existing CMEK with a Customer Key in Astra DB.

In Astra DB console, under Organization Settings, see the Key Encryption section of Security Settings.

In addition to the console UI, BYOK provides three DataStax DevOps API calls that allow you to programmatically:

  • Create a new association between a CMEK (created in Google Cloud KMS) and your Astra DB data, in a specific region.

  • List all Customer Keys that are associated with protecting the Google Cloud based Astra DB data for your organization.

  • List a specific Customer Key for an organization, based on the specified cloud provider (gcp) & region combination.

Please contact DataStax Support if you need to delete Customer Keys from your Astra DB organization. If you agree, the DataStax Support team may perform the key deletion on your behalf. Once a registered association with a Google Cloud Customer Managed Encryption Key is deleted from your Astra DB organization, the default data encryption provided by Astra DB is used.

Prerequisites

  1. Create your Astra DB database using the Astra DB console. In the case of this BYOK feature, create a Google Cloud based database, and choose one of the available regions to start.

  2. Set up a Customer Managed Encryption Key (one per region, as needed) in Google Cloud KMS.

  3. Ensure you have the required Roles and permissions.

  4. Ensure that you know your Astra DB organization ID. When you log into the Astra DB console, your organization ID is in the generated URL. For example, in the URL https://astra.datastax.com/org/a99999c7-b934-436c-9999-9999999a3b5d/manage, the organization ID is a99999c7-b934-436c-9999-9999999a3b5d.

Multi-region support

The BYOK feature is supported in multi-region Astra DB environments; however, each region is encrypted using its own key. Keys cannot be shared across regions. For a given organization:

  • if you have an Astra DB database that spans multiple regions,

  • and if you’ve defined a Customer Managed Key in AWS or Google Cloud,

  • create a Customer Key in each (provider + region) combination.

Pricing

There is no additional cost to using BYOK with Astra DB. As noted previously, BYOK is not available in the Astra DB Free Tier.

Customer Managed Encryption Keys in Google Cloud may incur a monthly fee, and a fee for use in excess of the Google Cloud free tier. The fees are counted against the Google Cloud KMS quotas for your project. For details, see Customer Managed Encryption Key in the Google Cloud documentation.

Roles and permissions

The following Astra DB roles can manage Customer Keys.

  • Organization Administrator

  • Database Administrator

To manage Customer Keys, your Astra DB account must have these permissions enabled.

  • org-cmk-read

  • org-cmk-write

Google Cloud KMS

This section describes steps you’ll perform in Google Cloud console to:

  • Create a custom role with specific permissions needed by Astra DB

  • Create a Customer Managed Encryption Key

Once your CMEK with assigned role and permissions are setup, you can use the DevOps API to submit calls that create and view associated Customer Keys for your Astra DB data.

Prerequisite API step to determine account for GCS buckets

Before you create a custom role and a Customer Managed Encryption Key (CMEK) in Google Cloud console, determine (for a given region) in which project does Astra DB store your Google Cloud Storage (GCS) buckets. To find that information, use cURl or Postman to submit a GET on /v2/kms/provider/gcp/region/<region-name>/accounts. Example:

curl --request GET \
 --url 'https://api.astra.datastax.com/v2/kms/provider/<provider>/region/<region>/accounts' \
 --header 'Accept: application/json' \
 --header 'Authorization: Bearer <application_token>'

The GET’s response includes the ID of the Google Cloud Storage (GCS) account, into which Astra DB will store your database’s encrypted data. Copy the ID number returned in the GET’s response. Then, when you configure your Google Cloud CMEK (see the steps below), you’ll specify this account number, and grant DataStax permission to access this account.

Create a custom role and grant permissions

In Google Cloud console, create a custom role and grant the minimum required permissions needed by Astra DB to access your Google Cloud Storage (GCS) buckets. Custom roles let you group permissions and assign them to principals in your project or organization. You can manually select permissions or import permissions from another role.

  1. After authenticating into your Google Cloud project, search for "IAM & Admin".

  2. On the "IAM & Admin" page, from the left-side navigation panel, choose Roles.

  3. Click Create Role.

  4. Click ADD PERMISSIONS and filter on kms.

  5. From among the filtered results, further narrow down by enabling Cloud KMS Viewer and Cloud KMS CryptoKey Encrypter/decrypter.

  6. Scroll down the refreshed results list, and at a minimum, enable the following permissions for your custom role:

    cloudkms.cryptoKeyVersions.useToDecrypt
    cloudkms.cryptoKeyVersions.useToEncrypt
    cloudkms.cryptoKeys.get

    Example:

    Google Cloud Key Management Add Permissions By Role to ensure DataStax access to CMEK.

  7. Click ADD.

  8. Modify values such as the custom role’s Title and ID string.

  9. Click CREATE.

You’ll specify this custom role during the CMEK steps outlined below.

Create a Customer Managed Encryption Key

  1. While authenticated into your Google Cloud project, search for, or navigate to, the "Key Management" service. The Cloud Key Management Service (Cloud KMS) lets you create, use, rotate, and manage cryptographic keys. A cryptographic key is a resource that is used for encrypting and decrypting data or for producing and verifying digital signatures.

  2. On the Cloud KMS page, if you haven’t already, click ENABLE.

  3. Click Create Key Ring, or open an existing Key Ring if already defined. Key rings group keys together to keep them organized.

    1. If new, provide a name for your Key Ring, such as testkeyring.

    2. For Location type, choose Region.

    3. Select the same region of your Astra DB database(s).

  4. On the "Create Key" page:

    1. Enter a Name for your key.

    2. For Key type, choose Generated Key.

    3. For Protection level, choose Software.

    4. For Purpose, choose Symmetric encrypt/decrypt.

    5. Decide on the key Rotation cadence to match your preference. The default is 90 days.

    6. When ready, click CREATE.

  5. On the "Keys for <your-key-ring-name> key ring" page, click the link for your newly created key’s name.

  6. On the "Key: <your-key-name>" page, click the PERMISSIONS tab.

  7. In this step, you’ll provide:

    • The storage account ID returned in the GET /v2/kms/provider/gcp/region/<region-name>/accounts API call. See Prerequisite API step to determine account for GCS buckets.

    • The custom role you created earlier. See Create a custom role and grant permissions.

      Provide those values so you can properly setup, with the necessary role and permissions, the following principles:

      <projectNumber>-compute@developer.gserviceaccount.com
      service-<projectNumber>@gs-project-accounts.iam.gserviceaccount.com

      Where <projectNumber> is the output of GET /v2/kms/provider/gcp/region/<region-name>/accounts.

      1. On this PERMISSIONS tab, click ADD.

      2. Paste into the New principals textbox the account ID that you copied from the response of the GET /v2/kms/provider/gcp/region/<region-name>/accounts. (Again, refer to Prerequisite API step to determine account for GCS buckets).

      3. Select the name of the custom role that you created earlier. As a reminder, in an earlier step, we applied the following permissions to the custom role:

        cloudkms.cryptoKeyVersions.useToDecrypt
        cloudkms.cryptoKeyVersions.useToEncrypt
        cloudkms.cryptoKeys.get

        Those are the minimum permissions needed in the role that’s assigned to your CMEK.

      4. When you’re ready, click SAVE.

  8. On the "Key: <your-key-name>" page, click the VERSIONS tab.

  9. Under Actions, expand the three vertical dots and select Copy resource name. Example:

    Google Cloud Key Management menu selecting Copy Resource Name option for a newly created key.

    The format of the copied resource name is:

    projects/<id>/locations/<region-name>/keyRings/<your-key-ring-name>/cryptoKeys/<your-key-name>/cryptoKeyVersions/<version>

    Optionally, you can remove the /cryptoKeyVersions/<version> portion at the end of the copied resource name. Copy the rest. Example:

    projects/this-that-999999/locations/us-east1/keyRings/testkeyring/cryptoKeys/mydefaultkey

Save the copied resource name of your Customer Managed Encryption Key. You’ll provide its value while adding a Customer Key association in Astra DB console. For those steps, see the next section.

Astra DB console’s Security Settings & Key Encryption

Reminder: By default, Astra DB encrypts data. The BYOK feature enhances data security by allowing you to instead associate a Customer Key in Astra DB with a Customer Managed Encryption Key (CMEK) that you defined via a cloud provider, such as in Google Cloud KMS.

Adding a new Customer Key

Having completed the steps in Google Cloud KMS to set up a CMEK, and granted DataStax access to your Google Cloud Storage (GCS) buckets, follow these steps in Astra DB console to add a new Customer Key:

  1. From your Organization name near the Astra DB console’s upper-left corner, click Organization Settings, and navigate to Security Settings.

  2. On Security Settings, if your account includes the necessary permissions and roles, notice Key Encryption, a Premium feature.

    Astra DB console’s Security Settings and Key Encryption UI
  3. On Key Encryption, click Add Keys.

  4. On Add Customer Key:

    1. Choose Google Cloud Platform as the provider.

    2. Choose the same region used by your Google Cloud KMS Customer Managed Encryption Key and the Astra DB database that will have its data encrypted. Example:

      Add Key dialog with GCP provider and region selected.

    3. For the Key ID value, paste in the resource name of your Customer Managed Encryption Key, which you copied in Google Cloud KMS.

    4. Click Add Key.

  5. The Security Settings page displays the one or more Customer Key(s) defined in your organization. Example:

    Security Settings displays keys for your organization.

In the Status column:

  • In-Use means a database in your organization is actively using the Customer Key.

  • Available status indicates a Customer Key has been added for your organization, but no Astra DB database with that Provider and Region combination is using the key.

Customer Key status indicators

In addition to the status on Key Encryption in Security Settings (illustrated above), look for Customer Key status indicators on the Dashboard’s Overview tab.

Here’s an example with the key icon for the current database, indicating a Customer Key is used to protect its data:

Astra DB database details has Customer Key icon

A Region Details example with a Customer Key status indicator:

Astra DB regions details with status indicator