BYOK GCP with DevOps API (Serverless)

Encryption is a widely accepted mechanism to secure data against breaches. By default, DataStax Astra DB encrypts data, and cloud providers such as Google Cloud offer encryption solutions. However, you may want to further limit data access, because cloud providers have access to the keys and ultimately to the data.

To address this security concern, DataStax Astra DB allows you to associate a Customer Managed Encryption Key (one per region) that you defined in Google Cloud’s Key Management Service (KMS) with a Customer Key that you create in Astra DB.

In this topic, we’ll use these terms:

Bring Your Own Key (BYOK) refers to the overall capability.
Customer Managed Encryption Key (CMEK) refers to a particular key type in Google Cloud KMS.
Customer Key refers to the corresponding key association in Astra DB.

This BYOK feature:

Is available for Astra DB Serverless users with these cloud providers:
- AWS - supported via Astra DB BYOK AWS with DevOps API (Serverless) and BYOK AWS with Astra Portal (Serverless).
- Google Cloud - supported via Astra DB BYOK GCP with DevOps API (Serverless) and BYOK GCP with Astra Portal (Serverless).
Is not available with the Astra DB Free Plan. Using the Chat icon in Astra Portal, ask the DataStax representative about upgrading your organization to a paid plan so you can use BYOK and additional features.

For related details, see the Customer Keys API reference.

BYOK for Astra DB Classic is available on AWS only, via DevOps API, and upon request. Not configurable via Astra Portal.

Introduction

Data encryption is defined as a process that transforms data into an encoded format. Once encoded, the data is incomprehensible without being decrypted. Data encryption is essential for organizations in all industries because it protects data from unauthorized access. When thinking of data encryption, two main scenarios are often considered:

Data at rest

Encrypting data while it is stored in the file storage in use.
Data in transit

Encrypting data while it travels through private or public networks.

BYOK allows customers to manage encryption for data at rest.

Benefits

BYOK allows you to take full control of the encryption keys when storing data in the cloud. Google Cloud KMS provides protection against data breaches by alerting you when tampering occurs. In KMS, you can configure specific policies to adhere to compliance guidelines, such as auditing, key rotation, and access.

Setting up a corresponding Customer Key for an Astra DB database hosted in a Google Cloud region enables the separation of:

The encrypted lock
The key that encrypts/decrypts data

This separation of lock and key is considered a best practice to secure data via encryption.

After setting up a Customer Managed Encryption Key (CMEK) in your Google Cloud project’s Key Management Service, use the DataStax DevOps API to associate an existing Google Cloud CMEK with a Customer Key in Astra DB:

Create a new association between the Google Cloud CMEK and your Astra DB data, in a specific region.
List all Customer Keys that are associated with protecting the Google Cloud based Astra DB data for your organization.
List a specific Customer Key for an organization, based on the specified cloud provider (gcp) & region combination.

Key deletion

Please contact DataStax Support if you need to delete a key from your Astra DB organization. If you agree, the DataStax Support team may perform the key deletion on your behalf. Once a registered association with a Google Cloud Customer Managed Encryption Key is deleted from your Astra DB organization, the default data encryption provided by Astra DB is used.

Prerequisites

Create your Astra DB database using Astra Portal. For the examples in this topic, in the case of this BYOK feature, create a Google Cloud based database, and choose one of the available regions to start.
Before submitting the DataStax DevOps API calls, set up a Customer Managed Encryption Key (one per region, as needed) in the Google Cloud KMS, as outlined in this topic.
Create an application token to authenticate your service account in the DevOps API.
Ensure you have permission to create and view Customer Keys. See Roles and permissions.
Ensure that you know your Astra DB organization ID. When you log into the Astra DB console, your organization ID is in the generated URL. For example, in the URL https://astra.datastax.com/org/a99999c7-b934-436c-9999-9999999a3b5d/manage, the organization ID is a99999c7-b934-436c-9999-9999999a3b5d.

Multi-region support

The BYOK feature is supported in multi-region Astra DB environments; however, each region is encrypted using its own key. Keys cannot be shared across regions. For a given organization:

if you have an Astra DB database that spans multiple regions,
and if you’ve defined a Customer Managed Key in AWS or Google Cloud,
create a Customer Key in each (provider + region) combination.

Pricing

There is no additional cost to using BYOK with Astra DB. As noted previously, BYOK is not available with the Astra DB Free Plan.

Customer Managed Encryption Keys in Google Cloud may incur a monthly fee, and a fee for use in excess of the Google Cloud free tier. The fees are counted against the Google Cloud KMS quotas for your project. For details, see Customer Managed Encryption Key in the Google Cloud documentation.

Roles and permissions

The following Astra DB roles can manage Customer Keys.

Organization Administrator
Database Administrator

To manage Customer Keys, your Astra DB account must have these permissions enabled.

org-cmk-read
org-cmk-write

Google Cloud console

This section describes steps you’ll perform in Google Cloud console to:

Create a custom role with specific permissions needed by Astra DB
Create a Customer Managed Encryption Key

Once your CMEK with assigned role and permissions are setup, you can use the DevOps API to submit calls that create and view associated Customer Keys for your Astra DB data.

Prerequisite API step to determine account for GCS buckets

Before you create a custom role and a Customer Managed Encryption Key (CMEK) in Google Cloud console, determine (for a given region) in which project does Astra DB store your Google Cloud Storage (GCS) buckets. To find that information, use cURl or Postman to submit a GET on /v2/kms/provider/gcp/region/<region-name>/accounts. Example:

curl --request GET \
 --url 'https://api.astra.datastax.com/v2/kms/provider/<provider>/region/<region>/accounts' \
 --header 'Accept: application/json' \
 --header 'Authorization: Bearer <application_token>'

The GET’s response includes the ID of the Google Cloud Storage (GCS) account, into which Astra DB will store your database’s encrypted data. Copy the ID number returned in the GET’s response. Then, when you configure your Google Cloud CMEK (see the steps below), you’ll specify this account number, and grant DataStax permission to access this account.

Create a custom role and grant permissions

In Google Cloud console, create a custom role and grant the minimum required permissions needed by Astra DB to access your Google Cloud Storage (GCS) buckets. Custom roles let you group permissions and assign them to principals in your project or organization. You can manually select permissions or import permissions from another role.

After authenticating into your Google Cloud project, search for "IAM & Admin".
On the "IAM & Admin" page, from the left-side navigation panel, choose Roles.
Click Create Role.
Click ADD PERMISSIONS and filter on kms.
From among the filtered results, further narrow down by enabling Cloud KMS Viewer and Cloud KMS CryptoKey Encrypter/decrypter.
Scroll down the refreshed results list, and at a minimum, enable the following permissions for your custom role:
```
cloudkms.cryptoKeyVersions.useToDecrypt
cloudkms.cryptoKeyVersions.useToEncrypt
cloudkms.cryptoKeys.get
```
Example:
Click ADD.
Modify values such as the custom role’s Title and ID string.
Click CREATE.

You’ll specify this custom role during the CMEK steps outlined below.

Create a Customer Managed Encryption Key

While authenticated into your Google Cloud project, search for, or navigate to, the "Key Management" service. The Cloud Key Management Service (Cloud KMS) lets you create, use, rotate, and manage cryptographic keys. A cryptographic key is a resource that is used for encrypting and decrypting data or for producing and verifying digital signatures.
On the Cloud KMS page, if you haven’t already, click ENABLE.
Click Create Key Ring, or open an existing Key Ring if already defined. Key rings group keys together to keep them organized.
1. If new, provide a name for your Key Ring, such as testkeyring.
2. For Location type, choose Region.
3. Select the same region of your Astra DB database(s).
On the "Create Key" page:
1. Enter a Name for your key.
2. For Key type, choose Generated Key.
3. For Protection level, choose Software.
4. For Purpose, choose Symmetric encrypt/decrypt.
5. Decide on the key Rotation cadence to match your preference. The default is 90 days.
6. When ready, click CREATE.
On the "Keys for <your-key-ring-name> key ring" page, click the link for your newly created key’s name.
On the "Key: <your-key-name>" page, click the PERMISSIONS tab.
In this step, you’ll provide:
- The storage account ID returned in the GET /v2/kms/provider/gcp/region/<region-name>/accounts API call. See Prerequisite API step to determine account for GCS buckets.
- The custom role you created earlier. See Create a custom role and grant permissions.
  
  Provide those values so you can properly setup, with the necessary role and permissions, the following principles:
  <projectNumber>-compute@developer.gserviceaccount.com service-<projectNumber>@gs-project-accounts.iam.gserviceaccount.com
  Where <projectNumber> is the output of GET /v2/kms/provider/gcp/region/<region-name>/accounts.
  1. On this PERMISSIONS tab, click ADD.
  2. Paste into the New principals textbox the account ID that you copied from the response of the GET /v2/kms/provider/gcp/region/<region-name>/accounts. (Again, refer to Prerequisite API step to determine account for GCS buckets).
  3. Select the name of the custom role that you created earlier. As a reminder, in an earlier step, we applied the following permissions to the custom role:
    
    cloudkms.cryptoKeyVersions.useToDecrypt cloudkms.cryptoKeyVersions.useToEncrypt cloudkms.cryptoKeys.get
    
    Those are the minimum permissions needed in the role that’s assigned to your CMEK.
  4. When you’re ready, click SAVE.
On the "Key: <your-key-name>" page, click the VERSIONS tab.
Under Actions, expand the three vertical dots and select Copy resource name. Example:

The format of the copied resource name is:

projects/<id>/locations/<region-name>/keyRings/<your-key-ring-name>/cryptoKeys/<your-key-name>/cryptoKeyVersions/<version>

Optionally, you can remove the /cryptoKeyVersions/<version> portion at the end of the copied resource name. Copy the rest. Example:

projects/this-that-999999/locations/us-east1/keyRings/testkeyring/cryptoKeys/mydefaultkey

Save the copied resource name of your Customer Managed Encryption Key. You’ll provide its value while adding a Customer Key association in Astra Portal. For those steps, see the next section.

DevOps v2 API to manage Customer Keys

Submit DevOps API calls to create and view Customer Keys in your Astra DB organization. In effect, you’ll register a Customer Key in Astra DB that is associated with an already created (see above) Customer Managed Encryption Key (CMEK) in Google Cloud.

If you are using Postman for your API calls, ensure you use the raw option to enter the body of your API call.

For descriptions of the /v2/kms calls, see the sections below. For more, see the DevOps v2 API reference details.

Create a Customer Key for Astra DB

Use the following POST to create a Customer Key in Astra DB that will be associated with an existing Customer Managed Encryption Key (CMEK) that was created in Google Cloud.

In the request payload, specify your Astra DB organizationID, the keyID, and the region.

Recall that keyID for the POST payload is the CMEK’s resource name, as generated in the Google Cloud console. See that step to copy the CMEK’s resource name, in Create a Customer Managed Encryption Key above.

In each Astra DB organization, there can be one Customer Key for a given cloud provider and region combination. A successful Customer Key creation returns 200 OK in the response.

When you log into the Astra DB console, your organization ID is in the generated URL. For example, in the URL https://astra.datastax.com/org/a99999c7-b934-436c-9999-9999999a3b5d/manage, the organization ID is a99999c7-b934-436c-9999-9999999a3b5d.

cURL command (/v2/kms)
Result

curl --request POST \
  --url 'https://api.astra.datastax.com/v2/kms' \
  --header 'Accept: application/json' \
  --header 'Authorization: Bearer <application_token>' \
  --data '{
  "orgId": "<organization-id>",
  "gcp": {
    "keyID": "String - Resource name of the key",
    "region": "String - Region in which the key exists"
  }
}'

200 OK

Or one of the following:

400 Bad request.
403 The user is forbidden to perform the operation.
404 The specified resource was not found.
500 A server error occurred.

{
  "errors": [
    {
      "description": "string",
      "internalCode": "string",
      "internalTxId": "string"
    }
  ]
}

Get all Customer Keys in an Astra DB organization

Retrieve all existing Customer Keys of an Astra DB organization.

cURL command (/v2/kms)
Result

curl --request GET \
 --url 'https://api.astra.datastax.com/v2/kms/' \
 --header 'Accept: application/json' \
 --header 'Authorization: Bearer <application_token>'

When one or more keys are found for the organization, 
the 200 OK response contains, for each key:

[
  {
    "OrganizationID": "String - The ID of your organization",
    "CloudProvider": "String - The cloud provider, such as aws or gcp",
    "KeyID": "String - The ID of the Customer Key",
    "Region": "String - The region in which the Customer Key is used"
  }
]

If there are no keys setup for the organization, 
the 404 response contains:

{
  "description": "Organization: <orgID> does not have 
                  any CMKs associated with it."
}

Other potential response codes:

400 Bad request.
403 The user is forbidden to perform the operation.
500 A server error occurred.

Get a single Customer Key in an Astra DB organization

Retrieves a Customer Key in an Astra DB organization for a given cloud provider & region combination. The provider value in this example should be gcp.

The GET response also returns the provider ID of the storage account where your Google Cloud Storage (GCS) buckets exist. If not done already, you can use this information to configure or modify (back in the Google Cloud console) your CMEK, so that Astra DB has access to the key.

cURL command (/v2/kms)
Result

curl --request GET \
 --url 'https://api.astra.datastax.com/v2/kms/provider/<provider>/region/<region>' \
 --header 'Accept: application/json' \
 --header 'Authorization: Bearer <application_token>'

When the key is found for the given provider and region, 
the 200 OK response contains:

{
  "orgId": "The ID of your organization",
  "provider": "aws or gcp": 
  { 
    "keyID": "The ID of the provider account where DataStax stores your data buckets",
    "region": "region-name"
  }
}

If there are no keys setup for the organization's provider 
and region combination, the 404 response contains:

{
  "description": "Could not find <provider> data in 
                  region <region> for organization <orgID>."
}

Other potential response codes:

400 Bad request.
403 The user is forbidden to perform the operation.
500 A server error occurred.