BYOK AWS with DevOps API (Serverless)

Encryption is a widely accepted mechanism to secure data against breaches. By default, DataStax Astra DB encrypts data, and cloud providers such as AWS or Google Cloud offer encryption solutions. However, you may want to further limit data access, because cloud providers have access to the keys and ultimately to the data.

To address this security concern, DataStax Astra DB allows you to associate a Customer Managed Key (one per region) that you defined in the AWS Key Management Service (KMS) with a Customer Key that you create in Astra DB.

In this topic, we’ll use these terms:

Bring Your Own Key (BYOK) refers to the overall capability.
Customer Managed Key (CMK) refers to a particular key type in AWS KMS.
Customer Key refers to the corresponding key association in Astra DB.

This BYOK feature:

Is available for Astra DB Serverless users with these cloud providers:
- AWS - supported via Astra DB BYOK AWS with DevOps API (Serverless) and BYOK AWS with Astra Portal (Serverless).
- Google Cloud - supported via Astra DB BYOK GCP with DevOps API (Serverless) and BYOK GCP with Astra Portal (Serverless).
Is not available with the Astra DB Free Plan. Using the Chat icon in Astra Portal, ask the DataStax representative about upgrading your organization to a paid plan so you can use BYOK and additional features.

For related details, see the Customer Keys API reference.

BYOK for Astra DB Classic is available on AWS only, via DevOps API, and upon request. Not configurable via Astra Portal.

Introduction

Data encryption is defined as a process that transforms data into an encoded format. Once encoded, the data is incomprehensible without being decrypted. Data encryption is essential for organizations in all industries because it protects data from unauthorized access. When thinking of data encryption, two main scenarios are often considered:

Data at rest

Encrypting data while it is stored in the file storage in use.
Data in transit

Encrypting data while it travels through private or public networks.

BYOK allows customers to manage encryption for data at rest.

Benefits

BYOK allows you to take full control of the encryption keys when storing data in the cloud. AWS KMS provides protection against data breaches by alerting you when tampering occurs. In KMS, you can configure specific policies to adhere to compliance guidelines, such as auditing, key rotation, and access.

Setting up a corresponding Customer Key for an Astra DB database hosted in an AWS region enables the separation of:

The encrypted lock
The key that encrypts/decrypts data

This separation of lock and key is considered a best practice to secure data via encryption.

After setting up a Customer Managed Key (CMK) in your AWS account’s Key Management Service, use Astra Portal, or the DataStax DevOps API, to associate an existing AWS CMK with a Customer Key in Astra DB.

In Astra Portal, under Settings, see the Key Encryption section of Security Settings.

In addition to the console UI, BYOK provides DevOps API calls that allow you to programmatically:

Create a new association between a CMK (created in AWS KMS) and your Astra DB data, in a specific region.
List all Customer Keys that are associated with protecting the AWS-based Astra DB data for your organization.
List a specific Customer Key for an organization, based on the specified cloud provider (aws) & region combination.

Key deletion

Please contact DataStax Support if you need to delete a key from your Astra DB organization. If you agree, the DataStax Support team may perform the key deletion on your behalf. Once a registered association with an AWS KMS Customer Managed Key is deleted from your Astra DB organization, the default data encryption provided by Astra DB is used.

Prerequisites

Create your Astra DB database using Astra Portal. In the case of this BYOK feature, create an AWS-based database, and choose one of the available regions to start.
Before submitting the DataStax DevOps API calls, set up a Customer Managed Key (one per region, as needed) in the AWS KMS, as outlined in this topic.
Create an application token to authenticate your service account in the DevOps API.
Ensure you have permission to create and view Customer Keys. See Roles and permissions.
Ensure that you know your Astra DB organization ID. When you log into the Astra DB console, your organization ID is in the generated URL. For example, in the URL https://astra.datastax.com/org/a99999c7-b934-436c-9999-9999999a3b5d/manage, the organization ID is a99999c7-b934-436c-9999-9999999a3b5d.

Multi-region support

The BYOK feature is supported in multi-region Astra DB environments; however, each region is encrypted using its own key. Keys cannot be shared across regions. For a given organization:

if you have an Astra DB database that spans multiple regions,
and if you’ve defined a Customer Managed Key in AWS or Google Cloud,
create a Customer Key in each (provider + region) combination.

Pricing

There is no additional cost to using BYOK with Astra DB. As noted previously, BYOK is not available with the Astra DB Free Plan.

Customer Managed Keys in AWS may incur a monthly fee, and a fee for use in excess of the AWS free tier. The fees are counted against the AWS KMS quotas for your AWS account. For details, see:

Roles and permissions

The following Astra DB roles can manage Customer Keys.

Organization Administrator
Database Administrator

To manage Customer Keys, your Astra DB account must have these permissions enabled.

org-cmk-read
org-cmk-write

AWS KMS console

Before you create a Customer Managed Key in the AWS Key Management Service (KMS), submit the following REST API call:

curl --request GET \
 --url 'https://api.astra.datastax.com/v2/kms/provider/<provider>/region/<region>/accounts' \
 --header 'Accept: application/json' \
 --header 'Authorization: Bearer <application_token>'

The GET’s response includes the provider_id of the AWS account where DataStax will store your S3 buckets. Copy the ID number returned in the GET’s response. Then, when you configure your AWS KMS Customer Managed Key (see the steps below), you’ll specify this account number, and grant DataStax permission to access this account.

After authenticating into your AWS organization and account, search for, or navigate to, the Key Management Service (KMS).
On the KMS page, click the left sidebar link for Customer managed keys.
Click Create key.
Choose the Symmetric option for a single key, which will be used to encrypt and decrypt data for your Astra DB database in an Amazon S3 bucket. Click Next.
On Add labels, enter a memorable alias for your CMK, such as datastax-byok-cmk, and optionally provide a brief Description and user-defined Tags. Click Next.
On Define key administrative permissions, decide if you want to keep the option to Allow key administrators to delete this key. Also on this dialog, expand the down arrow in Key administrators section and select the relevant AWS accounts. Click Next.
On Define key usage permissions, decide which AWS accounts you want to authorize for the CMK’s usage. For now, skip the Other AWS Accounts option; there’s an option on the next screen to edit the in-progress Key policy definition. Click Next.
On the Review page, check the selected options, and then scroll through the generated Key policy. There are a couple manual edits to perform. Add the full section (see below) that starts with:
```
        {
            "Sid": "Allow an external account to use this KMS key",
            .
            .
            .
        }
```

Ensure that you have actions defined that are shown in the following example. Of course, your arn:aws:iam::… values will be different. The first sample ARN shown below is the account owner’s ID, and the second ARN in this sample is the ID of the AWS S3 storage account.

Again, to get the appropriate storage account ID value, if you haven’t already, use the following GET to retrieve the provider_id of the account that DataStax will use, and for which you’ll grant access permission.

curl --request GET \
 --url 'https://api.astra.datastax.com/v2/kms/provider/<provider>/region/<region>/accounts' \
 --header 'Accept: application/json' \
 --header 'Authorization: Bearer <application_token>'

Make sure that your Key policy here in KMS, again in the section under "Allow an external account to use this KMS key", is using the intended S3 storage account ID’s arn:aws:iam::<storage-account-id>:role/creator value.

{
    "Version": "2012-10-17",
    "Id": "key-consolepolicy-3",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::111111111111:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        },
        {
            "Sid": "Allow an external account to use this KMS key",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::222222222222:role/creator"
            },
            "Action": [
                "kms:EnableKey",
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:GenerateDataKey",
                "kms:GenerateDataKeyWithoutPlaintext",
                "kms:ReEncryptTo",
                "kms:ReEncryptFrom",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Allow attachment of persistent resources",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::222222222222:role/creator"
            },
            "Action": [
                "kms:CreateGrant",
                "kms:ListGrants",
                "kms:RevokeGrant"
            ],
            "Resource": "*"
        }
    ]
}

In your edits, match the kms:… values in the Action:… section, under the Sid label that begins with "Allow an external account to use this KMS key". Ensure that your Key policy has the following actions defined. To use the KMS CMK with your Astra DB data, DataStax requires these eight actions:

            "Action": [
                "kms:EnableKey",
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:GenerateDataKey",
                "kms:GenerateDataKeyWithoutPlaintext",
                "kms:ReEncryptTo",
                "kms:ReEncryptFrom",
                "kms:DescribeKey"
            ]

You can copy/paste in those values (if not present already) into your KMS CMK’s Key policy textbox.

If you have questions, please contact us at Astra DB Product Management.

Also, in the Key policy section that starts with:

            "Sid": "Allow attachment of persistent resources",

Be sure to use the three actions shown in the sample below:

"Action": [
                "kms:CreateGrant",
                "kms:ListGrants",
                "kms:RevokeGrant"
            ],

When you are ready, click Finish.
On the AWS KMS Customer managed keys page, notice the alias name that you defined for your newly created CMK and its ID value.

Copy the ARN of the key you just created. That ARN can serve as the keyID that you’ll provide in:

Astra Portal’s Add Keys UI of your organization’s Security Settings
Or in the DevOps API /v2/kms (or functionally equivalent /v2/kms/serverless) POST request payload

DevOps v2 API to manage Customer Keys

Submit DevOps API calls to create and view Customer Keys in your Astra DB organization. If you are using Postman for your API calls, ensure you use the raw option to enter the body of your API call.

For descriptions of the /v2/kms calls, see the sections below. For more, see the DevOps v2 API reference details.

By default, a POST /v2/kms call registers a new Customer Key for an Astra DB Serverless database. A POST v2/kms/serverless call performs the same request.

If you’re using Astra DB Classic on AWS, submit the POST payload to /v2/kms/classic.

To retrieve existing, registered keys used with Astra DB Serverless databases on AWS, use GET /v2/kms or /v2/kms/serverless.

For Astra DB Classic databases on AWS, use GET /v2/kms/classic.

Create a Customer Key for Astra DB Serverless

Use the following POST to create a Customer Key in Astra DB Serverless that will be associated with an existing Customer Managed Key in the AWS Key Management Service (KMS).

In the request payload, specify your Astra DB organizationID, and the keyID of the Customer Managed Key that you created in AWS KMS, and the region. In each Astra DB organization, there can be one Customer Key for a given cloud provider and region combination. A successful Customer Key creation returns 200 OK in the response.

When you log into Astra Portal, your organization ID is in the generated URL. For example, in the URL https://astra.datastax.com/org/a99999c7-b934-436c-9999-9999999a3b5d/manage, the organization ID is a99999c7-b934-436c-9999-9999999a3b5d.

cURL command (/v2/kms or /v2/kms/serverless)
Result

curl --request POST \
  --url 'https://api.astra.datastax.com/v2/kms' \
  --header 'Accept: application/json' \
  --header 'Authorization: Bearer <application_token>' \
  --data '{
  "orgId": "<organization-id>",
  "aws": {
    "keyID": "String - Resource name of the key",
    "region": "String - Region in which the key exists"
  }
}'

200 OK

Or one of the following:

400 Bad request.
403 The user is forbidden to perform the operation.
404 The specified resource was not found.
500 A server error occurred.

{
  "errors": [
    {
      "description": "string",
      "internalCode": "string",
      "internalTxId": "string"
    }
  ]
}

Get all Customer Keys in an Astra DB Serverless organization

Retrieve all existing Customer Keys of an Astra DB Serverless organization.

cURL command (/v2/kms or /v2/kms/serverless)
Result

curl --request GET \
 --url 'https://api.astra.datastax.com/v2/kms/' \
 --header 'Accept: application/json' \
 --header 'Authorization: Bearer <application_token>'

When one or more keys are found for the organization, 
the 200 OK response contains, for each key:

[
  {
    "OrganizationID": "String - The ID of your organization",
    "CloudProvider": "aws",
    "KeyID": "String - The ID of the Customer Key",
    "Region": "String - The region in which the Customer Key is used"
  }
]

If there are no keys setup for the organization, 
the 404 response contains:

{
  "description": "Organization: <orgID> does not have 
                  any CMKs associated with it."
}

Other potential response codes:

400 Bad request.
403 The user is forbidden to perform the operation.
500 A server error occurred.

Get a single Customer Key in an Astra DB Serverless organization

Retrieves a Customer Key in an Astra DB Serverless organization for a given cloud provider & region combination. The provider value in this case should be aws.

cURL command (/v2/kms or /v2/kms/serverless)
Result

curl --request GET \
 --url 'https://api.astra.datastax.com/v2/kms/provider/aws/region/<region>' \
 --header 'Accept: application/json' \
 --header 'Authorization: Bearer <application_token>'

When the key is found for the given provider and region, 
the 200 OK response contains:

{
  "orgId": "The ID of your organization",
  "provider": "aws": 
  { 
    "keyID": "The ID of the provider account where DataStax stores your data buckets",
    "region": "region-name"
  }
}

If there are no keys setup for the organization's provider 
and region combination, the 404 response contains:

{
  "description": "Could not find <provider> data in 
                  region <region> for organization <orgID>."
}

Other potential response codes:

400 Bad request.
403 The user is forbidden to perform the operation.
500 A server error occurred.

Get an array of Astra DB Serverless Storage Cloud provider accounts

Retrieves a Customer Key in an Astra DB Serverless organization for a given cloud provider & region combination. The provider value in this case should be aws.

The GET response also returns the provider ID of the storage account where your Amazon S3 buckets exist. If not done already, you can use this information to configure or modify (back in the AWS Key Management Service) your KMS Customer Managed Key, so that Astra DB has access to the key.

cURL command (/v2/kms or /v2/kms/serverless)
Result

curl --request GET \
 --url 'https://api.astra.datastax.com/v2/kms/provider/aws/region/<region>/accounts' \
 --header 'Accept: application/json' \
 --header 'Authorization: Bearer <application_token>'

When the key is found for the given provider and region, 
the 200 OK response contains an array of:

[
  {
    "organization_id": "ID of your organization",
    "provider_id": "ID of the provider account
                   where DataStax stores your data buckets",
    "provider": "aws"
  }
]

If there are no keys setup for the organization's provider 
and region combination, the 404 response contains:

{
  "description": "Could not find <provider> data in 
                  region <region> for organization <orgID>."
}

Other potential response codes:

400 Bad request.
403 The user is forbidden to perform the operation.
500 A server error occurred.