Bulk uploading S3 backups using the AWS CLI 

Use the AWS CLI instead of the AWS SDK when bulk loading backups to Amazon S3 locations. Using the S3 CLI is a Labs feature that must be enabled.

address.yaml

The location of the address.yaml file depends on the type of installation:

  • Package installations: /var/lib/datastax-agent/conf/address.yaml
  • Tarball installations: install_location/conf/address.yaml

cluster_name.conf 

The location of the cluster_name.conf file depends on the type of installation:

  • Package installations: /etc/opscenter/clusters/cluster_name.conf
  • Tarball installations: install_location/conf/clusters/cluster_name.conf

Use the AWS CLI instead of the AWS SDK when bulk loading backups to Amazon S3 locations. Using the AWS CLI rather than the AWS SDK can result in a performance increase, with a noticeable decrease in the time it takes to complete a backup. This is an OpsCenter Labs feature (that is, under ongoing development but available for use). The feature is available in OpsCenter versions 6.1.3 and later.

For more information, see AWS CLI in the Amazon documentation.

Note: When the AWS S3 CLI is enabled, the S3 throttling setting is ignored by OpsCenter during backups. See Tuning throttling for AWS CLI.

Prerequisites

Add an S3 location for backups.

Procedure

  1. Open cluster_name.conf for editing. Substitute cluster_name with the name of your cluster. Setting agent options through the cluster configuration file sets the corresponding property in address.yaml on every node.
    If necessitated by your environment, open address.yaml for editing and configure at the node level. Do so for every node that requires a specific configuration override.
  2. Add the following configuration option:
    [labs]
    use_s3_cli = True
  3. Save the configuration file or files.
  4. Restart the OpsCenter daemon.
  5. Optional: If you made changes to address.yaml, restart the DataStax agents.

What's next

Tune throttling for AWS CLI