• Glossary
  • Support
  • Downloads
  • DataStax Home
Get Live Help
Expand All
Collapse All

DataStax Enterprise OpsCenter 6.8

    • About OpsCenter
      • New features
      • Key features
      • Labs features
        • Exporting and importing dashboard presets
        • Adding a Swift CLI backup location
        • Configuring named route linking
        • Viewing logs from node details
      • Architecture overview
      • OpsCenter policy for DDAC and OSS
      • Feedback about OpsCenter
    • Release notes
    • Installing OpsCenter
    • Upgrading OpsCenter
    • OpsCenter recommended settings
      • OpsCenter basic configurations
      • Cluster synchronization settings
      • Backup Service settings
      • Knowledge Base articles
    • Configuring OpsCenter
      • OpsCenter Security
        • OpsCenter SSL overview
          • Enabling/Disabling HTTPS for the OpsCenter server
          • Configuring SSL/TLS between OpsCenter and the DataStax Agents
          • Connect to DSE with client-to-node encryption in OpsCenter and the DataStax Agents
          • Editing/OpsCenter cluster connections for authentication or encryption
          • SSL configuration options for OpsCenter
        • Configuring OpsCenter role-based security
        • Encrypting sensitive configuration values
          • Activating configuration encryption
          • Creating a system key to encrypt sensitive configuration values
          • Manually encrypting a configuration value
          • Deactivating configuration encryption
        • Authenticating with LDAP
          • Configuring LDAP
          • Adding a role for an LDAP user
          • Troubleshooting OpsCenter LDAP
        • Kerberos authentication
          • Configuring OpsCenter for Kerberos authentication
          • OpsCenter Kerberos configuration options
          • Troubleshooting Kerberos in OpsCenter
        • Configuring security logging
      • Configuring alerts for events
        • SNMP alerts overview
          • Enabling SNMP alerts
        • Enabling SMTP email alerts
        • Enabling alerts posted to a URL
          • Verifying that events are posting correctly
          • Posting URL alerts to a Slack channel
      • Configuring data collection and expiration
        • Controlling data collection
        • Storing collection data on a separate cluster
      • OpsCenter DSE definitions files updates
        • Updating and configuring definitions files properties
      • Automatic failover overview
        • Enabling automatic failover
        • Failover configuration options reference
      • Backing up critical configuration data
      • Configuring named route linking
      • Configuring the OpsCenter JVM
      • Configuring the DataStax Agent JVM
        • Setting and securing the tmp directory for the DataStax Agent
        • Encrypting JMX communications
      • Changing the replication strategy for the OpsCenter keyspace
      • Configuration files for OpsCenter
        • OpsCenter configuration properties
          • Statistics reporter properties
        • Cluster configuration properties
          • Cassandra connection properties
          • Metrics Collection Properties
        • DataStax Agent configuration
        • OpsCenter logback.xml configuration
      • Customize scripts for starting and stopping DataStax Enterprise
      • Example configuration scenarios
        • Configuring for multiple regions
        • Configuring for very large clusters
    • Using OpsCenter
      • OpsCenter workspace overview
        • Ring View
        • List View
        • DataStax Agents Status View
        • Nodes Detail View
          • Node management operations
      • Configuring alerts
        • Adding an alert for agent issues
          • Troubleshooting DataStax Agent Issues
        • Adding an alert for down nodes
        • Configuring an alert for KMIP errors
        • Configuring an alert for percentage of in-memory usage
        • Configuring an alert for percentiles
      • Monitoring node operations
        • Viewing the Spark Console
        • Monitoring in-memory usage
        • Viewing logs from node details
      • Managing and maintaining nodes
        • Running cleanup
        • Performing garbage collection
        • Running compaction
        • Flushing tables
        • Decommission a node
        • Draining a node
        • Moving a node
        • Running a manual repair
        • Configure an alias for a node
      • Starting and stopping DSE
        • Starting DSE on a node
        • Stopping DSE on a node
        • Restarting DSE on a node
      • Managing keyspaces and tables
        • Keyspaces
          • Managing a keyspace
          • Managing tables
        • Browsing data deprecated
      • Cluster administration
        • Adding an existing cluster
        • Disconnecting a cluster from OpsCenter and Lifecycle Manager
        • Rebalancing a cluster overview
          • Rebalancing a cluster
          • Configuring an alert for rebalancing a cluster
        • Restarting a cluster
        • Changing the display name of a cluster
        • Downloading diagnostic data
          • Diagnostic tarball reference
          • Creating an alternate directory for diagnostic information
        • Downloading Insights diagnostic data
        • Generating a cluster report
      • OpsCenter Metrics Tooltips Reference
        • Dashboard performance metrics
        • Performance metrics overview
          • Working with metrics performance graphs
          • Organizing performance metrics presets
          • Exporting and importing dashboard presets
        • Cluster performance metrics
        • Pending task metrics
          • Pending task metrics for writes
          • Pending task metrics for reads
          • Pending task metrics for cluster operations
        • Table performance metrics
        • Tiered storage performance metrics
          • Configuring tiered storage metric graphs
          • Configuring tiered storage alerts
        • Message latency metrics
          • Adding dashboard graphs for datacenter and node messaging latency
          • Adding alerts for DC and node message latency
        • Search performance metrics
        • Graph metrics
        • NodeSync metrics
        • Thread Pool (TP) metrics
          • Viewing TP stats in Node Details
          • Enabling network backpressure
        • Dropped Messages metrics
        • Operating system performance metrics
        • Alert metrics
          • Advanced system alert metrics
    • OpsCenter 6.8 Reference
      • OpsCenter ports reference
      • Installation and configuration locations
        • Default file locations for package installations
        • Default file locations tarball installations
      • Starting, stopping, and restarting OpsCenter
        • Startup log for OpsCenter
      • Stopping, starting, and restarting DataStax Agents
    • DSE Management Services
      • Backup Service
        • Quick Video Tour: Backup Service
        • Adding a backup location
          • Adding a local file system backup location
          • Adding an Amazon S3 backup location
          • Adding an Azure backup location
        • Backing up data
          • Backing up a cluster
          • Backing up to Amazon S3
          • Backing up and restoring DataStax Graphs in OpsCenter
          • Viewing backup and restore history
          • Synchronizing backup data after an upgrade
          • Deleting backup data
        • Restoring a cluster
          • Restoring from a backup
          • Restoring a backup to a specific point-in-time
          • Monitoring sufficient disk space for restoring backups
        • Cloning cluster data
          • Cloning cluster data from a defined other location
          • Cloning cluster data from clusters managed by the same OpsCenter instance
        • Configuring the Backup Service
          • Configuring commit log backups
          • Configuring the free disk space threshold for backups
          • Configuring encryption key storage for backups
          • Configuring custom scripts to run before and after backups
          • Configuring restore to continue after a download failure
          • Backup Service configuration options
        • Troubleshooting Backup Service errors
      • NodeSync Service
        • Enabling NodeSync
        • Configuring the NodeSync refresh data interval
        • Viewing NodeSync Status
        • Configuring the NodeSync rate using LCM
        • NodeSync metrics
      • Repair Service
        • Repair Service overview
          • Subrange repairs overview
          • Distributed subrange overview
          • Incremental repairs overview
          • Repair Service behavior during environment changes
          • Estimating remaining repair time
        • Turning the Repair Service on
        • Turning the Repair Service off
        • Viewing repair status
        • Basic repair configuration
          • Configuring incremental repairs
          • Excluding keyspaces or tables from subrange repairs
          • Enabling distributed subrange repairs
          • Logging for the Repair Service
          • Basic Repair Service configuration reference
        • Advanced repair configuration
          • Adjusting or disabling the throttle for subrange repairs
          • Running validation compaction sequentially
          • Advanced Repair Service configuration reference
        • Expert repair configuration
          • Setting the maximum for parallel subrange repairs
          • Expert Repair Service configuration reference
          • Tuning Repair Service for multi-datacenter environments
        • Expedited Repair Service configuration
        • Troubleshoot Repair Service errors
        • Learn more about repairs
      • Capacity Service
        • Forecasting trends for metric graphs
        • Advanced forecast configuration
      • Best Practice Service
        • Configuring Best Practice service rules
        • Monitoring the results of Best Practice service scans
        • Best Practice Rules Reference
      • Performance Service
        • Performance Service Overview
        • Why use the OpsCenter Performance Service?
        • Enabling the OpsCenter Performance Service
        • Disabling the OpsCenter Performance Service
        • Setting permissions for the OpsCenter Performance Service
        • Tuning a database cluster with the Performance Service
          • Identifying and tuning slow queries
    • Identifying poorly performing tables
    • Monitoring node thread pool statistics
    • Troubleshooting OpsCenter
    • Lifecycle Manager
      • Overview of Lifecycle Manager
        • Supported capabilities
        • Defining the topology
        • Using configuration profiles
        • Defining repositories
        • Running jobs in LCM
          • Job types in LCM
          • Job concurrency in LCM
        • Monitoring job status
      • Installing DSE using LCM
        • Accessing OpsCenter Lifecycle Manager
        • Creating custom data directories
        • Adding SSH credentials
        • Adding a configuration profile
        • Adding a repository
        • Defining the cluster topology
          • Adding a cluster
          • Adding a datacenter
          • Adding a node
        • Running an installation job
        • Viewing job details
        • Using LCM in an offline environment
          • Required software for offline DSE installs
          • Downloading DSE in an offline environments
      • Managing SSH credentials
        • Adding SSH credentials
        • Editing SSH credentials
        • Deleting SSH credentials
        • Configuring SSH connection thresholds for LCM jobs
      • Managing configuration profiles
        • Adding a configuration profile
        • Editing a configuration profile
        • Customizing configuration profile files
        • Cloning a configuration profile
        • Deleting a configuration profile
        • Configuring an HTTP or HTTPS proxy
      • Configuring repositories
        • Adding a repository
        • Editing a repository
        • Deleting a repository
      • Defining DSE topologies
        • Managing cluster topologies
          • Adding a cluster
          • Editing a cluster
          • Deleting a cluster
          • Importing a cluster topology
        • Managing datacenter topologies
          • Adding a datacenter
          • Editing a datacenter
          • Deleting a datacenter
        • Managing node topologies
          • Adding a node
          • Editing a node
          • Deleting a node
      • Running LCM jobs
        • Running an installation job
        • Running an configure job
        • Running an upgrade job
          • Example: Upgrading DSE to a minor release using LCM
        • Aborting a job
        • Adjusting idle timeout
      • Configuring Java options
        • Choosing a Java vendor in LCM
        • Managing Java installs
        • Configuring JVM options for DSE using LCM
      • Configuring DSE security using LCM
        • Native transport authentication schemes and limitations in LCM
          • Configuring row-level access control
        • Configuring SSL/TLS for DSE
        • Configuring a JMX Connection to DSE
      • Lifecycle Manager configuration options
      • Configuration known issues and limitations
      • Using advanced configurations with LCM
        • Exporting metrics collection
        • Configuring AlwaysOn SQL
        • Configuring DSE Graph
        • Configuring the NodeSync rate
        • Configuring tiered storage
    • OpsCenter API reference for developers
      • Enable and access the Datastax Agent API
      • Authentication
      • OpsCenter configuration
      • Retrieving cluster and node information
      • Performing Cluster Operations
      • Managing Keyspaces and Tables
      • Retrieving Metric Data
      • Managing Events and Alerts
      • Schedule management
      • Backup Management and Restoring from Backups
      • Best Practice Rules
      • Hadoop
      • Spark
      • Managing Performance Service Configuration
      • User Interface
      • Agent Install and Status
      • Cluster Lifecycle Management
      • DataStax Agent API example curl commands
  • DataStax Enterprise OpsCenter 6.8
  • DSE Management Services
  • Backup Service
  • Adding a backup location
  • Adding an Amazon S3 backup location

Adding an Amazon S3 backup location

Set a retention policy for the backup location.

About this task

Add an Amazon S3 or S3-compatible backup location. For more details, see backing up to Amazon S3 and the Amazon S3 transfer acceleration documentation.

Moving backup files from Amazon S3 to Amazon Glacier is not supported by the OpsCenter Backup Service.

The Backup Service requires control over the data and structure of its destination locations. The backup destinations must be dedicated for use only by OpsCenter. Any additional directories or files in those destinations can prevent the Backup Service from properly conducting a Backup or Restore operation.

Prerequisites

  • Locate the cluster_name.conf configuration file. The location of this file depends on the type of installation:

    • Package installations: /etc/opscenter/clusters/cluster_name.conf

    • Tarball installations: install_location/conf/clusters/cluster_name.conf

  • Ensure Java 8 is installed on the same machine where DataStax Agents are running. Agents require Java 8 to store at an S3 location.

  • Make sure you have the proper AWS IAM privileges for the AWS account that the S3 bucket is linked to.

  • Ensure that the selected Amazon S3 bucket meets the Amazon S3 bucket requirements.

The AWS credentials and bucket names are stored in cluster_name.conf (with the exception of ad-hoc backups). Be sure to use proper security precautions to ensure that this file is not readable by unauthorized users.

Procedure

  1. Access the Create (or Edit) Backup dialog:

  2. In the Create or Edit Backup dialog, under Location, click +Add Location.

    The Add Location dialog appears.

    Add Location dialog S3 location with Retention Policy for scheduled backups

  3. Select Amazon S3 or S3 Compatible as the backup Location.

  4. Enter the location of the S3 bucket so that OpsCenter can locate it.

    Option Description

    Amazon S3

    Enter the Region where the S3 bucket is located.

    If blank, OpsCenter will try to query S3 for the bucket region or use the remote_backup_region as a default.

    Some regions, such as China (Beijing), require a region to be specified and cannot be queried.

    S3 Compatible

    Enter a URL that points to an S3 Endpoint.

    For example, mys3endpoint:9000.

  5. Enter the S3 Bucket name.

    The bucket name must be at least 4 characters long. Bucket names must only contain lowercase letters, numbers, and hyphens. Additionally, OpsCenter requires that bucket prefixes contain only lowercase letters, numbers, and safe characters. See the S3 guidelines for more details about bucket naming restrictions.

    To indicate a bucket subfolder location, delineate the bucket name from the folder name with a forward slash (/) character.

    Example: mybucket/myfolder/mysubfolder. Remember that slashes are not allowed within bucket or folder names themselves.

  6. Select the source type of your AWS credentials.

    The AWS credentials and bucket names are stored in cluster_name.conf (with the exception of ad hoc backups). Be sure to use proper security precautions to ensure that this file is not readable by unauthorized users.

    Option Description

    User-Supplied Credentials

    Enter your AWS Key and AWS Secret.

    AWS Credential Provider chain

    Use the default credential provider chain to locate AWS credentials. See Working with AWS Credentials on the AWS website.

  7. Select any throttling, compressing, encryption, or acceleration of the data:

    For S3 Compatible backups, Throttle S3 transfer rate is the only option available.

    1. To avoid saturating your network, set a maximum upload rate. Select Throttle transfer rate and set the maximum MB per second.

      When the AWS CLI feature is enabled, the S3 throttle is ignored. A tooltip also mentions this current limitation. See Tuning throttling when using AWS CLI.

    2. To compress the backup data, select Enable compression. Compression reduces the amount of data going through your network and reduces the disk and data usage but increases the CPU load for the server.

    3. To enable server-side S3 encryption, select Enable S3 server-side encryption. Enabling server-side encryption increases the security of your backup files, but increases the time it takes to complete a backup. For more information on S3 server-side encryption, see Using Server Side Encryption on the AWS website.

      Choose the type of encryption you want to use:

      Option Description

      256-bit Advanced Encryption Standard

      SSE-S3 encryption encrypts each file in the backup set with a unique key, including the key itself, using a 256-bit AES cypher.

      KMS Managed Encryption

      SSE-KMS encryption uses customer master keys (CMKs) to encrypt Amazon S3 objects.Enter a KMS Key ID that is associated with your AWS account.

    4. To back up nodes running in multiple regions to a single bucket, select Enable S3 transfer acceleration. Instead of traffic crossing over the internet, acceleration mode uses Amazon CloudFront to cache S3 requests. Because the CloudFront servers are closer to the nodes in each region, the backup latency is reduced.

      Enabling S3 transfer acceleration can cause performance degradation, and might slow a standard backup configuration. Use this option only if backing up nodes in multiple regions to a single bucket.

  8. Optional: For scheduled backups, indicate how long the snapshot data should be retained by selecting a Retention Policy. Retain All (default) saves the snapshot data indefinitely. Or, define a set period of time. After the snapshot data is older than the time set in Retention Policy, the snapshot data is deleted.

    DataStax strongly recommends setting a retention policy to periodically remove backups. This practice helps to avoid long-term performance issues caused by an excessive number of backups.

    Setting a Retention Policy is not available for an ad hoc (Run Now) backup.

  9. Click Save Location.

    The newly added S3 location displays in the Location pane of the Create or Edit Backup dialog.

    opscBSFSLocalLocationPane

    Click the edit icon to the edit a location and its retention policy if applicable.

    Click the delete icon to delete a location. The On Server location cannot be deleted.

  10. Click Save Backup or Create Backup as applicable.

Bulk uploading S3 backups using the AWS CLI

Using the S3 CLI is a feature that must be enabled.

About this task

Use the AWS CLI instead of the AWS SDK when bulk loading backups to Amazon S3 locations. Using the AWS CLI rather than the AWS SDK can result in a performance increase, with a noticeable decrease in the time it takes to complete a backup.

This feature is available in OpsCenter versions 6.1.3 and later as an OpsCenter Labs feature. As of OpsCenter version 6.5 and later, the AWS CLI feature is officially a production feature.

For more information, see AWS CLI in the Amazon documentation.

When the AWS S3 CLI is enabled, the S3 throttling setting is ignored by OpsCenter during backups. See Tuning throttling for AWS CLI.

Prerequisites

  1. Install the AWS CLI package on every node. DataStax recommends using the Amazon bundled installer method and upgrading to the latest version of AWS CLI if it is already installed. See Install the AWS CLI using the bundled installer in the Amazon documentation for installation procedures.

    As a recommended best practice for OpsCenter, install the AWS CLI bundle using APT as follows:

    sudo apt-get install -y unzip
    curl 'https://s3.amazonaws.com/aws-cli/awscli-bundle.zip' -o awscli-bundle.zip
    unzip awscli-bundle.zip
    sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws

    Regardless of the install procedure used, make sure that the AWS CLI package is installed in the PATH of the cassandra user, or whichever user the DataStax agent runs as.

  2. Add an S3 location for backups.

Procedure

  1. Locate the cluster_name.conf file. The location of this file depends on the type of installation:

    • Package installations: /etc/opscenter/clusters/cluster_name.conf

    • Tarball installations: install_location/conf/clusters/cluster_name.conf

  2. Open cluster_name.conf for editing. Substitute cluster_name with the name of your cluster. Setting agent options through the cluster configuration file sets the corresponding property in address.yaml on every node.

    To configure the setting for all clusters managed by an OpsCenter instance, open opscenterd.conf for editing.

    The location of this file depends on the type of installation:

    • Package installations: /etc/opscenter/opscenterd.conf

    • Tarball installations: install_location/conf/opscenterd.conf

    If necessitated by your environment, open address.yaml for editing and configuring at the node level. The location of this file depends on the type of installation:

    • Package installations: /var/lib/datastax-agent/conf/address.yaml

    • Tarball installations: install_location/conf/address.yaml

Do so for every node that requires a specific configuration override.

  1. Add the following configuration option:

    [backups]
    use_s3_cli = True
  2. Save the configuration file or files.

  3. Restart the OpsCenter daemon.

  4. If you made changes to address.yaml, restart the DataStax agents.

What’s next

Tune throttling for AWS CLI

Tuning throttling when using AWS CLI

About this task

Use alternative throttle options when using the AWS CLI for bulk uploads because the OpsCenter S3 throttle is ignored when the OpsCenter AWS CLI for S3 feature is enabled.

Procedure

  1. Adjust the max_concurrent_requests available in the AWS SDK. Refer to the AWS CLI S3 configuration documentation for details.

  2. If necessary, use a tool such as Trickle to limit bandwidth.

Adding a local file system backup location Adding an Azure backup location

General Inquiries: +1 (650) 389-6000 info@datastax.com

© DataStax | Privacy policy | Terms of use

DataStax, Titan, and TitanDB are registered trademarks of DataStax, Inc. and its subsidiaries in the United States and/or other countries.

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.

Kubernetes is the registered trademark of the Linux Foundation.

landing_page landingpage