Planning a DSE cluster on Amazon EC2
This topic provides information about deploying a production DataStax Enterprise cluster on Amazon EC2.
Before planning an Amazon Elastic Computer Cloud (Amazon EC2) cluster, read Amazon EC2 - Virtual Server Hosting.
|
These recommendations are guidelines:
|
Use AMIs from trusted sources
Use only AMIs for supported platforms and from a trusted source. Random AMIs pose a security risk and may perform slower than expected due to the way the EC2 install is configured. The following are examples of trusted AMIs:
EC2 deployments for multiple regions/availability zones
For these deployments use any of the supported platform on each node.
It is best practice to use the same platform on all nodes. If your cluster is instantiated using the DataStax AMI (no longer supported), use Ubuntu for the additional nodes. Configure the cluster as a multiple datacenter cluster using the Configuring Amazon EC2 multi-region snitch.
Guidelines for EC2 production clusters
DSE requires 10,000 IOPS (Input/Ouput Operations Per Second) minimum per node. The AWS storage choices for achieving this performance level are:
-
EBS General Purpose SSD (gp2) volumes
To achieve IO required by DSE, you must use 3.5TB volumes, regardless of the actual space used, because gp2 provides 3 IOPS per GB.
-
Amazon EBS Provisioned IOPS SSD (io1) volumes
EBS io1 with 10,000 provisioned IOPS (PIOPS) provides the same performance level as gp2 using smaller volumes but at a higher cost.
-
Directly attached local SSDs
Also called ephemeral or instance SSDs. This storage type makes i3 instances the optimal cost versus performance choice. See pricing in Amazon AWS.
Use these guidelines for choosing the instance types:
-
Light production with only transactional nodes and very light-weight usage: m4.2xlarge (absolute minimum). Also suitable for development.
-
Moderate production: i3.4xlarge
-
Large production: i3.8xlarge
-
DSE Search and DSE Analytic nodes: i3.4xlarge or i3.8xlarge
|
In EC2, each vCPU is a hyperthread of an Intel Xeon core, which means that two virtual cores exist on one physical core. For example, an i3.8xlarge instance type has 32vcCPUs, which is the equivalent of 16 physical cores. |
EBS volumes recommended for M4 instance types
SSD-backed general purpose volumes (GP2) or provisioned IOPS volumes (io1) are suitable for production workloads when using m4 instances (transactional nodes with very light-weight usage). These volume types are designed to deliver consistent, low latency performance:
| GP2 | PIOPS |
|---|---|
|
|
EBS magnetic volumes not recommended
EBS magnetic volumes are not recommended for DSE data storage volumes for the following reasons:
-
EBS magnetic volumes contend directly for network throughput with standard packets. This contention means that EBS throughput is likely to fail when a network link is saturated.
-
EBS magnetic volumes have unreliable performance. I/O performance can be exceptionally slow, causing the system to back load reads and writes until the entire cluster becomes unresponsive.
-
Adding capacity by increasing the number of EBS volumes per host does not scale. You can easily surpass the ability of the system to keep effective buffer caches and concurrently serve requests for all of the data it is responsible for managing.
|
Use only ephemeral instance-store or the recommended EBS volume types for data storage. |
For more information and graphs related to ephemeral versus EBS performance, see a Systematic Look at EC2 I/O.
Disk performance optimization
To ensure high disk performance to mounted drives, it is recommended that you pre-warm your drives. Write once to every drive location before production use. Depending on EC2 conditions, you can get moderate to enormous increases in throughput. See Optimizing Disk Performance in the Amazon Elastic Compute Cloud Documentation.
Storage recommendations
DSE supports JBOD (just a bunch of disks). JBOD excels at tolerating partial failures in a disk array. Configure the disk_failure_policy in cassandra.yaml. See Recovering from a single disk failure using JBOD.
|
JBOD support allows you to use standard disks. However, RAID0 may provide better throughput because it splits every block to be on another device so that writes are written in parallel fashion instead of written serially on disk. |
EC2 security group
When deploying a cluster on EC2, you must create security rules that open ports to other nodes in the same security group. An EC2 Security Group acts as a firewall that allows you to choose which protocols and ports are open in your cluster. You can specify the protocols and ports either by a range of IP addresses or by security group. For more information, see the Amazon EC2 help on Security Groups.
|
Specifying a Source IP of 0.0.0.0/0 opens externally accessible ports to incoming traffic from any IP address. The risk of data loss is high. |
See the list of ports that should be opened to internode and client communications in the DSE ports table.
|
Generally, when you have firewalls between machines, it is difficult to run JMX across a network and maintain security. This is because JMX connects on port 7199, handshakes, and then uses any port within the 1024+ range. Instead use SSH to execute commands remotely to connect to JMX locally or use the DSE OpsCenter. |
