Add an alert for agent issues

DSE OpsCenter can alert the operator when certain conditions are met. For example, you can be alerted when a node is down and latency is too high for too long.

Alerts can be sent by email, SNMP, and HTTP requests.

Notable alerts for cluster monitoring

You can use metrics data to create alerts.

If you aren’t sure which metrics to monitor or the alert thresholds to use, the following list can help you get started with effective cluster monitoring.

These recommendations assume you are monitoring a production cluster.

Metric Trigger Notification severity

Metric	Trigger	Notification severity
Node Down	When a node is marked as down by DSE OpsCenter for more than a given duration. Recommendation: Set the duration to `immediately` or `1 minute`, depending on your system’s level of tolerance for down nodes.	Urgent
Agent Issue	When a monitored DataStax Agent experiences issues for more than a given duration. Recommendation: Set the duration to `30 minutes`.	Low
CPU Usage	When CPU usage exceeds a given threshold for a given duration. Recommendation: Notify when above `100` for `1 hour`.	Low
Load	When the total amount of work being processed exceeds a given threshold for a given duration. Recommendation: Notify when above `0.7` for `1 hour`. This ensures you are notified when long-lasting workload spikes occur. If this level of load continues for too long without scaling up the cluster, unacceptable performance degradation can occur.	High
Write Request Latency (percentiles)	When the response time for successful write operations exceeds a given latency threshold for a given duration. Only monitors a specific percentile of write operations. Recommendation: Notify when writes in the 99th percentile are above `n` milliseconds per operation (ms/op) for `4 hours`. The value of `n` depends on your SLA targets for write request latency. This alert monitors the latency of write requests at the specified percentile only. Monitoring all percentiles is noisy because lower percentiles inherently have lower latency. Use other alerts to monitor latency at lower percentiles or capture excessive numbers of slow writes.	Medium
Read Request Latency (percentiles)	When the response time for successful read operations exceeds a given latency threshold for a given duration. Only monitors a specific percentile of read operations. Recommendation: Notify when reads in the 99th percentile are above `n` milliseconds per operation (ms/op) for `4 hours`. The value of `n` depends on your SLA targets for read request latency. This alert monitors the latency of read requests at the specified percentile only. Monitoring all percentiles is noisy because lower percentiles inherently have lower latency. Use other alerts to monitor latency at lower percentiles or capture excessive numbers of slow reads.	Medium
Disk Usage(%)	When disk usage for a disk partition exceeds a given threshold for a given duration. Recommendation: Notify when above `50` percent for `4 hours`. This metric is listed under Advanced > System.	Medium
TBL: SSTables per Read (percentiles)	When the number of SSTables accessed during reads exceeds a given threshold for a given duration. Only monitors SSTable access for a specific percentile of reads for a specific table. Recommendation: For a specific table, notify when reads in the 99th percentile are above `10` SSTables for `1 day`. To monitor multiple tables, create separate alerts for each table. At minimum, monitor the tables that are most important to your workloads. Monitoring all tables might be noisy and not useful. This metric is listed under Advanced > Tables.	Low
TBL: Tombstones per Read (percentiles)	When the number of tombstones accessed during reads exceeds a given threshold for a given duration. Only monitors tombstone access for a specific percentile of reads for a specific table. Recommendation: For a specific table, notify when reads in the 99th percentile are above `n` tombstones for `1 day`. Set `n` to the value of `tombstone_warn_threshold` in your cluster’s `cassandra.yaml`. To monitor multiple tables, create separate alerts for each table. At minimum, monitor the tables that are most important to your workloads. Monitoring all tables might be noisy and not useful. This metric is listed under Advanced > Tables.	Low
TBL: Partition Size (percentiles)	When the size of partitions for a table exceeds a given threshold for a given duration. Only monitors partition size for a specific percentile for a specific table. Recommendation: For a specific table, notify when operations in the 99th percentile are above `200 MB` (in bytes) for `1 day`. To monitor multiple tables, create separate alerts for each table. At minimum, monitor the tables that are most important to your workloads. Monitoring all tables might be noisy and not useful. This metric is listed under Advanced > Tables.	Low

Node Down

When a node is marked as down by DSE OpsCenter for more than a given duration.

Recommendation: Set the duration to immediately or 1 minute, depending on your system’s level of tolerance for down nodes.

Urgent

Agent Issue

When a monitored DataStax Agent experiences issues for more than a given duration.

Recommendation: Set the duration to 30 minutes.

Low

CPU Usage

When CPU usage exceeds a given threshold for a given duration.

Recommendation: Notify when above 100 for 1 hour.

Low

Load

When the total amount of work being processed exceeds a given threshold for a given duration.

Recommendation: Notify when above 0.7 for 1 hour.

This ensures you are notified when long-lasting workload spikes occur. If this level of load continues for too long without scaling up the cluster, unacceptable performance degradation can occur.

High

Write Request Latency (percentiles)

When the response time for successful write operations exceeds a given latency threshold for a given duration. Only monitors a specific percentile of write operations.

Recommendation: Notify when writes in the 99th percentile are above n milliseconds per operation (ms/op) for 4 hours.

The value of n depends on your SLA targets for write request latency.

This alert monitors the latency of write requests at the specified percentile only. Monitoring all percentiles is noisy because lower percentiles inherently have lower latency.

Use other alerts to monitor latency at lower percentiles or capture excessive numbers of slow writes.

Medium

Read Request Latency (percentiles)

When the response time for successful read operations exceeds a given latency threshold for a given duration. Only monitors a specific percentile of read operations.

Recommendation: Notify when reads in the 99th percentile are above n milliseconds per operation (ms/op) for 4 hours.

The value of n depends on your SLA targets for read request latency.

This alert monitors the latency of read requests at the specified percentile only. Monitoring all percentiles is noisy because lower percentiles inherently have lower latency.

Use other alerts to monitor latency at lower percentiles or capture excessive numbers of slow reads.

Medium

Disk Usage(%)

When disk usage for a disk partition exceeds a given threshold for a given duration.

Recommendation: Notify when above 50 percent for 4 hours.

This metric is listed under Advanced > System.

Medium

TBL: SSTables per Read (percentiles)

When the number of SSTables accessed during reads exceeds a given threshold for a given duration. Only monitors SSTable access for a specific percentile of reads for a specific table.

Recommendation: For a specific table, notify when reads in the 99th percentile are above 10 SSTables for 1 day.

To monitor multiple tables, create separate alerts for each table. At minimum, monitor the tables that are most important to your workloads. Monitoring all tables might be noisy and not useful.

This metric is listed under Advanced > Tables.

Low

TBL: Tombstones per Read (percentiles)

When the number of tombstones accessed during reads exceeds a given threshold for a given duration. Only monitors tombstone access for a specific percentile of reads for a specific table.

Recommendation: For a specific table, notify when reads in the 99th percentile are above n tombstones for 1 day.

Set n to the value of tombstone_warn_threshold in your cluster’s cassandra.yaml.

To monitor multiple tables, create separate alerts for each table. At minimum, monitor the tables that are most important to your workloads. Monitoring all tables might be noisy and not useful.

This metric is listed under Advanced > Tables.

Low

TBL: Partition Size (percentiles)

When the size of partitions for a table exceeds a given threshold for a given duration. Only monitors partition size for a specific percentile for a specific table.

Recommendation: For a specific table, notify when operations in the 99th percentile are above 200 MB (in bytes) for 1 day.

To monitor multiple tables, create separate alerts for each table. At minimum, monitor the tables that are most important to your workloads. Monitoring all tables might be noisy and not useful.

This metric is listed under Advanced > Tables.

Low

Create an alert

The following example creates an alert to monitor DataStax Agent issues:

Click the Alerts menu.

If any alerts are active, the Alerts menu item shows the number of active alerts.
In the Active Alerts dialog, click Manage Alerts.
For Notify me when, select Agent Issue.
Specify how long the condition must be present before triggering the alert.
For Notify every, select how frequently to send notifications once the alert is triggered.

This is a reflection of the alert’s importance. For the most critical alerts, select a more frequent notification interval, such as Urgent or High.
Click Save Alert.
If the alert is triggered, investigate the issue in the Agents View.

Create an alert for a percentile

See Configure an alert for percentiles.

Add an alert for agent issues

Notable alerts for cluster monitoring

Create an alert

Create an alert for a percentile

Was this helpful?

Give Feedback