Alerts

Mission Control provides alerts to notify you of important events in your cluster. Alerts are triggered by specific conditions, such as a node being down for more than 10 minutes or CPU usage above 80% for 5 minutes.

All alerts contain information about each captured event. Optionally, you can configure Mission Control to send alerts for selected levels of events or specific clusters.

Mission Control provides support for routing alerts to Slack channels. For more information, see Manage alert recipients.

Default alerts

Mission Control includes the following default alters for Hyper-Converged Database (HCD), DataStax Enterprise (DSE), and Apache Cassandra® databases.

A severity label marks the criticality value of an alert.

The three values can be:

critical: Requires immediate action.
warning: Requires eventual but not urgent action.
info: Marks something out of the ordinary that doesn’t necessarily require action.

Default alerts
Description	Severity	Details
Node down for more than 10 minutes	Sev 2 - Warning	Source metric: `org_apache_cassandra_metrics_thread_pools_completed_tasks`.
Node down for 30 minutes	Sev 1 - Error	Source metric: `org_apache_cassandra_metrics_thread_pools_completed_tasks`.
Nodes down in different racks of same datacenter		Two nodes down across rack boundaries can lead to LOCAL_QUORUM CL errors in applications. Source metric: `org_apache_cassandra_metrics_thread_pools_completed_tasks`.
CPU above 80% for 5 minutes		An error that, if triggered too often, indicates low disk space and that the cluster should be scaled. Source metric: `host_cpu_seconds_total`.
Used disk space above 50% for one minute		A signal to expand the cluster before it gets into a state where cleanups are impossible due to insufficient disk space. Source metric: `host_filesystem_used_ratio`.
Used disk space above 75% for one minute	Sev 1 - Error	A signal to expand the cluster before it gets into a state where cleanups are impossible due to insufficient disk space. Source metric: `host_filesystem_used_ratio`.
Used disk space above 50% for one minute	Sev 2 - Warning	A signal to expand the cluster before it gets into a state where cleanups are impossible due to insufficient disk space. Source metric: `host_filesystem_used_ratio`.
Load average above 20 for 5 minutes	Sev 2 - Warning	Good indicator for performance issues, the root cause of which can vary. Source metric: `host_load5`.
Load average above 32 for 5 minutes	Sev 1 - Error	Good indicator for performance issues, the root cause of which can vary. Source metric: `host_load5`.
Dropped messages over 5 minutes	Sev 1 - Error for >= 10,000 + Sev 2 - Warning for < 10,000	Thread pools cannot keep up with the pace of queries entering and being processed within the cluster. This leads to errors within the application stack and potentially incorrect replicas. Source metric: `org_apache_cassandra_metrics_dropped_message_dropped_total`.

Manage alerts

You can view active alerts, create custom alerts, and update alert preferences.

View active alerts

In the Mission Control UI, click notifications Alerts, then click New Alert.

The Active Alerts sidebar displays the active alerts in your cluster.

Create custom alerts

You can create custom alerts to monitor specific events in your cluster. Custom alerts are useful to notify your team of important information for your unique workflows. For example, you might create an alert to monitor the number of dropped messages in your cluster.

Mission Control uses PromQL expressions for custom alerts. For more information, see PromQL expression examples in the Prometheus documentation.

To create a custom alert, do the following:

In the Mission Control UI, click notifications Alerts, then click New Alert.
Enter an Alert Expression.
Define how long the alert condition must exist before the alert is triggered:
1. In the Active for more than field, enter an integer representing an amount of time.
2. Select the units, Minutes or Hours.
Under Labels, select the Severity Value.
(Optional) Select a Group Value.
Enter an Alert Summary and an Alert Description.
(Optional) Enter additional Alert Context.
Click Create Alert.

Update alert preferences

You can update alert preferences to modify or delete notifications for specific events:

In the Mission Control UI, click notifications Alerts, then click Alert Settings.
Select the Alert Preferences tab.
Click more_vert More Options to modify or delete an alert preference.

Manage alert recipients

Mission Control provides support for routing alerts to recipients in Slack channels.

Edit the Slack Webhook URL

In the Mission Control UI, click notifications Alerts, then click Alert Settings.
Select the Notifications tab.
Click edit Edit.
Click Configure Notifications.

View alert notification recipients

In the Mission Control UI, click notifications Alerts, then click Alert Settings.
Select the Notifications tab. The Notifications tab displays the alert notification recipients.
Click keyboard_arrow_down Expand to view recipient details.

Add alert notification recipients

To add alert notification recipients, do the following:

In the Mission Control UI, click notifications Alerts, then click Alert Settings.
Select the Notifications tab.
Click Add Recipient.
Enter a Recipient Name.
Enter a Slack Channel.
(Optional) Enable Set as default recipient.
(Optional) Select Notification Policy to set up a notification policy for the recipient.
1. Select a Key.
2. Select an operator: equal, not equal, regexp, or not regexp.
3. Enter a value to define the policy condition.
4. (Optional) Select Add Routing Label to add additional routing labels.
In the Notify Every field, enter an integer representing an amount of time.
Select the units, Minutes or Hours.
Click Add Recipient.

Edit alert notification recipients

To edit alert notification recipients, do the following:

In the Mission Control UI, click notifications Alerts, then click Alert Settings.
Select the Notifications tab.
Click more_vert More Options for the recipient you want to edit.
Select Modify.
Edit the recipient details.
Click Modify Recipient.

Add a notification policy for an alert recipient

To add a notification policy for an alert recipient, do the following:

In the Mission Control UI, click notifications Alerts, then click Alert Settings.
Select the Notifications tab.
Click Add Notification Policy.
Enter a Recipient Name.
Select a Key.
Select an operator: equal, not equal, regexp, or not regexp.
Enter a value to define the policy condition.
(Optional) Select Add Routing Label to add additional routing labels.
In the Notify Every field, enter an integer representing an amount of time.
Select the units, Minutes or Hours.

Alerts

Default alerts

Manage alerts

View active alerts

Create custom alerts

Update alert preferences

Manage alert recipients

Edit the Slack Webhook URL

View alert notification recipients

Add alert notification recipients

Edit alert notification recipients

Add a notification policy for an alert recipient

Was this helpful?

Give Feedback