Create custom alerting rules
Mission Control provides advanced capabilities for managing custom alerting rules through Helm values.
This guide explains how to create custom alerting rules for Mission Control environments and integrate them with your monitoring workflow.
Prerequisites
-
Mission Control version 1.15.0 or later installed.
-
Access to the Kubernetes cluster where Mission Control is deployed.
-
Basic understanding of Prometheus alerting rules and PromQL.
-
Helm chart access for configuration updates.
How custom alerting rules work
Mission Control separates default and custom alerting rules into different management systems:
-
Default rules: Mission Control automatically manages these rules and updates them during system upgrades.
-
Custom rules: You manage these rules through Helm values, providing advanced configuration options.
This separation provides several benefits:
-
Mission Control preserves your custom rules during updates.
-
You can manage custom rules independently through Helm chart settings.
-
Rules created in the UI are managed separately from Helm-configured rules, allowing you to use both approaches simultaneously.
-
Multiple Mission Control instances can coexist without conflicts.
Create custom alerting rules
Create custom alerting rules by configuring them in your Helm values.
Custom alerting rules follow the Prometheus alerting rules format:
- alert: AlertName
annotations:
context: ""
description: "DESCRIPTION"
summary: "SUMMARY"
expr: promql_expression
for: duration
labels:
group: ""
severity: warning|critical|infoConfigure rules through Helm values
Mission Control provides the customRules field to add custom alerting rules to your Helm values.
Helm creates the ConfigMap automatically.
Add custom alerting rules to your Helm values using the customRules field:
alerting:
customRules: |
- alert: my-custom-alert-rule
annotations:
context: ""
description: test alert
summary: test alert
expr: <promql expression>
for: 1m
labels:
group: ""
severity: warningApply the configuration
Update your Mission Control installation with the new configuration:
-
Local chart
-
Remote chart
helm upgrade mission-control ./mission-control-chart \
--values CUSTOM_VALUES_FILE.yamlReplace CUSTOM_VALUES_FILE with the path to your custom values file.
helm upgrade mission-control datastax/mission-control \
--values CUSTOM_VALUES_FILE.yamlReplace CUSTOM_VALUES_FILE with the path to your custom values file.
Verify the rules
Check that your custom rules are loaded:
kubectl logs -n mission-control deployment/mission-control-mimir-ruler | grep -i "rule"Verify that the rules are loaded in the mission-control-mimir-ruler deployment:
kubectl get configmap -n mission-control | grep rulerExample custom rules
Here are some sample custom alerting rules for Mission Control environments.
High memory usage alert
- alert: HighMemoryUsage
annotations:
context: ""
description: "Memory usage is above 90% for more than 5 minutes on {{ $labels.instance }}"
summary: "High memory usage detected"
expr: (host_memory_total_bytes - host_memory_available_bytes) / host_memory_total_bytes > 0.9
for: 5m
labels:
group: ""
severity: warningCassandra latency alert
- alert: HighCassandraLatency
annotations:
context: ""
description: "95th percentile latency is above 100ms for more than 2 minutes"
summary: "High Cassandra latency detected"
expr: histogram_quantile(0.95, rate(org_apache_cassandra_metrics_client_request_latency_seconds_bucket[5m])) > 0.1
for: 2m
labels:
group: ""
severity: criticalLow disk space alert
- alert: LowDiskSpace
annotations:
context: ""
description: "Disk space is above 90% on {{ $labels.instance }}"
summary: "Low disk space"
expr: host_filesystem_used_ratio{mountpoint="/"} > 0.9
for: 1m
labels:
group: ""
severity: criticalUpdate rules
Update existing rules by modifying your Helm values and applying the changes:
helm upgrade mission-control ./mission-control-chart --values CUSTOM_VALUES_FILE.yamlReplace CUSTOM_VALUES_FILE with the path to your custom values file.
Delete rules
Remove specific rules by editing your Helm values and applying the changes:
helm upgrade mission-control ./mission-control-chart --values CUSTOM_VALUES_FILE.yamlReplace CUSTOM_VALUES_FILE with the path to your custom values file.
See also
After creating custom alerting rules, you can use Mission Control’s features to work with your monitoring through the alerting system. For more information, see:
