Create custom alerting rules
Mission Control provides advanced capabilities for managing custom alerting rules through Helm values.
This guide explains how to create custom alerting rules for Mission Control environments and integrate them with your monitoring workflow.
Prerequisites
-
Mission Control version 1.15.0 or later installed.
-
Access to the Kubernetes cluster where Mission Control is deployed.
-
Basic understanding of Prometheus alerting rules and PromQL.
-
Helm chart access for configuration updates.
How custom alerting rules work
Mission Control separates default and custom alerting rules into different management systems:
-
Default rules: Mission Control automatically manages these rules and updates them during system upgrades.
-
Custom rules: You manage these rules through Helm values, providing advanced configuration options.
This separation provides several benefits:
-
Mission Control preserves your custom rules during updates.
-
You can manage custom rules independently through Helm chart settings.
-
Rules created in the UI are managed separately from Helm-configured rules, allowing you to use both approaches simultaneously.
-
Multiple Mission Control instances can coexist without conflicts.
Create custom alerting rules
Create custom alerting rules by configuring them in your Helm values.
Custom alerting rules follow the Prometheus alerting rules format:
- alert: AlertName
annotations:
context: ""
description: "DESCRIPTION"
summary: "SUMMARY"
expr: promql_expression
for: duration
labels:
group: ""
severity: warning|critical|infoConfigure rules through Helm values
Mission Control provides the customRules field to add custom alerting rules to your Helm values.
Helm creates the ConfigMap automatically.
Add custom alerting rules to your Helm values using the customRules field:
alerting:
customRules: |
- alert: my-custom-alert-rule
annotations:
context: ""
description: test alert
summary: test alert
expr: <promql expression>
for: 1m
labels:
group: ""
severity: warningApply the configuration
Update your Mission Control installation with the new configuration:
# If using a local chart
helm upgrade mission-control ./mission-control-chart \
--values your-custom-values.yaml
# If using a remote chart
helm upgrade mission-control datastax/mission-control \
--values your-custom-values.yamlVerify the rules
Check that your custom rules are loaded:
# Check for successful rule loading
kubectl logs -n mission-control deployment/mission-control-mimir-ruler | grep -i "rule"
# Verify the alerting rules are loaded
kubectl get configmap -n mission-control | grep rulerExample custom rules
Here are some sample custom alerting rules for Mission Control environments.
High memory usage alert
- alert: HighMemoryUsage
annotations:
context: ""
description: "Memory usage is above 90% for more than 5 minutes on {{ $labels.instance }}"
summary: "High memory usage detected"
expr: (host_memory_total_bytes - host_memory_available_bytes) / host_memory_total_bytes > 0.9
for: 5m
labels:
group: ""
severity: warningCassandra latency alert
- alert: HighCassandraLatency
annotations:
context: ""
description: "95th percentile latency is above 100ms for more than 2 minutes"
summary: "High Cassandra latency detected"
expr: histogram_quantile(0.95, rate(org_apache_cassandra_metrics_client_request_latency_seconds_bucket[5m])) > 0.1
for: 2m
labels:
group: ""
severity: criticalLow disk space alert
- alert: LowDiskSpace
annotations:
context: ""
description: "Disk space is above 90% on {{ $labels.instance }}"
summary: "Low disk space"
expr: host_filesystem_used_ratio{mountpoint="/"} > 0.9
for: 1m
labels:
group: ""
severity: criticalUpdate rules
Update existing rules by modifying your Helm values and applying the changes:
helm upgrade mission-control ./mission-control-chart --values your-custom-values.yamlDelete rules
Remove specific rules by editing your Helm values and applying the changes:
helm upgrade mission-control ./mission-control-chart --values your-custom-values.yamlSee also
After creating custom alerting rules, you can use Mission Control’s features to work with your monitoring through the alerting system. For more information, see:
