Node down for more than 10 minutes

Sev 2 - Warning

All

Source metric: org_apache_cassandra_metrics_thread_pools_completed_tasks.

Node down for 30 minutes

Sev 1 - Error

All

Source metric: org_apache_cassandra_metrics_thread_pools_completed_tasks.

Nodes down in different racks of same datacenter

All

Two nodes down across rack boundaries can lead to LOCAL_QUORUM CL errors in applications. Source metric: org_apache_cassandra_metrics_thread_pools_completed_tasks.

CPU above 80% for 5 minutes

All

An error that, if triggered too often, indicates low disk space and that the cluster should be scaled. Source metric: host_cpu_seconds_total.

Used disk space above 50% for one minute

All

Used disk space above 75% for one minute

Sev 1 - Error

All

A signal to expand the cluster before it gets into a state where cleanups are impossible due to insufficient disk space. Source metric: host_filesystem_used_ratio.

Used disk space above 50% for one minute

Sev 2 - Warning

All

A signal to expand the cluster before it gets into a state where cleanups are impossible due to insufficient disk space. Source metric: host_filesystem_used_ratio.

Load average above 20 for 5 minutes

Sev 2 - Warning

All

Good indicator for performance issues, the root cause of which can vary. Source metric: host_load5.

Load average above 32 for 5 minutes

Sev 1 - Error

All

Good indicator for performance issues, the root cause of which can vary. Source metric: host_load5.

Dropped messages over 5 minutes

Sev 1 - Error for >= 10,000 + Sev 2 - Warning for < 10,000

All

Thread pools cannot keep up with the pace of queries entering and being processed within the cluster. This leads to errors within the application stack and potentially incorrect replicas. Source metric: org_apache_cassandra_metrics_dropped_message_dropped_total.