Kafka Alerting Best Practices: Beyond Threshold Monitoring

The Alert Fatigue Problem

Research consistently shows that on-call engineers who receive more than 5–10 alerts per shift begin ignoring them. When every alert is urgent, no alert is urgent. The goal of a good alerting strategy is not to catch every anomaly — it is to ensure that every alert that fires demands and deserves immediate human attention.

Multi-Condition Alert Rules

Single-metric thresholds produce the most false positives. Combining conditions dramatically improves precision — a lag of 50,000 records during a known batch window is fine; 50,000 records that have been growing for 20 minutes is not.

# Fire only when lag is high AND still growing
# (prevents false alarms during normal batch catch-up)

ALERT ConsumerLagCritical
  IF consumer_lag > 50000
     AND rate(consumer_lag[10m]) > 0   # still growing
     AND rate(consumer_lag[10m]) > 500  # growing fast
  FOR 5m
  LABELS { severity="critical" }
  ANNOTATIONS {
    summary = "{{ $labels.consumer_group }} lag growing rapidly",
    description = "Lag: {{ $value }}, rate: {{ rate(consumer_lag[10m]) }} records/sec"
  }

# Alert when disk is high AND will fill within 6 hours
ALERT BrokerDiskCritical
  IF broker_disk_used_percent > 80
     AND predict_linear(broker_disk_used_bytes[2h], 6 * 3600) > broker_disk_total_bytes
  FOR 10m
  LABELS { severity="critical" }

# Alert when produce latency p99 is 3× baseline for 5 consecutive minutes
# (Not just a momentary spike)
ALERT ProduceLatencyHigh
  IF produce_latency_p99 > (
    avg_over_time(produce_latency_p99[1h] offset 1h) * 3
  )
  FOR 5m
  LABELS { severity="high" }

Alert Anti-Patterns to Avoid