Monitoring

Kafka SLOs and Burn-Rate Alerts

“Consumer lag above 10,000” is a threshold, not an objective. It pages during harmless spikes and stays quiet during slow degradation. Service level objectives fix both problems by alerting on how fast you are using up an agreed amount of unreliability.

Published November 20, 2026 · 6 min read · KLogic Team

Most Kafka alerting is built on thresholds: lag above a number, latency above a number, disk above a percentage. They are easy to write and hard to live with. A deployment that briefly raises lag pages someone for nothing, while a consumer that is a little too slow all day never crosses the line until the backlog is hours deep. Service level objectives (SLOs) give you a better question to alert on: are we on track to break our promise to the people who depend on this data?

Step 1: choose SLIs that reflect user impact

A service level indicator (SLI) is a measurement of good events over valid events. For Kafka platforms, three SLIs cover most of what users care about:

SLIGood eventWho cares
Produce availabilityA produce request that succeedsEvery team writing to Kafka
Data freshnessA minute in which a consumer group is less than N seconds behindTeams consuming the data downstream
End-to-end latencyA record consumed within N seconds of being producedReal-time products and dashboards

Measure lag in time, not offsets

"50,000 messages behind" means seconds for a busy topic and days for a quiet one. Freshness SLIs need lag expressed in time: either the age of the oldest unconsumed record, or offset lag divided by the recent consumption rate. Our consumer lag guide compares the methods.

Step 2: set objectives and error budgets

An SLO is a target for an SLI over a window, for example "99.9% of minutes in a 30-day window, the orders consumer is under 60 seconds behind". The remaining 0.1%, about 43 minutes a month, is the error budget: the amount of badness you have agreed to accept.

  • Set objectives from what downstream teams actually need, not from what the system happens to achieve today.
  • Start slightly looser than current performance, then tighten once alerts are trustworthy.
  • Give critical pipelines their own SLOs; one cluster-wide SLO hides the group that matters.

Step 3: alert on burn rate

Burn rate is how fast you are spending the error budget relative to plan. A burn rate of 1 uses the whole budget in exactly one SLO window. A burn rate of 10 uses it in a tenth of the window.

burn rate = observed error rate ÷ (1 − SLO)

# 99.9% SLO, 1% of minutes currently stale:
burn rate = 0.01 ÷ 0.001 = 10

The widely used multi-window approach from Google's SRE workbook combines a long window, which proves the problem is significant, with a short window, which proves it is still happening:

SeverityLong windowShort windowBurn rateBudget spent
Page1 hour5 minutes14.42% in an hour
Page6 hours30 minutes65% in six hours
Ticket3 days6 hours110% in three days

The figures assume a 30-day SLO window. A short deployment spike burns budget briefly and never satisfies the long window, so it does not page. A slow, steady problem eventually satisfies the six-hour or three-day rule, so it does get noticed, which a fixed threshold would miss.

An example in Prometheus

Assuming a recording rule consumer_group_stale that is 1 when a group's time lag exceeds its target and 0 otherwise, a fast-burn page for a 99.9% objective looks like this:

- alert: OrdersFreshnessFastBurn
  expr: |
    avg_over_time(consumer_group_stale{group="orders"}[1h]) > (14.4 * 0.001)
    and
    avg_over_time(consumer_group_stale{group="orders"}[5m]) > (14.4 * 0.001)
  labels:
    severity: page
  annotations:
    summary: "orders consumer is burning its freshness error budget fast"

Keep cause-based alerts for the things SLOs miss

SLO alerts tell you users are affected. A few cause-based alerts are still worth keeping because they predict impact before the budget starts burning:

  • Partitions below min ISR or offline, because writes are failing or about to.
  • Disk forecast to fill within days, because it ends in an outage.
  • A consumer group with no active members, because its lag will grow without limit.

Route these to the platform team and SLO burn alerts to the owners of each pipeline, with runbooks attached.

A worked example: one week, two kinds of alert

Consider an (illustrative) week for an orders pipeline with a 99.9% freshness SLO and a 60-second target, compared against a classic "lag above 10,000 messages" threshold.

  • Monday, deployment. Lag spikes to 25,000 messages for three minutes during a rolling restart. The threshold alert pages the on-call engineer. The SLO fast-burn rule does not fire: three stale minutes are a small fraction of the hour.
  • Wednesday, slow consumer. A dependency slows down and the consumer falls a little behind all day, staying between 4,000 and 9,000 messages, which is 2 to 4 minutes stale. The threshold alert never fires. The six-hour SLO rule pages in the early afternoon because a large share of minutes are over target.
  • Friday, broker failure. A broker crash stalls several partitions. Both alerts fire, and the SLO alert also tells the owning team how much budget is left for the month.

One false page removed, one missed incident caught, and the same result where it matters. That is the practical case for SLOs: they align alerts with what downstream teams experience, using metrics you already have. Pair them with cause-based alerts for replication and disk, which predict trouble before the budget burns.

Rolling this out

  1. Pick your three most important pipelines and agree a freshness target with their consumers.
  2. Build the SLIs and watch them on a dashboard for two weeks before alerting.
  3. Add fast-burn pages and slow-burn tickets; delete the threshold alerts they replace.
  4. Review budget consumption monthly and adjust targets that are too strict or too loose.

The data for these SLIs comes from per-partition lag, consumption rates and produce success rates, which KLogic already collects for every cluster, alongside alert routing to Slack, Microsoft Teams, PagerDuty and email. Starting from that data makes the first SLO a matter of days rather than a project.

Producer-side SLIs

Freshness covers consumers. Producers need their own SLI, because a cluster that rejects or delays writes breaks every pipeline at once. The simplest is produce availability: successful produce requests divided by all produce requests, measured from client metrics (record-error-rate against send rate) or from broker failed-request rates. Latency SLIs can use produce request p99 from the brokers, as described in our JMX metrics guide.

Exclude errors caused by clients themselves, such as authorisation failures for a misconfigured application, or one bad deployment will burn the platform team's budget.

Reporting error budgets

SLOs pay off most when they shape decisions, not only alerts. A short monthly report per pipeline works well:

  • Budget remaining at the end of the month, and the biggest incidents that consumed it.
  • Whether the objective is still right, too strict or too loose.
  • What will change next month: capacity, a fix, or a revised target.

When a pipeline keeps exhausting its budget, that is evidence for investing in capacity or reliability work, such as the changes in our performance optimization guide, rather than another round of alert tuning.

Common questions

What is a good SLO for Kafka consumer lag?

It depends on the use case: seconds for fraud detection or user-facing features, minutes for analytics pipelines. Agree the target with the teams that consume the data, then express it as lag in time, not offsets.

What is error budget burn rate?

How fast you are using the allowed unreliability relative to plan. A burn rate of 1 uses the whole budget in one SLO window; a burn rate of 14.4 uses 2% of a 30-day budget in an hour.

Should I delete threshold alerts when adopting SLOs?

Replace symptom thresholds such as raw lag with SLO burn alerts, but keep cause-based alerts that predict outages, such as partitions below min ISR or disks forecast to fill.

How do I measure Kafka lag in seconds?

Divide offset lag by the recent consumption rate, or compare the timestamp of the last consumed record with the latest produced one. Our consumer lag guide compares the methods.

Can I build SLOs on managed Kafka like MSK or Confluent Cloud?

Yes. The SLIs come from consumer lag and produce success rates, which are available on managed services too. See Confluent Cloud monitoring and AWS MSK monitoring.

See your Kafka clusters the way KLogic does

Consumer lag, replication health, alerts and AI-assisted debugging in one place, for self-hosted Kafka, AWS MSK and Confluent.

Free 14-day trial · No credit card required