Most Kafka alerting is built on thresholds: lag above a number, latency above a number, disk above a percentage. They are easy to write and hard to live with. A deployment that briefly raises lag pages someone for nothing, while a consumer that is a little too slow all day never crosses the line until the backlog is hours deep. Service level objectives (SLOs) give you a better question to alert on: are we on track to break our promise to the people who depend on this data?
Step 1: choose SLIs that reflect user impact
A service level indicator (SLI) is a measurement of good events over valid events. For Kafka platforms, three SLIs cover most of what users care about:
| SLI | Good event | Who cares |
|---|---|---|
| Produce availability | A produce request that succeeds | Every team writing to Kafka |
| Data freshness | A minute in which a consumer group is less than N seconds behind | Teams consuming the data downstream |
| End-to-end latency | A record consumed within N seconds of being produced | Real-time products and dashboards |
Measure lag in time, not offsets
"50,000 messages behind" means seconds for a busy topic and days for a quiet one. Freshness SLIs need lag expressed in time: either the age of the oldest unconsumed record, or offset lag divided by the recent consumption rate. Our consumer lag guide compares the methods.
Step 2: set objectives and error budgets
An SLO is a target for an SLI over a window, for example "99.9% of minutes in a 30-day window, the orders consumer is under 60 seconds behind". The remaining 0.1%, about 43 minutes a month, is the error budget: the amount of badness you have agreed to accept.
- Set objectives from what downstream teams actually need, not from what the system happens to achieve today.
- Start slightly looser than current performance, then tighten once alerts are trustworthy.
- Give critical pipelines their own SLOs; one cluster-wide SLO hides the group that matters.
Step 3: alert on burn rate
Burn rate is how fast you are spending the error budget relative to plan. A burn rate of 1 uses the whole budget in exactly one SLO window. A burn rate of 10 uses it in a tenth of the window.
burn rate = observed error rate ÷ (1 − SLO)
# 99.9% SLO, 1% of minutes currently stale:
burn rate = 0.01 ÷ 0.001 = 10The widely used multi-window approach from Google's SRE workbook combines a long window, which proves the problem is significant, with a short window, which proves it is still happening:
| Severity | Long window | Short window | Burn rate | Budget spent |
|---|---|---|---|---|
| Page | 1 hour | 5 minutes | 14.4 | 2% in an hour |
| Page | 6 hours | 30 minutes | 6 | 5% in six hours |
| Ticket | 3 days | 6 hours | 1 | 10% in three days |
The figures assume a 30-day SLO window. A short deployment spike burns budget briefly and never satisfies the long window, so it does not page. A slow, steady problem eventually satisfies the six-hour or three-day rule, so it does get noticed, which a fixed threshold would miss.
An example in Prometheus
Assuming a recording rule consumer_group_stale that is 1 when a group's time lag exceeds its target and 0 otherwise, a fast-burn page for a 99.9% objective looks like this:
- alert: OrdersFreshnessFastBurn
expr: |
avg_over_time(consumer_group_stale{group="orders"}[1h]) > (14.4 * 0.001)
and
avg_over_time(consumer_group_stale{group="orders"}[5m]) > (14.4 * 0.001)
labels:
severity: page
annotations:
summary: "orders consumer is burning its freshness error budget fast"Keep cause-based alerts for the things SLOs miss
SLO alerts tell you users are affected. A few cause-based alerts are still worth keeping because they predict impact before the budget starts burning:
- Partitions below min ISR or offline, because writes are failing or about to.
- Disk forecast to fill within days, because it ends in an outage.
- A consumer group with no active members, because its lag will grow without limit.
Route these to the platform team and SLO burn alerts to the owners of each pipeline, with runbooks attached.
A worked example: one week, two kinds of alert
Consider an (illustrative) week for an orders pipeline with a 99.9% freshness SLO and a 60-second target, compared against a classic "lag above 10,000 messages" threshold.
- Monday, deployment. Lag spikes to 25,000 messages for three minutes during a rolling restart. The threshold alert pages the on-call engineer. The SLO fast-burn rule does not fire: three stale minutes are a small fraction of the hour.
- Wednesday, slow consumer. A dependency slows down and the consumer falls a little behind all day, staying between 4,000 and 9,000 messages, which is 2 to 4 minutes stale. The threshold alert never fires. The six-hour SLO rule pages in the early afternoon because a large share of minutes are over target.
- Friday, broker failure. A broker crash stalls several partitions. Both alerts fire, and the SLO alert also tells the owning team how much budget is left for the month.
One false page removed, one missed incident caught, and the same result where it matters. That is the practical case for SLOs: they align alerts with what downstream teams experience, using metrics you already have. Pair them with cause-based alerts for replication and disk, which predict trouble before the budget burns.
Rolling this out
- Pick your three most important pipelines and agree a freshness target with their consumers.
- Build the SLIs and watch them on a dashboard for two weeks before alerting.
- Add fast-burn pages and slow-burn tickets; delete the threshold alerts they replace.
- Review budget consumption monthly and adjust targets that are too strict or too loose.
The data for these SLIs comes from per-partition lag, consumption rates and produce success rates, which KLogic already collects for every cluster, alongside alert routing to Slack, Microsoft Teams, PagerDuty and email. Starting from that data makes the first SLO a matter of days rather than a project.
Producer-side SLIs
Freshness covers consumers. Producers need their own SLI, because a cluster that rejects or delays writes breaks every pipeline at once. The simplest is produce availability: successful produce requests divided by all produce requests, measured from client metrics (record-error-rate against send rate) or from broker failed-request rates. Latency SLIs can use produce request p99 from the brokers, as described in our JMX metrics guide.
Exclude errors caused by clients themselves, such as authorisation failures for a misconfigured application, or one bad deployment will burn the platform team's budget.
Reporting error budgets
SLOs pay off most when they shape decisions, not only alerts. A short monthly report per pipeline works well:
- Budget remaining at the end of the month, and the biggest incidents that consumed it.
- Whether the objective is still right, too strict or too loose.
- What will change next month: capacity, a fix, or a revised target.
When a pipeline keeps exhausting its budget, that is evidence for investing in capacity or reliability work, such as the changes in our performance optimization guide, rather than another round of alert tuning.
Common questions
What is a good SLO for Kafka consumer lag?
It depends on the use case: seconds for fraud detection or user-facing features, minutes for analytics pipelines. Agree the target with the teams that consume the data, then express it as lag in time, not offsets.
What is error budget burn rate?
How fast you are using the allowed unreliability relative to plan. A burn rate of 1 uses the whole budget in one SLO window; a burn rate of 14.4 uses 2% of a 30-day budget in an hour.
Should I delete threshold alerts when adopting SLOs?
Replace symptom thresholds such as raw lag with SLO burn alerts, but keep cause-based alerts that predict outages, such as partitions below min ISR or disks forecast to fill.
How do I measure Kafka lag in seconds?
Divide offset lag by the recent consumption rate, or compare the timestamp of the last consumed record with the latest produced one. Our consumer lag guide compares the methods.
Can I build SLOs on managed Kafka like MSK or Confluent Cloud?
Yes. The SLIs come from consumer lag and produce success rates, which are available on managed services too. See Confluent Cloud monitoring and AWS MSK monitoring.