Fundamentals

Kafka Retention Settings: A Practical Guide

Retention decides how long Kafka keeps data and how much disk it needs. The settings look simple, but they apply to segments rather than records, which is why data so often lives longer than retention.ms says it should.

Published November 20, 2026 · 6 min read · KLogic Team

Kafka is a log, not a queue: consuming a record does not delete it. Records stay until the retention policy removes them, whether anyone has read them or not. Getting retention right is the difference between a cluster with comfortable disk headroom and one that runs out of space during a traffic spike.

The settings that control retention

Topic settingBroker defaultWhat it does
retention.mslog.retention.hours=168 (7 days)Delete segments whose newest record is older than this.
retention.byteslog.retention.bytes=-1 (no limit)Delete the oldest segments once a partition exceeds this size.
segment.byteslog.segment.bytes=1 GiBRoll to a new segment file at this size.
segment.mslog.roll.hours=168 (7 days)Roll to a new segment after this long, even if it is small.
cleanup.policydeletedelete, compact, or both.

Topic-level settings override broker defaults. Broker defaults use different names (log.retention.ms wins over log.retention.minutes, which wins over log.retention.hours).

Why data lives longer than retention.ms

Kafka deletes whole segments, never individual records, and it never deletes the active segment that is still being written. A segment becomes eligible only when it is closed and its newest record is older than retention.ms.

On a low-traffic topic with the defaults, a segment might take the full 7 days of segment.ms to roll, and then another 7 days of retention before its newest record expires. Records written at the start of that segment can survive for nearly two weeks. If you need data gone within a deadline, for privacy reasons for example, lower segment.ms as well as retention.ms.

Deletion also is not instant: the retention check runs every log.retention.check.interval.ms (5 minutes by default), and deleted segments are removed from disk after file.delete.delay.ms (1 minute).

How retention.bytes is counted

retention.bytes is a per-partition limit. A topic with 12 partitions, replication factor 3 and retention.bytes=10 GiB can use up to 360 GiB across the cluster, plus up to one extra segment per partition, because the active segment is never deleted. When both limits are set, whichever is reached first triggers deletion.

Timestamps decide what "old" means

Time-based retention uses record timestamps. With the default message.timestamp.type=CreateTime, producers set the timestamp. A producer with a wrong clock that writes timestamps in the future creates segments that never look old enough to delete. Newer Kafka versions let you reject timestamps too far in the future with log.message.timestamp.after.max.ms, or you can switch sensitive topics to LogAppendTime.

Changing retention safely

Retention is a dynamic topic config, so it takes effect without a restart:

# keep 3 days on one topic
kafka-configs.sh --bootstrap-server $BOOTSTRAP --alter \
  --entity-type topics --entity-name clickstream.raw \
  --add-config retention.ms=259200000

# check what is actually set
kafka-configs.sh --bootstrap-server $BOOTSTRAP --describe \
  --entity-type topics --entity-name clickstream.raw

Before lowering retention, check two things:

  • Consumers that lag. A consumer further behind than the new retention will lose data: its offset falls off the log and it resets according to auto.offset.reset.
  • Who depends on replay. Teams that reprocess history after a bug fix need the old data to still be there.

Retention changes are a common cause of surprise data loss, so record them. KLogic's configuration tracking keeps a history of topic config changes with before and after values.

Compaction and tiered storage

With cleanup.policy=compact, Kafka keeps the latest record for each key instead of a time window, which suits changelog and state topics. compact,delete combines both: compacted, but still bounded by time. Our log compaction guide covers the details.

With tiered storage, older segments move to object storage. local.retention.ms and local.retention.bytes control how much stays on broker disks, while retention.ms still controls total lifetime. That lets you keep months of history without months of local disk. See our tiered storage guide.

A worked example: sizing a clickstream topic

Suppose (illustratively) a clickstream topic receives 20 MB per second at peak and averages 8 MB per second over a day.

  • Daily volume at the average rate: 8 MB × 86,400 seconds ≈ 690 GB per day.
  • With 3 days of retention: about 2.1 TB of unique data.
  • With replication factor 3: about 6.2 TB across the cluster.
  • Plus the active segments, plus 30% headroom for spikes and reassignments: roughly 8 TB of disk for one topic.

If the topic has 24 partitions, each partition holds around 2.1 TB ÷ 24 ≈ 90 GB at steady state. Setting retention.bytes to 100 GB per partition acts as a safety net: if traffic doubles unexpectedly, size-based retention kicks in before the disks fill, at the cost of keeping less than three days for that period.

Compression changes the picture considerably. If producers compress with zstd and the payload is JSON, the stored size can be a fraction of the raw size, which is why enabling producer compression is often the cheapest way to afford longer retention.

Choosing values

  1. Start from the longest outage a consumer must be able to recover from, plus a safety margin.
  2. Add time for replays after bad deployments, usually a few days.
  3. Multiply by the topic's bytes-in rate and replication factor to get disk usage, and check you have headroom.
  4. Lower segment.ms for low-traffic topics with strict deletion requirements.
  5. Use compaction or tiered storage instead of very long local retention.

Retention is also the biggest lever on disk usage, so revisit it whenever traffic grows. Our guide to Kafka disk usage shows how to forecast space from these numbers.

Retention for internal topics

Kafka and its ecosystem create internal topics with their own retention rules, and they are easy to overlook:

  • Consumer offsets. Committed offsets are stored in __consumer_offsets, a compacted topic. Offsets for a group that has had no members are expired after offsets.retention.minutes (7 days by default). A batch job that runs less often than that loses its position and starts again according to auto.offset.reset.
  • Kafka Connect. The config, offset and status topics should be compacted with no time-based deletion; losing them loses connector state.
  • Kafka Streams. Changelog topics are compacted; repartition topics are purged by the application once records are processed. Do not add time-based deletion to changelogs.

If a group's offsets expire unexpectedly, lag monitoring shows it as a sudden jump to zero or to the full topic size. Consumer group monitoring makes those jumps easy to spot.

Retention and replay strategy

Retention is also an architectural decision. Topics used as an event store need long retention or compaction; topics used as a transport between services need only enough to survive outages. Mixing the two on one topic usually means paying for long retention everywhere. For long-lived history, consider tiered storage or archiving to a lakehouse, as in our Kafka to Iceberg guide.

Common questions

What is the default retention in Kafka?

Seven days (log.retention.hours=168), with no size limit (log.retention.bytes=-1). Both can be overridden per topic with retention.ms and retention.bytes.

Why is my Kafka topic using more space than retention allows?

Usually because deletion works on whole segments and the active segment is never deleted, because records have future timestamps, or because the topic is compacted rather than deleted.

Is retention.bytes per topic or per partition?

Per partition. Multiply by the number of partitions and the replication factor to get the maximum space a topic can use across the cluster.

Does consuming a message delete it from Kafka?

No. Kafka keeps records until retention or compaction removes them, regardless of whether they have been consumed, which is what allows multiple consumer groups and replays.

How do I make Kafka delete data faster?

Lower retention.ms and segment.ms together so segments roll and expire sooner. Track who changed retention and when with configuration tracking, because retention changes can delete data consumers still need.

See your Kafka clusters the way KLogic does

Consumer lag, replication health, alerts and AI-assisted debugging in one place, for self-hosted Kafka, AWS MSK and Confluent.

Free 14-day trial · No credit card required