Kafka keeps every record on disk until retention removes it, so disk usage is roughly bytes written per second × retention × replication factor. When that number outgrows the disks, the consequences are severe, which is why disk deserves more attention than most broker metrics.
What happens when a Kafka disk fills up
When a broker cannot write to a log directory, it marks that directory offline. With a single log directory, the broker cannot work and shuts down. With several directories (JBOD), the partitions on the failed directory go offline on that broker while the rest keep working. Either way, leadership moves to other replicas, those brokers take on more traffic and more data, and a cluster that was close to full everywhere can cascade from one full disk into several.
Recovery is slower than prevention: you need free space before the broker can start, and catching replicas up afterwards puts extra load on the cluster.
How to measure disk usage
Filesystem usage per broker
The most important number is free space on each log directory's filesystem, from your host monitoring (for example node_filesystem_avail_bytes from node exporter) or, on Kubernetes, the kubelet's volume metrics. Alert on the fullest broker, not the cluster average; partitions are rarely spread perfectly.
Size per partition
To find out what is using the space, Kafka reports log size per partition through JMX (kafka.log:type=Log,name=Size) and through the admin API:
kafka-log-dirs.sh --bootstrap-server $BOOTSTRAP --describe \
--broker-list 1,2,3 --topic-list clickstream.raw,orders.v2The output lists each partition's size per log directory. Summing by topic usually shows that a handful of topics use most of the disk, which is where retention changes have the most effect.
Forecast instead of reacting
A simple forecast catches most problems weeks ahead:
expected disk per topic ≈ bytes in per second × retention seconds × replication factor
days until full ≈ free bytes ÷ net daily growthNet daily growth is what matters: during the first retention period after a new topic starts, usage grows every day; after that, old segments are deleted as fast as new ones arrive and usage levels off. A topic whose disk usage keeps rising after its retention period has elapsed is receiving more traffic than before, or has data that is not being deleted, often because of future timestamps or a compacted topic that keeps growing.
KLogic's forecasting projects storage growth from your clusters' own history, so a broker heading for a full disk shows up as a planning task rather than an outage.
Alert thresholds that work
- Warn at 75% on any log directory, to start a planned fix.
- Page at 85–90%, leaving time to act before writes fail.
- Forecast alert when a broker is projected to be full within 7–14 days.
- Imbalance alert when the fullest broker is much fuller than the average.
Freeing space safely
- Lower retention on the biggest topics. This is the fastest fix. Space is released at the next retention check, but remember that consumers lagging beyond the new retention will lose data. See our retention guide before changing it.
- Delete topics nobody uses. Test topics and abandoned pipelines are common. Confirm there are no active producers or consumers first.
- Move partitions from the fullest broker to emptier ones with
kafka-reassign-partitions.sh, or between log directories on the same broker. Always set a throttle so the move does not starve client traffic:
Remove the throttle once the reassignment finishes; a forgotten throttle slows replication later.kafka-reassign-partitions.sh --bootstrap-server $BOOTSTRAP \ --reassignment-json-file move.json --execute --throttle 50000000 - Add capacity: expand volumes if your platform allows it, add disks as new log directories, or add brokers and rebalance partitions onto them.
- Use tiered storage for topics that need long history, so only recent segments stay on local disk.
Never delete segment files by hand on a running broker. Kafka tracks segments and their indexes; removing files behind its back leads to errors and data inconsistency. Use retention and the admin tools instead.
A worked example: forecasting one broker
Take an (illustrative) broker with 4 TB of disk, currently 62% full, so about 1.5 TB free.
- Over the last two weeks its usage grew by about 30 GB per day, after accounting for data that retention deleted.
- Days until the 85% alert threshold: (85% − 62%) × 4 TB ≈ 920 GB ÷ 30 GB ≈ 30 days.
- Days until completely full: 1.5 TB ÷ 30 GB ≈ 50 days.
That is comfortable today, but a new high-volume topic or a retention increase changes the slope immediately. Re-running the forecast whenever retention or traffic changes is what turns it from a report into an early warning.
Two refinements make forecasts more reliable. First, forecast each broker separately, because partition placement is rarely even and the fullest broker fails first. Second, use the growth trend from the last few weeks rather than a single day, so weekly traffic patterns do not skew the result. KLogic's forecasting handles both, using seasonal patterns from each cluster's own history.
Prevention checklist
- Set retention deliberately for every high-volume topic instead of relying on the 7-day default.
- Turn on producer compression; it reduces disk usage as well as network traffic.
- Keep at least 25–30% free on every broker as a buffer for spikes and reassignments.
- Alert on the fullest log directory and on the forecast, not only on current usage.
- Review the top ten topics by size every month.
Disk performance, not just capacity
A disk with plenty of free space can still hurt Kafka if it is too slow. Kafka writes sequentially and relies heavily on the operating system's page cache for reads, so the pattern to watch is:
- Write latency and utilisation. Rising write latency slows producers using
acks=alland makes followers fall behind, which shows up as under-replicated partitions. - Read throughput from disk. Consumers reading recent data are served from the page cache. Lagging consumers reading old data hit the disk and can slow everyone else down.
- Cloud volume limits. Network-attached volumes have provisioned throughput and IOPS limits. On AWS, for example, gp3 volumes start at a baseline of 3,000 IOPS and 125 MiB/s unless you provision more, which a busy broker can exceed.
JBOD: several disks per broker
Brokers can use several log directories, one per disk. That adds capacity and throughput, and a single disk failure only takes the partitions on that disk offline instead of the whole broker. The catch is balance: Kafka places new partitions on the directory with the fewest partitions, not the most free space, so directories can fill unevenly. Monitor each log directory separately, and use kafka-reassign-partitions.sh to move partitions between directories when one runs ahead. See cluster configuration for broker storage settings.
Common questions
What happens when a Kafka broker runs out of disk?
The broker marks the affected log directory offline. With one directory the broker shuts down; with several (JBOD) only the partitions on the failed directory go offline. Leadership moves to other brokers, which can then fill up faster.
How do I check disk usage per topic in Kafka?
Use kafka-log-dirs.sh --describe, which reports the size of every partition per log directory, or the kafka.log Log Size JMX metric. Sum by topic to find the biggest consumers of disk.
What disk usage threshold should trigger an alert?
Warn around 75% and page around 85–90% on the fullest log directory, and add a forecast alert for brokers projected to fill within one to two weeks.
Can I delete Kafka log files manually to free space?
No. Deleting segment files on a running broker causes errors and inconsistency. Lower retention, delete unused topics or move partitions instead. See our retention guide.
How much free disk should Kafka brokers keep?
Keep 25–30% free as a buffer for traffic spikes, partition reassignments and broker failures, which push extra data onto the remaining brokers. Our capacity planning guide covers sizing in more detail.