Every Kafka partition has one leader and, usually, two or more followers. The followers that are fully caught up with the leader form the in-sync replica set (ISR). When a partition has fewer in-sync replicas than its replication factor, it is under-replicated. Nothing is broken yet: producers can still write and consumers can still read. But the safety margin you designed for is gone, and one more failure can turn a warning into an outage or data loss.
What the UnderReplicatedPartitions metric measures
Each broker reports kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions: the number of partitions it leads whose ISR is smaller than the full replica list. Because it is reported by leaders, you sum it across brokers to get the cluster total. A healthy, idle cluster reports zero.
A follower drops out of the ISR when it has not caught up to the leader's log end offset for longer than
replica.lag.time.max.ms=30000 # default since Kafka 2.5So under-replication is about time behind, not message count. A follower 10,000 messages behind that is still fetching and closing the gap stays in sync; a follower that stops fetching for 30 seconds is removed, even if it was only a few records behind.
The related metrics you should collect alongside it
| Metric | What it tells you |
|---|---|
| UnderMinIsrPartitionCount | Partitions whose ISR is below min.insync.replicas. Producers using acks=all now get NotEnoughReplicas errors. |
| OfflinePartitionsCount | Reported by the active controller. Partitions with no leader at all: reads and writes fail. |
| IsrShrinksPerSec / IsrExpandsPerSec | How often replicas leave and rejoin the ISR. Constant flapping points to an unstable broker or network. |
| Replica fetcher lag | Per-partition lag of each follower's fetcher thread, the earliest sign that a follower is slipping. |
The five causes behind almost every case
1. A broker is down or restarting
The most common and least worrying cause. During a rolling restart every partition with a replica on the restarting broker is under-replicated for a while. The count should rise as the broker stops and fall back to zero within minutes of it starting. If it does not return to zero, the broker is up but not catching up, which is one of the causes below.
2. Slow or saturated disks
Followers write everything they fetch. A disk that is slow, nearly full or shared with another noisy workload makes the follower fall behind. Look at disk utilisation and write latency on the brokers that host the lagging followers, and atLogFlushRateAndTimeMs. Full disks are worse: the affected log directory goes offline. See our guide to monitoring Kafka disk usage.
3. Network or cross-zone bandwidth limits
Replication traffic competes with client traffic. Clusters spread across availability zones often hit per-instance network limits during traffic peaks, so followers in one zone fall behind together. A pattern where all lagging followers live in the same rack or zone is the giveaway.
4. Not enough replica fetcher capacity
By default each broker uses a single fetcher thread per source broker (num.replica.fetchers=1). On brokers with many partitions or high throughput this becomes the bottleneck. Raising it to 2–4 often fixes chronic, throughput-driven under-replication, at the cost of a little extra CPU.
5. Long GC pauses or resource starvation
A broker that stops the world for several seconds stops fetching. Repeated pauses show up as ISR flapping: shrinks followed by expands a minute later. Check GC logs and the broker's heap usage, and make sure the page cache is not being squeezed by an oversized heap.
Reassignments and throttles count too. While partitions are being moved, new replicas are not yet in the ISR, and a replication throttle that is too low can keep them out for hours. Always check for in-progress reassignments before chasing hardware.
A runbook for under-replicated partitions
- Measure the blast radius. List affected partitions and check whether any are also below min ISR or offline:
kafka-topics.sh --bootstrap-server $BOOTSTRAP --describe --under-replicated-partitions kafka-topics.sh --bootstrap-server $BOOTSTRAP --describe --under-min-isr-partitions kafka-topics.sh --bootstrap-server $BOOTSTRAP --describe --unavailable-partitions - Find the common replica. If every affected partition has the same broker missing from its ISR, the problem is that broker. If the missing replicas are spread out, suspect the network or a cluster-wide load spike.
- Check the broker itself: is the process up, is it fetching, and what do disk, CPU, GC and network look like for the last hour?
- Check for reassignments and throttles:
Look forkafka-reassign-partitions.sh --bootstrap-server $BOOTSTRAP --list kafka-configs.sh --bootstrap-server $BOOTSTRAP --describe --entity-type brokersleader.replication.throttled.rateandfollower.replication.throttled.rateleft behind by an old reassignment. - Fix the cause, then confirm recovery. Restart a hung broker, free disk space, raise
num.replica.fetchers, or remove a stale throttle. Watch the count fall back to zero and IsrExpandsPerSec spike as replicas rejoin. - Rebalance leadership. After a broker returns, leaders often stay on the brokers that took over. Trigger a preferred leader election so load spreads again:
kafka-leader-election.sh --bootstrap-server $BOOTSTRAP \ --election-type preferred --all-topic-partitions
A worked example: one slow disk
Here is how the runbook plays out in a typical (illustrative) incident. At 14:05 the cluster reports 212 under-replicated partitions. No broker is down, no deployment is running, and producers are not seeing errors yet because every partition still has two of its three replicas in sync.
Listing the affected partitions shows the same follower missing in every case: broker 4. Its process is up and its fetcher threads are running, but replica fetcher lag on broker 4 is climbing steadily. The broker's host metrics show the answer: disk write latency on its data volume jumped from about a millisecond to well over 50 milliseconds at 14:02, and the volume is at 100% utilisation. A noisy neighbour on shared storage, or a failing disk, is the usual explanation.
The fix is not a Kafka setting. The team moves broker 4 to a healthy volume (on cloud instances, replacing the volume or the instance), restarts it, and watches the fetcher catch up. IsrExpandsPerSec spikes as partitions rejoin the ISR, and the under-replicated count falls to zero over the next twenty minutes. A preferred leader election then moves leadership back to broker 4.
Two lessons generalise. First, the shape of the problem (one common missing replica) pointed at the broker within a minute. Second, the alert that mattered was the sustained under-replication warning, not a page: the cluster was still durable enough to take writes, and the team had time to fix the cause calmly. That is the difference between alerting on under-replication and alerting on under-min-ISR.
How to alert without paging on every restart
A raw "UnderReplicatedPartitions > 0" alert pages during every deployment. Better rules separate durability risk from routine noise:
- Page when
UnderMinIsrPartitionCount > 0orOfflinePartitionsCount > 0for more than a minute. Writes are failing or about to. - Warn when under-replicated partitions stay above zero for 10–15 minutes and no planned maintenance is running.
- Track trends in ISR shrink rate per broker. A broker whose shrink rate creeps up week over week is your next incident.
KLogic ships under-replicated partition and broker-offline alert rules out of the box, so this coverage exists from the first day. Its WatchDog learns each cluster's normal behaviour and flags anomalies, which catches slow-building replication problems that a fixed threshold misses.
Prevention checklist
- Replication factor 3 and
min.insync.replicas=2for any topic you cannot afford to lose. - Rack awareness (
broker.rack) so replicas of a partition never share a zone. - Disk headroom of at least 30% and alerts long before 85%.
- Enough fetcher threads for your partition count and throughput.
- Rolling restarts that wait for zero under-replicated partitions before moving to the next broker.
Under-replicated partitions are rarely the root problem. They are the symptom that tells you which broker, disk or network path needs attention. Watch them alongside the metrics above and they become one of the most useful early warnings Kafka gives you.
Related signals to check during an incident
Under-replication rarely travels alone. When you open an incident, check these alongside it so you do not chase the wrong cause:
- Consumer lag. If leaders moved to fewer brokers, those brokers serve more fetches and consumers may slow down. See consumer lag monitoring.
- Producer errors.
NotEnoughReplicaserrors mean you have crossed into under-min-ISR territory; see producer tuning for how acks and retries behave. - Recent changes. A config change, a new topic with a high replication factor or a reassignment often lines up with the start of the problem. Configuration tracking shows what changed and when.
Common questions
Is a non-zero UnderReplicatedPartitions value always a problem?
No. It is expected during broker restarts, rolling upgrades and partition reassignments. It becomes a problem when it stays above zero with no planned work running, or when it is accompanied by partitions below min ISR.
What is the difference between under-replicated and under-min-ISR?
Under-replicated means the ISR is smaller than the replication factor. Under-min-ISR means it is smaller than min.insync.replicas, so producers using acks=all start getting errors. The first is a warning; the second is an outage for those producers.
Should I raise replica.lag.time.max.ms to stop ISR shrinks?
Rarely. Raising it hides followers that are genuinely falling behind and makes failover slower to react. Fix the cause (disk, network or fetcher capacity) instead, and only raise it if short, harmless pauses are removing healthy replicas.
Can under-replicated partitions cause data loss?
Not on their own. Data loss happens if the remaining in-sync replicas fail too, or if an unclean leader election promotes an out-of-sync replica. Keep unclean.leader.election.enable=false and alert on under-min-ISR to stay safe.
How do I monitor this across many clusters?
Sum the metric per cluster and alert on duration rather than presence. A platform such as KLogic cluster health monitoring does this per cluster and shows which broker the missing replicas belong to.