Distributed systems usually offer at-most-once (you might lose a message) or at-least-once (you might process it twice). Kafka adds a third option for a specific shape of application: consume from Kafka, process, produce results back to Kafka. Three features make that exactly-once: the idempotent producer, transactions, and read-committed consumers.
Building block 1: the idempotent producer
When a producer retries a send because an acknowledgement was lost, the broker could store the record twice. The idempotent producer prevents this: each producer gets an ID and numbers its records per partition, and the broker discards duplicates. Since Kafka 3.0 it is on by default:
enable.idempotence=true # default since 3.0
acks=all # required, also the default since 3.0
max.in.flight.requests.per.connection=5 # up to 5 keeps orderingIdempotence covers retries within one producer session. If the application restarts and resends, the new producer instance has a new ID and the broker cannot tell the records are duplicates. That is what transactions add.
Building block 2: transactions
A transactional producer has a stable transactional.id. It can write to many partitions and commit consumer offsets as one atomic unit: either all of it becomes visible, or none of it does.
producer.initTransactions();
while (running) {
ConsumerRecords<String, Order> records = consumer.poll(Duration.ofMillis(200));
producer.beginTransaction();
for (ConsumerRecord<String, Order> r : records) {
producer.send(new ProducerRecord<>("invoices", r.key(), toInvoice(r.value())));
}
producer.sendOffsetsToTransaction(currentOffsets(records), consumer.groupMetadata());
producer.commitTransaction();
}Because the output records and the input offsets commit together, a crash anywhere in the loop rolls back both. On restart the consumer resumes from the last committed offsets and the aborted output is never seen. The stable transactional.id also fences out a "zombie" instance of the same application that is still running after a failover.
Building block 3: read_committed consumers
Aborted records are still written to the log; they are just marked as aborted. Downstream consumers must opt in to skipping them:
isolation.level=read_committed # default is read_uncommittedA read-committed consumer only reads up to the last stable offset (LSO): the first offset belonging to a transaction that is still open. Everything after it waits until that transaction commits or aborts.
Kafka Streams does this for you
For Kafka Streams applications, all of the above is a single setting:
processing.guarantee=exactly_once_v2The v2 variant uses one producer per stream thread rather than per task, which scales much better and is the one to use on any recent Kafka version.
What exactly-once does not cover
- Side effects outside Kafka. Writing to a database, sending an email or calling an API inside the loop is not part of the transaction. If the transaction aborts, the email has still been sent. Use idempotent writes or the outbox pattern for external systems.
- Non-transactional producers upstream. If the producer feeding your input topic can create duplicates, exactly-once processing faithfully processes those duplicates once each.
- read_uncommitted consumers. Any consumer that leaves the default isolation level sees aborted records.
The cost is latency, not just throughput. Read-committed consumers cannot see records until the transaction commits, so end-to-end latency grows with your commit interval. Commit every few hundred milliseconds for low latency, or less often for higher throughput.
What to monitor
Hanging transactions
A transaction that is never committed or aborted, for example because a producer crashed in a way the coordinator did not detect, pins the LSO. Every read-committed consumer of that partition stops making progress, and their lag grows while the producer side looks perfectly healthy. Since Kafka 3.0 the kafka-transactions.sh tool can find and abort them:
kafka-transactions.sh --bootstrap-server $BOOTSTRAP find-hanging --broker-id 1
kafka-transactions.sh --bootstrap-server $BOOTSTRAP abort \
--topic invoices --partition 3 --start-offset 1840211The symptom to alert on is lag that grows on one or a few partitions for read-committed consumers while producers keep writing. That is exactly the pattern partition-level lag monitoring catches.
Timeouts and aborts
transaction.timeout.ms(producer, default 60 seconds) must be shorter than the broker'stransaction.max.timeout.ms(default 15 minutes). Transactions that exceed it are aborted.- Track the rate of aborted transactions and producer errors such as fencing exceptions. A sudden rise often means two instances are using the same
transactional.id, or processing is taking longer than the timeout.
Commit latency
Producer metrics include the time spent in transaction operations. Rising commit time increases end-to-end latency for every read-committed consumer, so trend it alongside your latency targets.
A worked example: the partition that stopped moving
In an (illustrative) billing pipeline, a Kafka Streams application aggregates usage events with exactly_once_v2, and a downstream invoicing service reads the results with read_committed. One morning, invoices for a subset of customers stop appearing. The Streams application looks healthy, its throughput is normal and its own lag is low.
Lag per partition for the invoicing group shows the pattern: one partition of the output topic has a lag that grows steadily while its neighbours stay near zero. The producer keeps writing to it, so the log end offset rises, but the last stable offset has not moved since an infrastructure incident the night before. A transaction on that partition was left open by a producer that disappeared during the incident.
kafka-transactions.sh find-hanging lists the open transaction on that partition with its start offset. After confirming the original producer instance is gone, the operator aborts it. The last stable offset jumps forward, the invoicing consumer reads the backlog, and lag drains within minutes.
Two habits would have caught this sooner: alerting on lag per partition rather than per group (a group-level average hid one stuck partition), and watching for a growing gap between the log end offset and the last stable offset. Our consumer lag guide explains why partition-level lag matters.
Should you use it?
Use exactly-once when duplicates would be expensive to detect downstream, such as financial aggregates, billing and stateful stream processing, and when the pipeline is Kafka-to-Kafka. For pipelines that end in an external system, an idempotent consumer with at-least-once delivery is usually simpler and just as correct. Whichever you choose, monitor lag per partition, because a stuck transaction or a slow consumer shows up there first. KLogic tracks lag per partition for every group, which surfaces exactly this kind of localised stall.
Exactly-once in Kafka Connect
Since Kafka 3.3, source connectors can run with exactly-once delivery when the Connect cluster enables it with exactly.once.source.support=enabled and the connector supports it. The worker then writes records and source offsets in one transaction, so a restart neither drops nor duplicates data from the source system. Connectors can declare exactly.once.support=required to refuse to start without it.
Sink connectors get exactly-once results differently: by writing idempotently, for example with upserts keyed on a unique ID, so replays after a restart overwrite rather than duplicate. Check each connector's documentation, and watch task health and errors with Connect monitoring.
Choosing transactional IDs
- Make each
transactional.idstable across restarts of the same logical instance, so fencing works. - Make it unique across instances that run at the same time, or they will fence each other off.
- On Kubernetes, derive it from a stable identity such as a StatefulSet pod name, the same approach used for static membership in our consumer rebalancing guide.
Producer errors from fencing are a useful signal: a sudden rise often means a deployment started two instances with the same ID. Track them with producer monitoring.
Common questions
Is exactly-once the default in Kafka?
Partly. Idempotent producers have been on by default since Kafka 3.0, which removes duplicates from retries. Transactions and read_committed consumers are opt-in, and Kafka Streams needs processing.guarantee=exactly_once_v2.
Does exactly-once slow Kafka down?
Transactions add coordination and make read_committed consumers wait for commits, which adds latency roughly equal to your commit interval. Throughput impact is usually modest with sensible commit intervals.
Can Kafka Connect provide exactly-once?
Source connectors can support exactly-once delivery when the worker and connector are configured for it, and some sink connectors achieve it by writing idempotently. Check each connector's documentation.
What is the last stable offset?
The offset up to which all transactions are decided. read_committed consumers cannot read past it, so an open transaction pins it and stalls those consumers on that partition.
How do I know if a transaction is hanging?
Lag that grows on a single partition for read_committed consumers while producers keep writing is the main symptom. Confirm with kafka-transactions.sh find-hanging. Partition-level lag monitoring makes the pattern visible.