Architecture

Kafka Dead Letter Queues: Patterns and Monitoring

Kafka processes a partition in order, so one record your consumer cannot handle blocks everything behind it. A dead letter queue moves that record aside so the rest can flow, as long as you design it, and watch it, deliberately.

Published November 20, 2026 · 6 min read · KLogic Team

A dead letter queue (DLQ) is a separate topic where you send records that failed processing, together with enough context to understand and replay them later. Kafka itself has no built-in DLQ for consumers; it is a pattern you implement in your consumer framework. Kafka Connect, on the other hand, supports it directly for sink connectors.

When you need one

Not every failure belongs in a DLQ. Separate the two kinds of error first:

  • Transient failures such as a database timeout or a rate-limited API will succeed if you try again. Retry them, with backoff, before giving up.
  • Permanent failures such as malformed payloads, schema mismatches or records that break a business rule will fail forever. Retrying them blocks the partition. These are what the DLQ is for.

Without that distinction, a DLQ fills with records that would have succeeded a second later, or a consumer retries a poison message forever, crash-looping and triggering rebalances across the group.

Pattern 1: dead letter queue in Kafka Connect

Sink connectors can route records that fail conversion or transformation, and records the connector reports as errant, to a DLQ topic instead of stopping the task:

errors.tolerance=all
errors.deadletterqueue.topic.name=dlq.orders-sink
errors.deadletterqueue.topic.replication.factor=3
errors.deadletterqueue.context.headers.enable=true
errors.log.enable=true
errors.log.include.messages=false
  • errors.tolerance=all keeps the task running past bad records; the default none fails the task on the first error.
  • Context headers add the original topic, partition, offset and the exception to each DLQ record, which is what makes replay and debugging possible.
  • Keep errors.log.include.messages=false if payloads may contain personal data, so they do not end up in plain-text logs.

Connect exposes task error metrics, including total record errors, skipped records and DLQ produce requests and failures. Our Kafka Connect best practices cover the rest of a production setup.

Pattern 2: dead letter queue in your own consumers

For application consumers, the usual shape is:

  1. Try to process the record.
  2. On a transient error, retry a few times with backoff, inside the poll interval.
  3. On a permanent error, or after retries are exhausted, produce the record to the DLQ topic with error headers.
  4. Commit the original offset so the partition moves on.

Spring for Apache Kafka implements this with an error handler and a dead letter publishing recoverer; by default it writes to a topic named after the source with a .DLT suffix. Other frameworks have equivalents, or you can write the few lines yourself.

Retry topics for slow retries

If a dependency can be down for minutes, retrying inside the consumer blocks the partition and risks exceeding max.poll.interval.ms. Retry topics solve this: failed records go to orders.retry.1m, then orders.retry.10m, and only reach the DLQ after the last attempt. Each retry topic is consumed with a delay, so the main topic keeps flowing.

What to put in a DLQ record

HeaderWhy
Original topic, partition, offsetFind the source record and replay in the right place.
Exception class and messageGroup failures by cause without reading payloads.
Consumer group and application versionKnow which deployment rejected it; often a bug fixed later.
Failure timestamp and attempt countOrder replays and spot records that keep failing.

Keep the original key and value unchanged, so a replay produces exactly what the producer sent.

Replaying from the DLQ

A DLQ nobody drains is just a slower way of losing data. Plan replay from day one:

  • Fix the cause first: deploy the bug fix, update the schema or correct the reference data.
  • Replay by producing records back to the source topic, or to a dedicated replay topic the consumer also reads.
  • Make consumers idempotent, because replay can deliver a record the original consumer partly processed.
  • Set a retention on DLQ topics long enough for your slowest investigation, often several weeks.

A worked example: a schema change gone wrong

In an (illustrative) e-commerce pipeline, the checkout team adds a required field to the order event and deploys their producer. The fulfilment consumer has not been updated yet, so every new order fails validation on its side.

Without a DLQ, the first new-format order stops the partition: the consumer retries, crashes, restarts and retries again. The group starts rebalancing and lag climbs on every partition that receives new orders. Old-format orders stuck behind them are delayed too.

With a DLQ, each failing order is written to orders.DLT with headers naming the exception ("missing field shippingTier") and the consumer moves on. Two things happen next:

  1. The DLQ inflow alert fires within minutes because the share of DLQ records jumps from near zero to a large fraction of orders. The cause is obvious from the exception header, so nobody has to read payloads.
  2. The fulfilment team deploys the updated consumer, then replays the DLQ records to the source topic. Because the consumer is idempotent on order ID, replaying orders it partially processed is safe.

The better long-term fix is upstream: schema compatibility rules in a schema registry would have rejected the breaking change before it was deployed. The DLQ turned an outage into a delay; compatibility checks prevent the delay too.

Monitoring the DLQ

The DLQ is an early-warning system for data quality. Watch:

  • Inflow rate. Messages in per second on each DLQ topic. A sudden jump almost always follows a deploy or an upstream schema change.
  • DLQ share. DLQ records as a fraction of source records. A steady 0.01% may be acceptable; 2% is an incident.
  • Age of the oldest record. Shows whether anyone is actually draining the queue.
  • Connect task error metrics for connector DLQs, and whether the DLQ producer itself is failing.

Because a DLQ is an ordinary topic, any Kafka monitoring that tracks topic throughput can alert on it. In KLogic you can watch DLQ topics' message rates and set alerts on them like any other topic, and use message inspection to read the failed records and their headers without writing a consumer.

Dead letter queues in Kafka Streams

Kafka Streams has handler hooks for the two most common failure points. A deserialization exception handler decides what happens when an input record cannot be read: the built-in LogAndContinueExceptionHandler skips it, while the default fails the application. A production exception handler does the same for records that cannot be written to an output topic.

Skipping silently loses data, so many teams implement a custom handler that writes the raw bytes and error details to a DLQ topic before continuing. Errors inside your own processing logic are not covered by these handlers; catch them in the processor and route the record to a DLQ branch of the topology instead.

Ordering and the DLQ

Moving a record aside breaks strict per-key ordering: later records for the same key are processed before the failed one is replayed. For many workloads that is fine. For others, such as account balances or state machines, it is not, and the consumer should either stop and alert, or park all later records for that key until the failed one is resolved. Decide this per topic, and document it next to the delivery guarantees each pipeline relies on.

Common questions

Does Kafka have a built-in dead letter queue?

Not for plain consumers. Kafka Connect supports a DLQ for sink connectors through the errors.deadletterqueue settings, and frameworks such as Spring for Apache Kafka and Kafka Streams provide error handlers you can configure to write to a DLQ topic.

Should the DLQ topic have the same partitions as the source?

It does not have to. DLQ traffic is usually low, so a few partitions are enough. Keep the original partition and offset in headers so ordering information is not lost.

How long should DLQ records be retained?

Long enough for your slowest investigation and replay, often two to four weeks. Make sure that is longer than the retention of the source topic you might replay into. See our retention guide.

Can I put transient errors in the DLQ?

Avoid it. Retry transient errors with backoff, or use retry topics for longer delays. Sending them straight to the DLQ fills it with records that would have succeeded and hides real data problems.

How do I inspect records in a DLQ?

Read them with a consumer that prints headers, or browse them with a tool such as KLogic message inspection, which shows keys, values and headers without writing code.

See your Kafka clusters the way KLogic does

Consumer lag, replication health, alerts and AI-assisted debugging in one place, for self-hosted Kafka, AWS MSK and Confluent.

Free 14-day trial · No credit card required