Performance

Kafka Producer Tuning: Batching, Acks, Compression

Most Kafka throughput problems are producer problems: tiny batches, no compression, or a buffer that blocks under load. A handful of settings fix most of them, and the producer's own metrics tell you whether a change actually helped.

Published November 20, 2026 · 6 min read · KLogic Team

A Kafka producer does not send records one by one. It groups records for the same partition into batches, compresses each batch, and sends batches to brokers in requests. Throughput, latency and broker load are all decided by how full those batches are. Tuning a producer is mostly about filling batches without adding too much waiting.

Batching: batch.size and linger.ms

batch.size=16384   # max bytes per batch, per partition (default 16 KiB)
linger.ms=5        # how long to wait for more records (default 5 ms since Kafka 4.0)
  • batch.size is an upper bound per partition, not a target. A batch is sent when it is full or when linger.ms expires, whichever comes first.
  • linger.ms trades a little latency for much bigger batches. Kafka 4.0 changed the default from 0 to 5 ms because a small wait helps almost every workload.

For high-throughput producers, values like batch.size=65536 to 262144 and linger.ms=10 to 50 are common. Raising batch.size alone often changes nothing, because batches are being sent before they fill up; raise linger.ms too and then check the metrics below.

Compression

Compression is applied per batch, so it works better with bigger batches. It cuts network and disk usage, often substantially for text formats like JSON, at the cost of some producer CPU.

compression.typeGood for
lz4A strong default: fast with a reasonable ratio.
zstdThe best ratio at moderate CPU cost; great for storage-heavy or cross-zone traffic.
snappyFast and widely supported, slightly worse ratio than lz4.
gzipGood ratio but CPU-heavy; rarely the best choice today.
noneThe default. Only right for payloads that are already compressed or encrypted.

Keep the topic's compression.type at its default of producer, so the broker stores batches as they arrive. A different topic-level codec forces the broker to recompress every batch, which costs broker CPU.

Durability: acks and idempotence

acks=all                 # default since Kafka 3.0
enable.idempotence=true  # default since Kafka 3.0

With acks=all, the leader waits for all in-sync replicas before acknowledging. Combined with min.insync.replicas=2 on the topic, an acknowledged record survives the loss of a broker. Lowering acks to 1 reduces latency slightly but risks losing acknowledged data during a leader failover, so treat it as a deliberate exception, not a tuning knob. Idempotence prevents duplicates from retries and keeps ordering with up to five in-flight requests per connection.

Buffering and back-pressure

buffer.memory=33554432   # 32 MiB of unsent records
max.block.ms=60000       # how long send() may block when the buffer is full
delivery.timeout.ms=120000

When brokers cannot keep up, records accumulate in the producer's buffer. Once it is full, send() blocks for up to max.block.ms and then throws. If your application calls send() on a request thread, that block becomes user-facing latency. Size the buffer for your peak burst, and treat a buffer that is regularly full as a capacity problem, not a setting to raise forever.

delivery.timeout.ms bounds the total time a record may spend being retried. After it, the send fails, so it should comfortably exceed the time a leader election takes.

Partitioning

Records with a key always go to the same partition. Records without a key use the sticky partitioner, which fills a batch for one partition before switching, and since Kafka 3.3 adapts to slow brokers by sending less to them. Hot keys are the main partitioning problem: one very common key sends most traffic to one partition and one broker, no matter how well the producer is tuned. Our partition strategy guide covers key design.

The metrics that prove a change worked

All of these come from kafka.producer:type=producer-metrics and per-topic variants:

MetricWhat to look for
batch-size-avgShould rise after increasing linger.ms. If it stays tiny, batches are not filling.
records-per-request-avgHigher means fewer, larger requests and less broker overhead.
compression-rate-avgCompressed size divided by uncompressed. Lower is better.
record-queue-time-avgTime records wait in the buffer. Roughly linger.ms when healthy; much higher means back-pressure.
request-latency-avgBroker round-trip time. Rising latency with flat throughput points at the cluster, not the producer.
bufferpool-wait-ratioShare of time send() waits for buffer space. Anything above zero under normal load needs attention.
record-error-rate / record-retry-rateFailures and retries. Errors should be zero; retries should be rare.

A worked example: from tiny batches to full ones

An (illustrative) event-ingestion service writes about 40,000 small JSON records per second across 48 partitions. The brokers are busy, request rates are high, and produce latency spikes at peak. The producer metrics show why:batch-size-avg is only a couple of kilobytes and records-per-request-avg is low, so the brokers handle a huge number of tiny requests.

The team makes three changes, one at a time, measuring after each:

  1. Enable lz4 compression. compression-rate-avg shows the JSON shrinking substantially, and bytes in on the brokers drop accordingly.
  2. Raise linger.ms to 20. Batches fill up: batch-size-avg grows several times over, request rate on the brokers falls sharply, and broker request handler idle time rises. Producer-side latency grows by roughly the linger time, which the service can afford.
  3. Raise batch.size to 64 KiB. Now that batches regularly hit the old 16 KiB limit, a bigger limit lets them grow further at peak.

The result is fewer, larger, compressed requests, lower broker CPU and more stable latency at peak, without touching the brokers at all. The same metrics are visible per topic in producer performance monitoring, which makes before-and-after comparisons easy.

A tuning order that works

  1. Measure the baseline: throughput, batch size, latency and errors.
  2. Turn on lz4 or zstd compression.
  3. Raise linger.ms step by step and watch batch-size-avg and end-to-end latency.
  4. Raise batch.size once batches regularly hit the current limit.
  5. Size buffer.memory for peak bursts and confirm bufferpool-wait-ratio stays at zero.
  6. Leave acks=all and idempotence on unless you have a measured reason not to.

Change one setting at a time and compare the same time of day, because traffic shape changes batch behaviour. KLogic's producer performance monitoring shows throughput, batching and errors per topic, so the effect of each change is visible on the broker side too.

Tuning for low latency instead

Some workloads care more about milliseconds than throughput, such as interactive features or trading signals. The trade-offs reverse:

  • Keep linger.ms at 0–1 so records are sent immediately.
  • Use lz4 or no compression, to minimise CPU time per batch.
  • Keep acks=all; the replication wait is usually small on a healthy cluster with nearby followers.
  • Watch broker request queues closely, because many small requests load brokers more.

Measure end-to-end latency rather than producer latency alone, because consumer fetch settings such as fetch.min.bytes and fetch.max.wait.ms add their own delay. Our performance optimization guide covers the consumer side.

Handle errors in the send callback

send() is asynchronous: it returns before the record is written. Errors arrive later, in the callback or the returned future. A common bug is ignoring them, so records are silently lost when a topic is misconfigured or the cluster rejects them.

producer.send(record, (metadata, exception) -> {
  if (exception != null) {
    log.error("send failed for key {}", record.key(), exception);
    failures.increment();   // alert on this counter
  }
});

Count failures in your own metrics as well as watching record-error-rate, and call flush() before shutting down so buffered records are not dropped. If records must not be lost, route permanent failures to a dead letter queue.

Common questions

What is a good batch.size for Kafka producers?

The default 16 KiB is conservative. High-throughput producers commonly use 64 KiB to 256 KiB, but only after raising linger.ms, because batches rarely fill without a small wait.

Does linger.ms add latency?

It adds up to linger.ms of waiting per batch, but it often lowers latency at peak by reducing the number of requests the brokers have to handle.

Which compression type should I use in Kafka?

lz4 is a strong default for speed. zstd gives the best ratio at moderate CPU cost and suits storage-heavy or cross-zone traffic. Leave the topic's compression.type as producer to avoid recompression on the broker.

Is acks=1 faster than acks=all?

Slightly, because the leader does not wait for followers. It risks losing acknowledged data during leader failover, so keep acks=all for important data and tune batching instead. See exactly-once semantics for how acks interacts with idempotence.

Why does producer send() block?

Because the buffer (buffer.memory) is full: the producer is generating data faster than brokers accept it. That is a capacity or broker-health problem; check broker saturation metrics in our JMX metrics guide.

See your Kafka clusters the way KLogic does

Consumer lag, replication health, alerts and AI-assisted debugging in one place, for self-hosted Kafka, AWS MSK and Confluent.

Free 14-day trial · No credit card required