A Kafka producer does not send records one by one. It groups records for the same partition into batches, compresses each batch, and sends batches to brokers in requests. Throughput, latency and broker load are all decided by how full those batches are. Tuning a producer is mostly about filling batches without adding too much waiting.
Batching: batch.size and linger.ms
batch.size=16384 # max bytes per batch, per partition (default 16 KiB)
linger.ms=5 # how long to wait for more records (default 5 ms since Kafka 4.0)- batch.size is an upper bound per partition, not a target. A batch is sent when it is full or when
linger.msexpires, whichever comes first. - linger.ms trades a little latency for much bigger batches. Kafka 4.0 changed the default from 0 to 5 ms because a small wait helps almost every workload.
For high-throughput producers, values like batch.size=65536 to 262144 and linger.ms=10 to 50 are common. Raising batch.size alone often changes nothing, because batches are being sent before they fill up; raise linger.ms too and then check the metrics below.
Compression
Compression is applied per batch, so it works better with bigger batches. It cuts network and disk usage, often substantially for text formats like JSON, at the cost of some producer CPU.
| compression.type | Good for |
|---|---|
| lz4 | A strong default: fast with a reasonable ratio. |
| zstd | The best ratio at moderate CPU cost; great for storage-heavy or cross-zone traffic. |
| snappy | Fast and widely supported, slightly worse ratio than lz4. |
| gzip | Good ratio but CPU-heavy; rarely the best choice today. |
| none | The default. Only right for payloads that are already compressed or encrypted. |
Keep the topic's compression.type at its default of producer, so the broker stores batches as they arrive. A different topic-level codec forces the broker to recompress every batch, which costs broker CPU.
Durability: acks and idempotence
acks=all # default since Kafka 3.0
enable.idempotence=true # default since Kafka 3.0With acks=all, the leader waits for all in-sync replicas before acknowledging. Combined with min.insync.replicas=2 on the topic, an acknowledged record survives the loss of a broker. Lowering acks to 1 reduces latency slightly but risks losing acknowledged data during a leader failover, so treat it as a deliberate exception, not a tuning knob. Idempotence prevents duplicates from retries and keeps ordering with up to five in-flight requests per connection.
Buffering and back-pressure
buffer.memory=33554432 # 32 MiB of unsent records
max.block.ms=60000 # how long send() may block when the buffer is full
delivery.timeout.ms=120000When brokers cannot keep up, records accumulate in the producer's buffer. Once it is full, send() blocks for up to max.block.ms and then throws. If your application calls send() on a request thread, that block becomes user-facing latency. Size the buffer for your peak burst, and treat a buffer that is regularly full as a capacity problem, not a setting to raise forever.
delivery.timeout.ms bounds the total time a record may spend being retried. After it, the send fails, so it should comfortably exceed the time a leader election takes.
Partitioning
Records with a key always go to the same partition. Records without a key use the sticky partitioner, which fills a batch for one partition before switching, and since Kafka 3.3 adapts to slow brokers by sending less to them. Hot keys are the main partitioning problem: one very common key sends most traffic to one partition and one broker, no matter how well the producer is tuned. Our partition strategy guide covers key design.
The metrics that prove a change worked
All of these come from kafka.producer:type=producer-metrics and per-topic variants:
| Metric | What to look for |
|---|---|
| batch-size-avg | Should rise after increasing linger.ms. If it stays tiny, batches are not filling. |
| records-per-request-avg | Higher means fewer, larger requests and less broker overhead. |
| compression-rate-avg | Compressed size divided by uncompressed. Lower is better. |
| record-queue-time-avg | Time records wait in the buffer. Roughly linger.ms when healthy; much higher means back-pressure. |
| request-latency-avg | Broker round-trip time. Rising latency with flat throughput points at the cluster, not the producer. |
| bufferpool-wait-ratio | Share of time send() waits for buffer space. Anything above zero under normal load needs attention. |
| record-error-rate / record-retry-rate | Failures and retries. Errors should be zero; retries should be rare. |
A worked example: from tiny batches to full ones
An (illustrative) event-ingestion service writes about 40,000 small JSON records per second across 48 partitions. The brokers are busy, request rates are high, and produce latency spikes at peak. The producer metrics show why:batch-size-avg is only a couple of kilobytes and records-per-request-avg is low, so the brokers handle a huge number of tiny requests.
The team makes three changes, one at a time, measuring after each:
- Enable lz4 compression.
compression-rate-avgshows the JSON shrinking substantially, and bytes in on the brokers drop accordingly. - Raise linger.ms to 20. Batches fill up:
batch-size-avggrows several times over, request rate on the brokers falls sharply, and broker request handler idle time rises. Producer-side latency grows by roughly the linger time, which the service can afford. - Raise batch.size to 64 KiB. Now that batches regularly hit the old 16 KiB limit, a bigger limit lets them grow further at peak.
The result is fewer, larger, compressed requests, lower broker CPU and more stable latency at peak, without touching the brokers at all. The same metrics are visible per topic in producer performance monitoring, which makes before-and-after comparisons easy.
A tuning order that works
- Measure the baseline: throughput, batch size, latency and errors.
- Turn on
lz4orzstdcompression. - Raise
linger.msstep by step and watchbatch-size-avgand end-to-end latency. - Raise
batch.sizeonce batches regularly hit the current limit. - Size
buffer.memoryfor peak bursts and confirmbufferpool-wait-ratiostays at zero. - Leave
acks=alland idempotence on unless you have a measured reason not to.
Change one setting at a time and compare the same time of day, because traffic shape changes batch behaviour. KLogic's producer performance monitoring shows throughput, batching and errors per topic, so the effect of each change is visible on the broker side too.
Tuning for low latency instead
Some workloads care more about milliseconds than throughput, such as interactive features or trading signals. The trade-offs reverse:
- Keep
linger.msat 0–1 so records are sent immediately. - Use
lz4or no compression, to minimise CPU time per batch. - Keep
acks=all; the replication wait is usually small on a healthy cluster with nearby followers. - Watch broker request queues closely, because many small requests load brokers more.
Measure end-to-end latency rather than producer latency alone, because consumer fetch settings such as fetch.min.bytes and fetch.max.wait.ms add their own delay. Our performance optimization guide covers the consumer side.
Handle errors in the send callback
send() is asynchronous: it returns before the record is written. Errors arrive later, in the callback or the returned future. A common bug is ignoring them, so records are silently lost when a topic is misconfigured or the cluster rejects them.
producer.send(record, (metadata, exception) -> {
if (exception != null) {
log.error("send failed for key {}", record.key(), exception);
failures.increment(); // alert on this counter
}
});Count failures in your own metrics as well as watching record-error-rate, and call flush() before shutting down so buffered records are not dropped. If records must not be lost, route permanent failures to a dead letter queue.
Common questions
What is a good batch.size for Kafka producers?
The default 16 KiB is conservative. High-throughput producers commonly use 64 KiB to 256 KiB, but only after raising linger.ms, because batches rarely fill without a small wait.
Does linger.ms add latency?
It adds up to linger.ms of waiting per batch, but it often lowers latency at peak by reducing the number of requests the brokers have to handle.
Which compression type should I use in Kafka?
lz4 is a strong default for speed. zstd gives the best ratio at moderate CPU cost and suits storage-heavy or cross-zone traffic. Leave the topic's compression.type as producer to avoid recompression on the broker.
Is acks=1 faster than acks=all?
Slightly, because the leader does not wait for followers. It risks losing acknowledged data during leader failover, so keep acks=all for important data and tune batching instead. See exactly-once semantics for how acks interacts with idempotence.
Why does producer send() block?
Because the buffer (buffer.memory) is full: the producer is generating data faster than brokers accept it. That is a capacity or broker-health problem; check broker saturation metrics in our JMX metrics guide.