🚀 Setup Guide

Kafka Cluster Setup Guide

Complete guide to setting up production-ready Apache Kafka clusters with best practices for configuration, security, monitoring, and scaling from single-node to multi-datacenter deployments.

Updated: August 3, 2025 • 40 min read • Production Guide

Prerequisites & Planning

Essential requirements before setting up your Kafka cluster

Hardware Requirements

CPU: 8+ cores per broker

RAM: 32GB+ (6GB heap)

Storage: SSD with 10K+ IOPS

Network: 10Gbps recommended

Software Dependencies

Java: OpenJDK 11 or 17

Kafka: 3.5+ (latest stable)

ZooKeeper: 3.8+ or KRaft mode

OS: Linux (Ubuntu/RHEL)

Network Architecture

Topology: Multi-AZ deployment

DNS: FQDN for each broker

Ports: 9092, 9093 (SSL)

Load Balancer: For client access

Step-by-Step Installation

Complete installation process for production Kafka cluster

System Preparation

Prepare the operating system and install required dependencies for optimal Kafka performance.

OS Configuration

# Update system

sudo apt update && sudo apt upgrade -y

# Install Java 17

sudo apt install openjdk-17-jdk -y

# Verify Java installation

java -version

System Limits

# Edit /etc/security/limits.conf

kafka soft nofile 65536

kafka hard nofile 65536

# Edit /etc/sysctl.conf

vm.swappiness=1

vm.dirty_ratio=80

vm.dirty_background_ratio=5

Kafka Installation

Download and install Apache Kafka with proper directory structure and permissions.

# Create kafka user and directories

sudo useradd -r -s /bin/false kafka

sudo mkdir -p /opt/kafka /var/log/kafka /var/lib/kafka

# Download and extract Kafka

wget https://downloads.apache.org/kafka/2.13-3.5.0.tgz

sudo tar -xzf kafka_2.13-3.5.0.tgz -C /opt/kafka --strip-components=1

# Set ownership

sudo chown -R kafka:kafka /opt/kafka /var/log/kafka /var/lib/kafka

ZooKeeper Configuration

Configure ZooKeeper ensemble for high availability and data consistency.

ZooKeeper Properties

# /opt/kafka/config/zookeeper.properties

dataDir=/var/lib/zookeeper

clientPort=2181

maxClientCnxns=0

admin.enableServer=true

admin.serverPort=8080

# Cluster configuration

server.1=zk1.example.com:2888:3888

server.2=zk2.example.com:2888:3888

server.3=zk3.example.com:2888:3888

Performance Tuning

# JVM settings

-Xms1G -Xmx1G

-XX:+UseG1GC

-XX:MaxGCPauseMillis=20

# ZooKeeper tuning

tickTime=2000

initLimit=10

syncLimit=5

autopurge.snapRetainCount=3

autopurge.purgeInterval=24

Kafka Broker Configuration

Configure Kafka brokers with production-ready settings for performance and reliability.

# /opt/kafka/config/server.properties

# Broker Configuration

broker.id=1

listeners=PLAINTEXT://kafka1.example.com:9092,SSL://kafka1.example.com:9093

advertised.listeners=PLAINTEXT://kafka1.example.com:9092,SSL://kafka1.example.com:9093

listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL

# Log Configuration

log.dirs=/var/lib/kafka/logs

num.network.threads=8

num.io.threads=16

socket.send.buffer.bytes=102400

socket.receive.buffer.bytes=102400

socket.request.max.bytes=104857600

# Topic Configuration

num.partitions=3

default.replication.factor=3

min.insync.replicas=2

unclean.leader.election.enable=false

Production Configuration

Essential configuration parameters for production deployments

Performance Optimization

JVM Settings

-Xms6g -Xmx6g

-XX:+UseG1GC

-XX:MaxGCPauseMillis=20

-XX:G1HeapRegionSize=16m

Throughput Settings

num.replica.fetchers4

replica.fetch.max.bytes1048576

Reliability & Durability

Replication Settings

default.replication.factor3

min.insync.replicas2

Safety Settings

unclean.leader.electionfalse

auto.create.topicsfalse

Monitoring & Alerting Setup

Essential monitoring configuration for production clusters

Key Metrics to Monitor

Under-Replicated Partitions

Critical indicator of cluster health issues

Request Latency

Producer/consumer request processing time

Disk Usage

Log segment storage and retention monitoring

Alerting Thresholds

Critical AlertsImmediate

Under-replicated partitions > 0, Offline partitions > 0

Warning Alerts15 min

Disk usage > 85%, Request latency > 100ms

Info Alerts1 hour

High throughput, Rebalancing events

Security Hardening

Essential security measures for production Kafka clusters

Authentication

Enable SASL/SCRAM

Configure mTLS

Strong password policies

Authorization

Configure ACLs

Principle of least privilege

Regular access reviews

Network Security

Enable TLS encryption

Network segmentation

Firewall configuration

Common Issues & Troubleshooting

Solutions to frequent setup and configuration problems

Broker Startup Issues

Common Symptoms

• Broker fails to start

• Cannot connect to ZooKeeper

• Port binding errors

Solutions

Check Java version compatibility

Verify ZooKeeper connectivity

Review firewall settings

Performance Issues

Performance Indicators

• High request latency

• Low throughput

• Frequent GC pauses

Optimization Steps

Tune JVM heap size

Optimize disk I/O settings

Adjust network buffer sizes

Simplify Kafka Cluster Management

Skip the complex setup and management overhead. KLogic provides automated Kafka cluster provisioning, configuration management, and comprehensive monitoring out of the box.

Try Managed Kafka Learn Monitoring Basics

Free 14-day trial • Automated setup • Production-ready configuration • 24/7 monitoring