🏗️ Kafka Architecture

Kafka Architecture: Kafka's architecture works like a distributed library system with multiple branches: each branch (broker) holds a copy of certain book sections (partition rep

Kafka's architecture works like a distributed library system with multiple branches: each branch (broker) holds a copy of certain book sections (partition replicas), every section has numbered shelves (offsets) and a librarian who tracks which shelf each reading group is currently on (consumer offset). The critical insight to sit with before moving on: why is it a topic split into partitions rather than one giant storage area? Because a single partition is a single sequential write lane — splitting a high-volume topic into 12 partitions means 12 consumers can read in parallel without ever blocking each other, scaling throughput linearly with partition count. In Java terms, picture a ConcurrentHashMap where each bucket is a Kafka partition: just as the map shards keys across buckets to avoid lock contention, Kafka shards events across partitions to avoid consumer bottlenecks. For QA engineers, the architecture decision has a direct test consequence: if your test consumer always assigns itself to partition 0, it will miss events that landed on partition 7 — a silent, hard-to-reproduce gap that only appears under real production load when the partition key hashing distributes events non-uniformly.

Kafka Cluster Architecture

Core Kafka Components

🎬 When the Leader Crashes: Automatic Failover

For an "orders" partition, Broker 1 is the leader — it handles all writes and reads. Brokers 2 and 3 are just replicating followers.

The Producer writes to Broker 1. Broker 1 mirrors the same bytes to Broker 2 and Broker 3 — just like a RAID array keeps every disk in sync.

The Consumer also reads from Broker 1 — the system runs healthy, Brokers 2 and 3 stay quietly in sync.

Suddenly Broker 1 crashes — hardware failure, OOM, doesn't matter. The machine that Producer and Consumer were talking to is simply gone.

The Controller detects the loss of Broker 1 within seconds and elects Broker 2 from the ISR (in-sync replica) list as the new leader.

Final — thanks to the Kafka client library, Producer and Consumer are automatically redirected to the new leader (Broker 2). Because the followers were in sync, NOT A SINGLE message is lost.

When a Broker Crashes, Who Takes Over?

Every partition has one leader broker…

Every partition has one leader broker — all reads/writes go through it, the others are just replicating followers.

The leader instantly mirrors every…