🗂️ Topics & Partitions Deep Dive

Topics & Partitions Deep Dive: Kafka CLI — create and manage topics How to Read Leader/Replica/ISR in the --describe Output When --create --partitions 3… When `--create --partiti

Kafka CLI — create and manage topics

How to Read Leader/Replica/ISR in the --describe Output

When --create --partitions 3…

When `--create --partitions 3 --replication-factor 3` runs, Kafka picks 1 leader + 2 follower brokers for each partition — this choice isn't random, it's spread evenly across the cluster.

The "Partition: 0 Leader: 1" line in the `--describe` output shows EVERY message written to that partition goes through broker 1 — the other brokers only replicate.

"Replicas: 1,2,3" shows which brokers the partition is COPIED to, "Isr: 1,2,3" shows which of those copies are actually UP TO DATE (in-sync) — the two lists can differ.

If a broker falls behind…

If a broker falls behind, it drops out of the ISR list — meaning the `min.insync.replicas` threshold is now met by fewer brokers, and this state should be monitored.

Replication — Fault Tolerance

Replication in Kafka works like a RAID array for event streams: every partition has one leader broker (the one that accepts writes and serves reads) and N-1 follower brokers that continuously mirror every byte — if the leader machine dies, one follower is elected leader in seconds with zero message loss. But here is the question worth sitting with: if the cluster already survives with replication-factor=2, why does every production guide insist on 3? Because with replication-factor=2, one broker failure leaves you with a single copy and no redundancy — the next failure triggers data loss, and Kafka is specifically designed to make data loss impossible even across rolling restarts and hardware failures. In Java terms, replication is analogous to a `ReentrantReadWriteLock` with write mirroring: writes are acknowledged only when all `min.insync.replicas` confirm receipt, giving you the same durability guarantee that a database transaction gives you with `acks=all`. For QA, the operational implication is direct: if your staging environment runs a single-broker Kafka (replication-factor=1) but your production runs replication-factor=3, performance tests and reliability checks are measuring a completely different system — a classic "worked in staging, failed in production" root cause that is almost always traced back to replication mismatch.

With cleanup.policy=compact, Kafka keeps only the LATEST message per key. Useful for "current state" topics: user profile updates, config changes. Example: 5 updates to user-123 profile → compacted topic keeps only the latest. This makes Kafka behave like a key-value store.

Produce and consume from command line (for testing)

🎬 Is acks=all Enough? The min.insync.replicas Trap

The Producer sends a payment event with acks=all and min.insync.replicas=2 — this looks like the safest possible setting combination.