🛠️ Real-World Kafka — Hands-On

Real-World Kafka — Hands-On: An event-driven e-commerce pipeline built on Kafka is like the dispatch coordination system at a large logistics company: a customer placing an order

An event-driven e-commerce pipeline built on Kafka is like the dispatch coordination system at a large logistics company: a customer placing an order is equivalent to a package entering the depot — the intake system logs it once, and then warehouse, shipping, billing, and customer-service departments all receive their own notification and process it independently, at their own pace, without any department waiting for another to finish first. The question worth sitting with before you build this is: why not just have the OrderService call the InventoryService, PaymentService, and NotificationService directly in one synchronous chain? Because synchronous chains fail as a unit — if the notification email provider is down at 11 PM on Black Friday, the entire checkout fails for every customer. In Java terms, the synchronous chain is like calling five `@Service` methods in a single `@Transactional` block: if the fifth call throws, the whole transaction rolls back. The Kafka pipeline is the opposite: each service consumes its own topic and owns its own retry logic — the checkout succeeds even if the email system is temporarily unavailable, because the notification consumer will process the event as soon as the email provider recovers. For QA, this architecture creates a specific testing challenge: verifying the entire order flow now requires subscribing to multiple topics and asserting that all downstream services received and processed their events correctly — a synchronous API test that only checks the HTTP 201 response is no longer sufficient.

Scenario: E-Commerce Order Processing Pipeline

Order event flows through 4 services via Kafka

Notification Service

Step 1: Start Kafka with Docker Compose

Why Are Three Different Topics Created With Different Partition Counts?

"orders" and "payments" topics are…

"orders" and "payments" topics are created with 3 partitions — this allows high-volume streams to be processed in parallel (3 consumers can run at once).

"orders-failed" is created with just 1…

"orders-failed" is created with just 1 partition — the error stream is low-volume, and ORDERED processing (all failures in one place, chronological) matters more than parallelism.

Every --create command uses…

Every `--create` command uses `--replication-factor 1` — this is for LOCAL development only; in production this must be at least 3 (so a single broker loss doesn't lose data).

The --list command confirms all three…

The `--list` command confirms all three topics actually exist — the produce/consume simulation in the next step depends on this verification.