Why Kafka Exists & Where It Fits in Microservices
๐ง Day 1: Why Kafka Exists & Where It Fits in Microservices
Introduction:
Imagine you’re designing Swiggy. Orders, payments, notifications, delivery updates — all happening simultaneously.
Without Kafka, services talk directly via APIs → tight coupling, slow failures, and retry storms.
Kafka solves this with a real-time, event-driven backbone.
๐ Concept: What Kafka Does
Kafka is a distributed streaming platform that lets services publish and subscribe to event streams asynchronously.
Key Features:
-
High-throughput
-
Fault-tolerant
-
Replayable events
๐ฐ️ Before Kafka: The Old World
| Method | Problems |
|---|---|
| REST APIs between services | Tight coupling, cascading failures |
| Shared relational DB polling | High latency, inefficiency |
| MQs (RabbitMQ, ActiveMQ) | Limited throughput, no event replay |
๐งฑ Kafka & Design Patterns
| Pattern | How Kafka Implements / Enables It |
|---|---|
| Event-Driven Architecture | Services communicate asynchronously via events |
| Pub/Sub | Producers publish; multiple consumers subscribe independently |
| Event Sourcing | State is built from event logs |
| CQRS | Write and read sides separated using topics |
| Saga Pattern | Distributed workflows coordinated via events |
๐ Alternatives
| Tool | Use Case | Why Not Kafka |
|---|---|---|
| RabbitMQ | Reliable queueing, transactional | Lower throughput, simpler workflows |
| ActiveMQ | Moderate enterprise messaging | Less scalable |
| AWS SQS/SNS | Serverless queueing | No streaming/replay |
| Redis Streams | Lightweight pub/sub | Not for distributed scale |
| Pulsar | Multi-tenancy, low latency | Newer ecosystem, smaller community |
๐ซ When Not to Use Kafka
-
Low message volume systems
-
Simple request-response communication
-
Systems needing strict per-message ack
-
Short-lived transactional workflows
๐งฉ Indian Real-Life Analogy: Indian Railways
| Kafka Component | Analogy |
|---|---|
| Producer | Passenger boarding station (Order Service) |
| Topic | Train route (Orders, Payments) |
| Partition | Train compartments (Parallel processing) |
| Broker | Train engine (Kafka broker) |
| Consumer | Destination station (Analytics, Delivery) |
| Cluster | Train network (Multiple brokers for scaling) |
๐ป Quick Code Snippets
Java Producer
Node.js Consumer
๐ฎ Future Improvements
| Feature | Benefit |
|---|---|
| KRaft Mode | Simplifies cluster management (no ZooKeeper) |
| Tiered Storage | Store months of events cost-effectively |
| Kafka Streams API | Real-time transformations without Spark |
| Schema Registry | Structured data validation across teams |
๐ฏ Architect-Level Takeaways
-
Kafka enables loosely coupled, scalable microservices
-
Implements log-based, immutable design
-
Supports Event Sourcing, CQRS, Saga patterns
-
Allows event replay and asynchronous processing
-
Requires careful ops for cluster management
๐ง Interview Prep Questions
-
Why Kafka instead of REST APIs?
-
How does Kafka decouple microservices?
-
Difference between Kafka & RabbitMQ?
-
Which design patterns does Kafka enable?
-
Real-world Indian startup example using Kafka?
Comments
Post a Comment