Day 2: Kafka Architecture Deep Dive
๐ง Day 2: Kafka Architecture Deep Dive
Introduction:
Kafka may look simple on the surface — just topics and messages — but under the hood, it’s a highly distributed, fault-tolerant, scalable system. Understanding brokers, topics, partitions, leaders, and clusters is essential for architecting real-time systems like Swiggy, Paytm, or IRCTC.
๐ Concept: Kafka Architecture
Core Components:
| Component | Role |
|---|---|
| Broker | A Kafka server that stores data and serves clients |
| Topic | Logical channel for messages (like “orders”) |
| Partition | Subdivision of a topic for parallelism & scalability |
| Leader & Follower | Leader handles reads/writes; followers replicate data |
| Consumer Group | Set of consumers sharing a topic for load balancing |
| Cluster | Multiple brokers working together for redundancy and scale |
High-Level Flow:
-
Producers send messages to a topic
-
Topic is split into partitions
-
Each partition has a leader broker and replica followers
-
Consumers read from partitions, maintaining offsets
๐ฐ️ Before Kafka
Before Kafka, distributed event streaming looked like this:
| Method | Problem |
|---|---|
| Single MQ (RabbitMQ/ActiveMQ) | Limited throughput, no partitioning |
| DB polling | High latency, poor scalability |
| REST fan-out | Direct coupling, cascading failures |
Problem: Scaling to millions of messages per second while supporting replayable events was nearly impossible.
๐งฑ Design Patterns Used / Supported
| Pattern | How Kafka Uses It |
|---|---|
| Log-Based Storage | Immutable append-only logs for reliability and replay |
| Leader-Follower Replication | High availability using ISR (In-Sync Replicas) |
| Partitioning & Sharding | Distributes load across brokers for parallel processing |
| Observer / Pub-Sub | Producers publish, consumers subscribe independently |
| Distributed Consensus | Ensures consistency across replicas (ZooKeeper or KRaft) |
๐ Alternatives
| Tool | Use Case | Why Not Kafka |
|---|---|---|
| RabbitMQ | Reliable, smaller queues | Lacks partitioning & high throughput |
| ActiveMQ | Enterprise-grade messaging | Less scalable for massive streams |
| AWS Kinesis | Cloud streaming | Vendor lock-in; lower flexibility |
| Pulsar | Multi-tenancy, geo-replication | Smaller ecosystem; more complex ops |
| Redis Streams | Lightweight streaming | Not fault-tolerant at large scale |
๐ซ When Not to Use Kafka
-
You don’t need partitioning or massive parallelism
-
Low-volume messaging system (<1000 messages/sec)
-
Simpler point-to-point communication suffices
-
You cannot manage cluster operations
๐งฉ Indian Real-Life Analogy: Indian Railways Network
| Kafka Component | Analogy |
|---|---|
| Broker | Train engine running a station (manages data flow) |
| Topic | Train route (e.g., “Mumbai to Delhi”) |
| Partition | Train compartments (parallel processing) |
| Leader | Main engine of compartment (writes & reads) |
| Follower | Coupled engines replicating data for safety |
| Cluster | Entire train network (multiple stations & engines) |
| Consumer Group | Passengers distributed among compartments (load-balanced consumption) |
๐ป Quick Code Snippets
Java: Creating a Topic (Admin API)
Node.js: Consumer Group Example
๐ฎ Future Improvements & Trends
| Feature | Benefit |
|---|---|
| KRaft Mode | No ZooKeeper, simpler cluster management |
| Tiered Storage | Cost-effective storage for long-term retention |
| Dynamic Partition Reassignment | Easy scaling & rebalancing |
| Cluster Metadata APIs | Better monitoring & tooling support |
๐ฏ Architect-Level Takeaways
-
Kafka’s distributed architecture ensures high availability and fault tolerance
-
Partitions allow horizontal scaling and parallel processing
-
Leader-follower replication ensures durability and resilience
-
Cluster design is critical — affects throughput, latency, and availability
-
Consumer groups allow independent scaling of services
๐ง Interview Prep Questions
-
What is a Kafka broker, and how does it differ from a topic?
-
How do partitions improve scalability?
-
What is the role of leader and follower replicas?
-
Explain the concept of consumer groups and offsets.
-
How does Kafka achieve fault tolerance?
Comments
Post a Comment