A distributed event streaming platform used for building real-time data pipelines and streaming applications, handling trillions of events per day.
Apache Kafka is the distributed log that ate the streaming world. Originally built at LinkedIn around 2011 to solve their cross-system data-integration nightmare, Kafka's core abstraction — an append-only, partitioned, replicated log that producers write to and multiple consumer groups read from independently — turned out to be the right primitive for an enormous range of problems: event-driven microservices, change data capture, real-time analytics, log aggregation, metrics pipelines, and more. If you're building any system where multiple services need to react to the same stream of facts, Kafka is almost certainly in the picture.
Kafka isn't a message queue in the RabbitMQ sense. It's a durable, replay-able log. Messages stay around (days, weeks, or forever with log compaction) after consumption. Consumers track their own offset — their position in the log — so two independent consumer groups can read the same topic at different speeds without affecting each other. Topics are split into partitions; each partition is ordered but the topic as a whole isn't. Keys determine which partition a message lands in, so "all events for user-42" stay ordered on one partition even while the topic as a whole is parallelized across dozens. Replication (usually 3x) across brokers gives durability even when a broker crashes.
A fintech client had a classic "database-of-doom" architecture: the mobile app wrote to Postgres, and 14 downstream systems (fraud scoring, notifications, risk engine, data warehouse, reporting) each polled Postgres every 30 seconds for new rows. Postgres CPU ran at 85% all day just serving reads, and a single slow consumer could back up replication. We inserted Kafka as a CDC buffer using Debezium → Kafka → 14 consumer groups. Each consumer moved at its own pace, Postgres CPU dropped to ~25%, and adding a 15th consumer (an ML feature pipeline) took 20 minutes instead of a two-week migration. The caveat: we spent six months operationally maturing — partition strategy, monitoring consumer lag, handling offset-commit-failures, wiring schema registry so producers couldn't ship breaking changes. Kafka itself is simple; operating Kafka well is not.
Apache Kafka is a distributed event streaming platform capable of handling trillions of events per day. Originally developed at LinkedIn, Kafka is now used by thousands of companies for real-time data pipelines, streaming analytics, and event-driven architectures.
Producers → [Topic: orders] → Consumers
│
┌─────┴─────┐
│ Partition │
│ 0 │
├───────────┤
│ Partition │
│ 1 │
├───────────┤
│ Partition │
│ 2 │
└───────────┘Kafka operational pitfalls that bite every team eventually:
partitions=200 on every topic.auto.offset.reset=latest in production. A consumer that restarts with no committed offset will skip all backlog. Use earliest for most batch-like consumers; reserve latest for real-time-only use cases where missed events are acceptable.enable.idempotence=true), transactions for multi-partition writes, and transactional consumer reads (isolation.level=read_committed). Misconfigure any piece and you're back to at-least-once.batch.size and linger.ms to aggregate small writes.Kafka is used as a durable, distributed event log that decouples producers from consumers. Primary use cases: (1) event-driven microservices architectures where many services react to the same events; (2) change data capture (CDC) — Debezium reads database changes into Kafka, many downstream consumers react; (3) real-time analytics pipelines — Kafka → Flink/Spark Streaming → warehouse; (4) log aggregation — collecting logs from thousands of servers; (5) metrics pipelines; (6) event sourcing — Kafka as the source-of-truth log for application state. Essentially: any time multiple systems need to react to the same stream of facts, Kafka is the default choice for the backbone.
Kafka can fill the role of a message queue but is more accurately a distributed log. Key differences from traditional queues (RabbitMQ, SQS): messages persist after consumption for configurable retention; multiple independent consumer groups can read the same topic at different speeds; consumers manage their own offset so they can replay history; and ordering is guaranteed per-partition, not across the topic. For pure work-queue semantics (one consumer takes a job, job disappears), traditional queues are simpler. For fan-out, replay, high-throughput, and multi-consumer scenarios, Kafka's log model is the right abstraction.
A topic is a named stream of messages (e.g., `orders`, `user-clicks`, `payment-events`). A partition is a subdivision of a topic — each topic has one or more partitions. Each partition is an ordered, immutable sequence of messages stored on a specific broker and replicated for durability. Partitioning enables parallelism: a topic with 12 partitions can have up to 12 consumers in a consumer group reading in parallel. Messages with the same key always go to the same partition (via `hash(key) % partitions`), preserving per-key ordering. Partition count is the primary tuning knob for throughput and consumer scalability.
A consumer group is a set of consumer processes that cooperate to read a topic. Kafka distributes the topic's partitions across the group's consumers — each partition is read by exactly one consumer in the group. If you have 12 partitions and 4 consumers, each consumer handles 3 partitions. Add a fifth consumer and Kafka rebalances so each reads 2-3 partitions. Remove a consumer and its partitions are redistributed. Multiple groups can subscribe to the same topic independently — group A (analytics pipeline) and group B (notification service) each see every message. Consumer groups are how you scale stream processing horizontally.
Learning basic producer/consumer usage takes an afternoon. Mastering Kafka's full surface area (consumer groups, rebalancing, offsets, exactly-once semantics, schema evolution, compaction, KRaft/ZooKeeper, MirrorMaker) takes months. Operating a production Kafka cluster responsibly is a specialty role — upgrades, rolling restarts, rebalances, capacity planning, monitoring dozens of metrics (consumer lag, under-replicated partitions, request latencies, disk usage). Small teams should use managed Kafka (Confluent Cloud, AWS MSK, Redpanda Cloud, Aiven) to avoid the operational burden. Only self-host when scale justifies a dedicated platform team.
Kafka is a distributed log optimized for high-throughput, replayable, multi-consumer workloads — it keeps messages around and lets consumers seek to any offset. RabbitMQ is a traditional message broker optimized for flexible routing (exchanges, bindings, routing keys), per-message delivery semantics, and queue-based work distribution. Kafka wins on throughput (millions of msgs/sec per broker), retention, and replay. RabbitMQ wins on flexible routing, priority queues, and operational simplicity at small scale. Use Kafka for event streaming and CDC; RabbitMQ for task queues, RPC, and complex routing. Many companies use both — they solve different problems.
Kafka Connect is a framework for moving data between Kafka and other systems via pre-built connectors (Debezium for CDC, JDBC for databases, S3 sink for data lake, Elasticsearch, etc.) — you configure YAML instead of writing code. Kafka Streams is a Java library embedded in your application for processing Kafka topics (filter, map, aggregate, join) with exactly-once semantics and state management; no separate cluster required. ksqlDB is a streaming SQL engine built on Kafka Streams — write SQL queries against topics as if they were tables. Together they cover most streaming use cases without Flink or Spark, though Flink is often preferred for complex event-time processing and larger state.