A data processing paradigm that continuously ingests, processes, and analyzes data in real-time as it arrives, rather than waiting for batch collection.
Stream processing is the data-engineering paradigm that says: "don't wait for events to pile up into a batch — process them as they arrive." The shift matters because a growing share of real-world problems care about data freshness measured in seconds, not hours: detecting fraudulent credit-card transactions before they clear, updating a user's recommendation feed after they click a product, alerting on server errors, tracking IoT sensor anomalies, reacting to market-price moves, personalizing ad bids. Batch-first architectures literally cannot solve these problems — by the time the nightly ETL runs, the opportunity to act has passed.
A modern streaming stack has three layers: (1) transport — Apache Kafka, Amazon Kinesis, Redpanda, Google Pub/Sub, Azure Event Hubs — the durable log that producers write to and processors read from; (2) processing — Apache Flink, Kafka Streams, Spark Structured Streaming, ksqlDB, Materialize, RisingWave — the engine that runs transformations, aggregations, joins, and pattern detection on streaming data; (3) sinks — real-time databases (ClickHouse, Pinot, Druid), real-time features stores for ML, dashboards, downstream microservices. The hard conceptual parts are windowing (grouping events by time), watermarks (handling late/out-of-order data), and exactly-once semantics (each event processed exactly once despite failures).
A SaaS monitoring client ingested 3M events/sec from customer agents and needed to surface customer-specific anomalies within 10 seconds. The existing architecture was a 5-minute batch in Spark writing to ClickHouse, which meant incidents were visible 5-8 minutes after they started — way too slow for an alerting product. We rebuilt: Kafka → Flink (per-customer stateful aggregation with 30-second tumbling windows + EWMA anomaly scoring) → ClickHouse sink. End-to-end latency from event arrival to anomaly visible dropped to ~4 seconds p99. Flink worker count scaled from 0 to 80 during the ingestion, back down at quiet hours. Exactly-once semantics from Kafka transactional producer through Flink's two-phase-commit sink meant zero double-counted anomalies even through failovers. The migration took 5 months; customer churn dropped because their SRE teams finally got real-time alerts. Streaming is more complex than batch, but for the right problems it's the only answer.
Stream Processing is a data processing paradigm where data is processed continuously, record-by-record or in micro-batches, as it arrives — rather than being collected into batches and processed later. This enables real-time analytics, instant reactions to events, and sub-second data freshness.
| Aspect | Batch Processing | Stream Processing |
|---|---|---|
| Latency | Minutes to hours | Milliseconds to seconds |
| Data | Bounded (finite datasets) | Unbounded (infinite streams) |
| Processing | Process all data at once | Process one event at a time |
| Example | Nightly ETL job | Real-time fraud detection |
| Tools | Spark Batch, dbt | Kafka, Flink, Spark Streaming |
| Framework | Language | Key Strength |
|---|---|---|
| Apache Kafka Streams | Java | Lightweight, embedded in apps |
| Apache Flink | Java/Python | True streaming, stateful processing |
| Spark Structured Streaming | Python/Scala | Unified batch + streaming |
| Apache Beam | Multi | Portable across runners |
| Redpanda | C++ | Kafka-compatible, fast |
| Amazon Kinesis | Managed | AWS-native streaming |
| Materialize | SQL | Streaming SQL materializations |
Stream-processing traps that every first-time streaming team falls into:
Stream processing is a data-processing paradigm where events are processed continuously as they arrive, rather than accumulated and processed later as a batch. Each event is handled within milliseconds to seconds of its arrival. This enables use cases requiring fresh data: real-time fraud detection, live dashboards, IoT anomaly detection, personalized recommendations, CDC pipelines feeding operational datastores. Common stack: a distributed log (Kafka, Kinesis) for durable transport + a stream-processing engine (Flink, Kafka Streams, Spark Structured Streaming) for transformation + a sink (real-time DB, downstream service). The paradigm adds complexity (watermarks, windowing, exactly-once) but is the only option when batch latency is too slow.
Batch processing works on bounded (finite) datasets — you know the input has a beginning and end (today's orders, last month's logs). You collect all the data, then process it as one unit. Latency is minutes to hours; code is simpler because you have the full context. Stream processing works on unbounded (infinite) streams — events keep arriving forever. You process each event as it arrives, maintaining state across events for aggregations. Latency is sub-second; code is more complex because you never have full context. Modern architectures use both: stream processing for real-time reactions, batch for daily reconciliation, historical analytics, and ML training. The two complement rather than replace each other.
Windowing groups streaming events into time-based or count-based buckets for aggregation. Types: tumbling windows (fixed-size, non-overlapping — "every 5-minute period"); sliding windows (fixed-size, overlapping — "5-minute window sliding every 1 minute"); session windows (dynamic, gap-based — "group user activity separated by 30-minute idle gaps"); global windows (custom trigger logic). Windows let you compute real-time aggregates: "count of orders per 5-minute window," "average response time over sliding 1-hour window," "sum of spending in a user session." Combined with watermarks (handling late data), windowing is the foundation of real-time analytics on streams.
Flink is true event-by-event streaming with millisecond latency and superior state management (RocksDB, exactly-once, TB-scale state). Spark Structured Streaming uses micro-batches (100ms-seconds) and has a simpler programming model integrated with Spark's batch API. Pick Flink for: sub-second latency requirements, stateful computations, complex event-time processing, very large state. Pick Spark Structured Streaming for: teams already on Spark, unified batch + streaming in one framework, workloads tolerating seconds-scale latency, simpler operations. Flink wins on latency and state; Spark wins on ecosystem continuity. Both are production-proven; the choice is usually determined by existing team expertise and latency SLA.
Exactly-once semantics (EOS) guarantees each event affects downstream state exactly once, even through failures and retries. The three delivery semantics: at-most-once (fire-and-forget, may lose events); at-least-once (retry until delivered, may duplicate); exactly-once (guaranteed no loss, no duplication). EOS is the hardest to implement. Flink achieves it via aligned checkpointing + transactional sinks (Kafka transactional producer). Kafka Streams achieves it via transactional producers + idempotent processing. True end-to-end EOS requires cooperation from producer, processor, and sink — if any link is non-transactional (a non-idempotent HTTP API, for instance), you degrade to at-least-once. Design accordingly; don't claim EOS unless you've verified every link in the chain.
Use streaming when latency requirements force it: fraud detection (need to block a card before auth completes), real-time personalization (need to update recommendations mid-session), operational monitoring (alert within seconds of anomaly), IoT control loops (react to sensor readings in milliseconds), real-time feature stores for ML inference, and regulatory requirements (log trade data in real time). Use batch when: latency of minutes-to-hours is acceptable, data processing is naturally periodic (daily reports, weekly ML retraining), historical analysis over bounded datasets, cost matters more than freshness (streaming is ~50x more expensive per compute-hour than equivalent batch). Most mature data teams use both for different purposes — not one or the other.
Transport layer: Apache Kafka (most popular), Amazon Kinesis, Google Pub/Sub, Azure Event Hubs, Redpanda (Kafka-compatible), RabbitMQ, NATS. Processing engines: Apache Flink (leader for complex stateful streaming), Kafka Streams (lightweight Java library), Spark Structured Streaming (good for Spark-centric teams), Apache Beam (portable across runners), ksqlDB (SQL on Kafka), Materialize (streaming SQL materializations), RisingWave (streaming PostgreSQL-compatible database). Storage/sinks: ClickHouse, Apache Pinot, Apache Druid, TimescaleDB for time-series, Feast for feature stores. A typical modern stack: Debezium CDC → Kafka → Flink → ClickHouse dashboard + Kafka → downstream microservices.