Stream Processing is a data processing paradigm where data is processed continuously, record-by-record or in micro-batches, as it arrives — rather than being collected into batches and processed later. This enables real-time analytics, instant reactions to events, and sub-second data freshness.
Batch vs Stream Processing
| Aspect | Batch Processing | Stream Processing |
|--------|-----------------|-------------------|
| Latency | Minutes to hours | Milliseconds to seconds |
| Data | Bounded (finite datasets) | Unbounded (infinite streams) |
| Processing | Process all data at once | Process one event at a time |
| Example | Nightly ETL job | Real-time fraud detection |
| Tools | Spark Batch, dbt | Kafka, Flink, Spark Streaming |
Core Concepts
Events
The fundamental unit of data in streaming:
- An event is an immutable fact that something happened
- Events have a timestamp, key, and value
- Examples: user clicked, order placed, sensor reading
Windowing
Grouping events by time for aggregation:
- Tumbling Window: Fixed-size, non-overlapping (every 5 minutes)
- Sliding Window: Fixed-size, overlapping (5 min window, sliding every 1 min)
- Session Window: Dynamic, gap-based (group events within 30 min of activity)
Watermarks
Handling late-arriving data:
- A watermark is a timestamp that says 'all events before this time have arrived'
- Allows the system to close windows and emit results
- Late data can still be handled through allowed lateness
Exactly-Once Semantics
Guaranteeing each event is processed exactly once:
- At-most-once: Fire and forget (may lose data)
- At-least-once: Retry on failure (may duplicate)
- Exactly-once: Most difficult, guaranteed no loss or duplication
Stream Processing Frameworks
| Framework | Language | Key Strength |
|-----------|----------|-------------|
| Apache Kafka Streams | Java | Lightweight, embedded in apps |
| Apache Flink | Java/Python | True streaming, stateful processing |
| Spark Structured Streaming | Python/Scala | Unified batch + streaming |
| Apache Beam | Multi | Portable across runners |
| Redpanda | C++ | Kafka-compatible, fast |
| Amazon Kinesis | Managed | AWS-native streaming |
| Materialize | SQL | Streaming SQL materializations |
Common Use Cases
1. Fraud Detection: Score transactions in real-time
2. IoT Analytics: Process sensor data as it arrives
3. Real-Time Dashboards: Live business metrics
4. Event-Driven Architecture: React to business events instantly
5. CDC Processing: Stream database changes to downstream systems
6. Personalization: Real-time user behavior for recommendations