Apache Flink is a distributed processing engine for stateful computations over both bounded (batch) and unbounded (streaming) data. Unlike Spark's micro-batch approach, Flink processes each event individually as it arrives, achieving true millisecond-level latency.
Architecture
``
┌─────────────────────────────────────────────┐
│ Flink Cluster │
│ ┌──────────┐ ┌────────────────────────┐ │
│ │JobManager│──→│ TaskManagers │ │
│ │(Master) │ │ ┌──────┐ ┌──────┐ │ │
│ │ │ │ │Slot 1│ │Slot 2│ ... │ │
│ └──────────┘ │ └──────┘ └──────┘ │ │
│ └────────────────────────┘ │
└─────────────────────────────────────────────┘
Core Concepts
Event Time vs Processing Time
Flink distinguishes between:
- Event Time: When the event actually occurred (embedded in the data)
- Processing Time: When Flink processes the event
- Ingestion Time: When the event enters Flink
This matters for handling late-arriving data correctly.
State Management
Flink's killer feature is its state management:
- Keyed State: State partitioned by key (e.g., per-user counters)
- Operator State: State per parallel operator instance
- State Backends: RocksDB (disk) or HashMaps (memory)
- Checkpointing: Automatic, consistent snapshots for fault tolerance
Windowing
Group events into time-based or count-based windows:
- Tumbling Windows: Fixed-size, non-overlapping (every 5 minutes)
- Sliding Windows: Fixed-size, overlapping (5-min window every 1 minute)
- Session Windows: Dynamic, gap-based (close after 30min idle)
- Global Windows: Custom trigger logic
Flink APIs (High to Low Level)
| API | Level | Use Case |
|-----|-------|----------|
| Flink SQL | Highest | SQL queries on streams |
| Table API | High | Relational operations |
| DataStream API | Low | Custom stream processing |
| ProcessFunction | Lowest | Full control over time and state |
Use Cases
1. Fraud Detection: Sub-second pattern matching on transactions
2. Real-time Recommendations: Update user profiles in real-time
3. IoT Analytics: Process millions of sensor events per second
4. CDC Processing: React to database changes instantly
5. Ad-tech Bidding: Make bid decisions in milliseconds