What is Apache Flink?

Apache Flink is the stream-processing engine that takes streaming seriously. Unlike Spark Streaming's micro-batch model (collect events for 100ms-few-seconds, then process as a tiny batch), Flink processes each event individually as it arrives — true event-at-a-time streaming with millisecond latency. Combined with the best state management in the open-source ecosystem (RocksDB-backed keyed state scaling to TB per worker), exactly-once semantics, and mature event-time processing with watermarks, Flink is the standard choice for any non-trivial streaming workload.

When Flink is the right answer (and when it's overkill)

Flink shines for: real-time fraud/anomaly detection, complex event processing (CEP), sessionization of user activity, stream-stream joins over time windows, real-time feature stores for ML, and any workload where latency below 1 second matters. It's overkill for: simple ETL-ish transformations where 5-minute latency is fine (use dbt or Spark batch), pure fan-out with no transformation (Kafka Streams or Connect), and teams without Java/Scala/Python expertise willing to learn Flink's conceptual model (event time, watermarks, state backends, checkpointing). Flink's power comes with genuine complexity.

Field story

An e-commerce client built real-time personalization: for every product-page view, enrich with the user's last-30-days purchase history and last-hour browsing behavior, score against a live model, and return recommendations within 200ms. The naïve Redis-based architecture fell over at 80k events/sec; every event needed a Redis lookup with network roundtrip. We rebuilt on Flink: user state (purchase history + browsing session) kept in keyed RocksDB state locally to the worker processing that user's events, no Redis roundtrip needed. End-to-end latency dropped from 450ms p99 to 65ms p99, and throughput scaled linearly to 400k events/sec on 20 workers. State size per worker reached 120GB, checkpointed every 90 seconds to S3. The rebuild took four months; the performance improvement and cost savings (no Redis cluster) paid back in six. Flink isn't free to learn, but the payoff for the right problem is enormous.

Apache Flink is a distributed processing engine for stateful computations over both bounded (batch) and unbounded (streaming) data. Unlike Spark's micro-batch approach, Flink processes each event individually as it arrives, achieving true millisecond-level latency.

Architecture

┌─────────────────────────────────────────────┐
│              Flink Cluster                  │
│  ┌──────────┐   ┌────────────────────────┐  │
│  │JobManager│──→│     TaskManagers       │  │
│  │(Master)  │   │ ┌──────┐ ┌──────┐     │  │
│  │          │   │ │Slot 1│ │Slot 2│ ... │  │
│  └──────────┘   │ └──────┘ └──────┘     │  │
│                  └────────────────────────┘  │
└─────────────────────────────────────────────┘

Core Concepts

Event Time vs Processing Time

Flink distinguishes between:

• Event Time: When the event actually occurred (embedded in the data)
• Processing Time: When Flink processes the event
• Ingestion Time: When the event enters Flink

This matters for handling late-arriving data correctly.

State Management

Flink's killer feature is its state management:

• Keyed State: State partitioned by key (e.g., per-user counters)
• Operator State: State per parallel operator instance
• State Backends: RocksDB (disk) or HashMaps (memory)
• Checkpointing: Automatic, consistent snapshots for fault tolerance

Windowing

Group events into time-based or count-based windows:

• Tumbling Windows: Fixed-size, non-overlapping (every 5 minutes)
• Sliding Windows: Fixed-size, overlapping (5-min window every 1 minute)
• Session Windows: Dynamic, gap-based (close after 30min idle)
• Global Windows: Custom trigger logic

Flink APIs (High to Low Level)

Use Cases

1. Fraud Detection: Sub-second pattern matching on transactions
2. Real-time Recommendations: Update user profiles in real-time
3. IoT Analytics: Process millions of sensor events per second
4. CDC Processing: React to database changes instantly
5. Ad-tech Bidding: Make bid decisions in milliseconds

Flink pitfalls from production deployments:

• Watermark strategy too aggressive. "All events within 1 second" misses the last 5% of real events that show up late, silently dropping them from windowed aggregations. Start with a generous watermark (1-5 minutes) and tighten only after measuring out-of-orderness in production.
• Forgetting setUidHash on operators. Without stable UIDs on operators, savepoints can't map old state to new operators after a topology change. Always assign UIDs to every operator (.uid("user-aggregator")) from day one.
• Default parallelism too high or too low. Setting parallelism matching CPU cores sounds right until you have more partitions than cores (under-utilized) or fewer (oversubscribed and slow). Tune per-operator; consider that keyed operations are bounded by max parallelism, which is set at job creation and hard to change.
• Java state serialization overhead. Default Kryo serializer is slow. For hot state types, register specific TypeInformation or use POJOs with Flink's built-in PojoSerializer. Saves significant CPU on high-throughput keyed state.
• Checkpoint storms blocking progress. If checkpoint duration > checkpoint interval, new checkpoints can't start. Symptom: pipeline throughput collapses. Enable incremental RocksDB checkpointing, increase checkpoint interval, or scale workers.
• PyFlink performance differences from Java. The Python API is more ergonomic but ~2-5x slower and more memory-intensive due to cross-JVM serialization. For latency-sensitive pipelines, prefer Java/Scala.
• Flink SQL looks like SQL but isn't. GROUP BY user_id on an unbounded stream produces a continuously-updating changelog, not a one-shot result. Teams with SQL-only background get confused by CDC semantics and retraction streams. Read the Flink SQL CDC docs before deploying.
• Stopping a job without a savepoint. cancel kills the job and its state; stop-with-savepoint gracefully stops and persists state for later restart. Using cancel in CI-automation has lost many TB of state.
• State TTL mis-applied. Setting TTL on keyed state expires entries but doesn't expire the map itself; state size can grow if you add keys but never remove them. Combine TTL with explicit state cleanup.
• Assuming exactly-once applies to external sinks. Flink guarantees EOS for internal state and checkpointed outputs (Kafka transactional producer, TwoPhaseCommitSinkFunction). Writing to a non-transactional sink (REST API, non-idempotent DB insert) degrades to at-least-once. Plan idempotency accordingly.