A stream processing technique where the application remembers information across multiple events in time.
Stateful processing is where streaming gets genuinely hard. Stateless streams (filter out bad rows, transform JSON into Parquet, route messages to topics) are easy — each event is independent. Stateful streams (running totals per user, sessionization, fraud pattern detection, deduplication, joins over time windows) require remembering what happened in prior events. In a single-process program this is just a variable in memory. In a distributed system processing millions of events/sec across dozens of workers, "a variable" becomes a hard distributed-systems problem: where is state stored? how does it survive worker crashes? how does it move when you scale out? how do you get exactly-once without double-counting?
Kafka Streams, Apache Flink, and Spark Structured Streaming all solve stateful processing — each with its own trade-offs. Flink is widely considered best-in-class: it stores state in a local RocksDB instance (fast), checkpoints snapshots to S3 every few seconds (durable), and during worker rebalances moves state alongside the data (consistent). Kafka Streams embeds RocksDB per-instance and uses a Kafka topic (the changelog) as the durable backing store. Spark uses checkpoints to durable storage. The frameworks differ meaningfully in latency (Flink processes event-by-event; Spark micro-batches every 100ms-few-seconds), state size they can handle (Flink scales to TB of state per worker; Spark is more memory-constrained), and API ergonomics.
A payments client needed real-time fraud detection: for each transaction, check whether the user had more than 5 transactions in the last 15 minutes, whether the merchant category had changed suddenly, whether the geographic distance from the last transaction was impossible given elapsed time. All stateful — all require keeping per-user state updated event-by-event. They started with Spark Streaming micro-batching every 30 seconds, which meant fraud was caught 30 seconds late — unacceptable for card-present transactions. We migrated to Flink, keyed on user_id, and stored the last 24 hours of per-user transaction history in keyed state backed by RocksDB. End-to-end latency dropped from 30s to 80ms, throughput held at 12k events/sec per worker, and checkpoints to S3 every 60 seconds gave us crash recovery with no double-processing. State size per worker grew to about 40GB — Flink handled it without noticeable impact.
Stateful Processing is a stream processing paradigm where the application maintains and updates internal state across multiple events. Unlike stateless operations (filtering, mapping) that process each event independently, stateful operations (running sums, session windows, pattern detection) need to remember what happened in previous events to produce correct results.
| Aspect | Stateless | Stateful |
|---|---|---|
| Memory | No memory of past events | Remembers past events |
| Example | Filter events where amount > 100 | Running total of amounts per user |
| Complexity | Simple | Complex (state management) |
| Scaling | Easy (just add workers) | Hard (state must move with data) |
| Recovery | Instant (no state to restore) | Requires checkpointing |
In a single-machine program, state is just a variable. In a distributed streaming system processing millions of events per second across dozens of machines, state becomes the hardest problem:
| Backend | Speed | Capacity | Best For |
|---|---|---|---|
| In-Memory (HashMap) | Fastest | Limited by RAM | Small state, low latency |
| RocksDB | Fast (SSD) | Very large (TB+) | Large state, production workloads |
| Remote (Redis) | Network latency | Unlimited | Shared state across applications |
Checkpointing periodically saves a consistent snapshot of all state to durable storage (S3, HDFS):
Normal processing:
Events → [State: counter=42] → Output
Checkpoint taken:
State snapshot → S3 (counter=42, offset=1000)
Worker crashes and recovers:
Restore from S3 → [State: counter=42] → Replay from offset 1000Flink's checkpointing provides exactly-once state guarantees — even during failures, each event affects state exactly once.
| Framework | State Management | Checkpointing |
|---|---|---|
| Apache Flink | Best-in-class (RocksDB, exactly-once) | Automatic, consistent |
| Kafka Streams | Good (RocksDB, changelog topics) | Via Kafka compacted topics |
| Spark Structured Streaming | Basic (in-memory, HDFS) | Micro-batch boundaries |
| Apache Beam | Portable (runner-dependent) | Runner-dependent |
Stateful-processing pitfalls that teams hit after the demo works:
StateTtlConfig, Kafka Streams' windowed state retention — or you'll blow up RAM/RocksDB/disk.country='US'), one worker handles most state while others idle. Choose keys with high cardinality and uniform distribution; use compound keys or hash-salting for skew.Stateful stream processing means the application maintains information about past events while processing new ones. Examples: running total of spending per user, current session state, "last seen" timestamp per device, count of login failures in the last 5 minutes, deduplication tables. Each incoming event reads and updates this state, so the output depends not just on the current event but on history. This contrasts with stateless processing (filter, map) where each event is independent. Stateful processing is what enables sessionization, pattern detection, real-time aggregations, and streaming joins — essentially, anything interesting in real-time analytics requires state.
Stateful processing requires solving distributed-systems problems that stateless doesn't. State must be: (1) partitioned so events with the same key always go to the same worker (otherwise you'd see different views of the counter); (2) durable — if a worker crashes, state can't vanish; (3) recoverable — after a crash, state must be restored so processing resumes correctly; (4) scalable — state must move when you add/remove workers; (5) consistent — state updates must be atomic with event processing for exactly-once guarantees. Stateless processing has none of these concerns because there's no state to protect. The hard problems in production streaming systems are almost always state-related.
Checkpointing is the mechanism that makes stateful streaming fault-tolerant. Periodically (e.g., every 60 seconds), the framework takes a consistent snapshot of all worker state plus the input offsets at that moment and writes it to durable remote storage (S3, HDFS, GCS). If a worker crashes, the framework restarts workers from the latest checkpoint — state is restored, and processing resumes from the saved offsets. This gives exactly-once state semantics: no event affects state twice and no state update is lost. Flink's checkpointing is incremental (only changed RocksDB SST files are written), keeping the cost reasonable even with TB of state.
Apache Flink is widely considered the leader for stateful stream processing. Reasons: (1) best-in-class state management with RocksDB scaling to TB of state per worker; (2) automatic, incremental checkpointing with exactly-once guarantees; (3) true event-by-event processing with millisecond latency; (4) mature event-time semantics with watermarks; (5) savepoints for zero-downtime upgrades with state schema evolution. Kafka Streams is good when you're already Kafka-native and want state inside your Java app. Spark Structured Streaming is fine for simpler state needs and Spark-first teams, but lags Flink on latency and state scale. Pick Flink for non-trivial stateful pipelines unless you have specific reasons otherwise.
Keyed state is partitioned by a key (user_id, device_id, account_id). Each key's state is stored on exactly one worker (determined by `hash(key) % parallelism`), so "user-42's counter" is always accessed consistently. This is the most common type of state and the only way to parallelize stateful processing. Operator state is state attached to each parallel operator instance (not a key) — typical uses: Kafka consumer offsets, broadcast configuration distributed to all workers, buffering state for batched writes. Keyed state is for business logic (per-entity counters/sessions/aggregations); operator state is for framework plumbing and coordination-style needs.
A state backend is the storage engine where streaming frameworks keep state. Options in Flink: (1) HashMap backend — in-process Java HashMap on the JVM heap, fastest access but capped by heap size, typically used for small state or testing; (2) RocksDB backend — embedded KV store on local SSD, scales to TB of state per worker, the production default for anything non-trivial, trades some CPU (serialization + LSM compaction) for near-unlimited capacity. Checkpoints from either backend go to remote durable storage (S3/HDFS) for fault tolerance. Choose HashMap for small-state, low-latency workloads; choose RocksDB for production-scale state.
State schema evolution is handling changes to the data types stored in state when upgrading a streaming job. Flink handles it via: (1) savepoints — triggered stop-point snapshots; (2) supported types with evolution rules — POJOs add/remove/reorder fields, Avro follows standard Avro schema-evolution rules; (3) restart from savepoint on the new job version. Gotchas: changing a keyed-state type fundamentally (e.g., Long to String) is a breaking change and usually requires a state migration job. Always test schema evolution in a staging environment before production, and version-control the state types alongside the job code.