NotesSystem
Apache Flink
Also: Flink
A distributed engine for stateful computation over unbounded streams, in event time. Its defining choice is local keyed state made fault-tolerant by asynchronous barrier snapshots, which gives exactly-once state and, with transactional sinks, exactly-once output.
Design choices
- Stateful Stream Processing: state is partitioned by key, held on the heap (the default) or in embedded RocksDB, or since 2.0 disaggregated to object storage (ForSt, still experimental).
- Checkpoints: barriers flow with the data, operators align and snapshot, and recovery rewinds sources to the snapshot's offsets. Unaligned checkpoints (1.11) help under backpressure. Savepoints are user-owned snapshots for upgrades.
- Event Time vs Processing Time and Watermarks: bounded-out-of-orderness strategies, idleness, alignment (1.15), allowed lateness and side outputs for late data.
- Windowing: tumbling, sliding, session and global windows. SQL window TVFs: TUMBLE, HOP, CUMULATE, SESSION (1.19).
- Stream Joins: interval, temporal (event-time, versioned), lookup and regular joins.
- Exactly-Once Semantics: two-phase-commit sinks (since 1.4) driven by checkpoint completion, e.g. Kafka transactions.
Notable trade-offs
- Exactly-once output adds latency (results are visible per checkpoint) and needs transactional sinks plus careful timeout settings.
- Large state means long checkpoints and slow recovery. That is the motivation for incremental checkpoints and disaggregated state.
- Watermarks are only as fast as the slowest input; idle partitions stall them.
Scale
- Alibaba: 4 billion records per second at its 2020 Double 11 peak. Netflix: more than 30,000 Flink jobs (2026). King: 30 billion+ events a day (2017).
- Releases: 2.0 (March 2025), 2.3 (June 2026).