Notesstream🌱 seedling
Lambda Architecture
Also: Lambda · Batch and Speed Layers
Keep all raw events immutable and recompute views from scratch in a batch layer, while a separate speed layer processes recent events in real time; queries merge the two. It trades a second codebase for recoverability and low latency.
Why it exists
Early stream processors were approximate and fragile, and batch was hours behind. Lambda keeps batch as the correct, rebuildable path and uses streaming only to cover the gap.
How it works
- Master dataset: every raw event, append-only, never overwritten. The core idea is that a query is a function of all the data.
- Batch layer: periodically recomputes batch views from the whole dataset (Hadoop, later Spark).
- Speed layer: a stream job (originally Storm) maintains views of only the events since the last batch run, approximately if need be.
- Serving: each query merges the batch view with the realtime view. The next batch run replaces the realtime results.
- Bugs are fixed by correcting the code and rerunning batch over the immutable input.
Trade-offs
| Gains | Costs |
|---|---|
| immutable input: any bug is recoverable | the same logic in two frameworks, kept in agreement |
| low latency from the speed layer | query-time merge logic and its bugs |
| batch path is simple to reason about | two systems to operate and scale |
In practice
- LinkedIn moved Who Viewed Your Profile from Lambda to a Samza nearline pipeline (2020), halving most measures of development time.
- Uber (2020) replaced dual codebases with one streaming job that also backfills from Hive.
Pitfalls
- Two implementations drift: rounding, time zones and late-data handling differ, so numbers change when batch overwrites realtime.
Connections
- Part of
- Stream Processing
- Contrasts with
- Kappa Architecture
Recall
These cards are also in your review deck, scheduled with spaced repetition.