Notesstream🌱 seedling

Lambda Architecture

Also: Lambda · Batch and Speed Layers

Keep all raw events immutable and recompute views from scratch in a batch layer, while a separate speed layer processes recent events in real time; queries merge the two. It trades a second codebase for recoverability and low latency.

Why it exists

Early stream processors were approximate and fragile, and batch was hours behind. Lambda keeps batch as the correct, rebuildable path and uses streaming only to cover the gap.

How it works

  • Master dataset: every raw event, append-only, never overwritten. The core idea is that a query is a function of all the data.
  • Batch layer: periodically recomputes batch views from the whole dataset (Hadoop, later Spark).
  • Speed layer: a stream job (originally Storm) maintains views of only the events since the last batch run, approximately if need be.
  • Serving: each query merges the batch view with the realtime view. The next batch run replaces the realtime results.
  • Bugs are fixed by correcting the code and rerunning batch over the immutable input.

Trade-offs

Gains Costs
immutable input: any bug is recoverable the same logic in two frameworks, kept in agreement
low latency from the speed layer query-time merge logic and its bugs
batch path is simple to reason about two systems to operate and scale

In practice

  • LinkedIn moved Who Viewed Your Profile from Lambda to a Samza nearline pipeline (2020), halving most measures of development time.
  • Uber (2020) replaced dual codebases with one streaming job that also backfills from Hive.

Pitfalls

  • Two implementations drift: rounding, time zones and late-data handling differ, so numbers change when batch overwrites realtime.

Recall

These cards are also in your review deck, scheduled with spaced repetition.