Notesstream🌱 seedling

Derived Data

Also: Derived Dataset · Derived View · Secondary Data

Data computed from a [[System of Record]] by some transformation: caches, indexes, materialized views, replicas, aggregates, models. It's redundant by design, and if it's lost you can rebuild it from the source.

Why it exists

No single store answers every query well. Search needs an inverted index, dashboards need column stores, hot keys need a cache. Each is the same facts, reshaped for one kind of read.

How it works

  • A derivation function maps the source (or its change stream) to the derived form.
  • Kept current by one of: recomputation in batch, incremental updates from a Change Data Capture stream or event log, or application dual writes (fragile).
  • If the function is deterministic and the input is retained, the derived data can be thrown away and recomputed. That turns bugs into reprocessing jobs.
  • It always lags the source by some amount. Readers of derived data are reading the past.

Trade-offs

Gains Costs
each read pattern gets a store shaped for it storage and pipelines for every copy
rebuildable: corruption is recoverable lag, so stale reads must be acceptable
load moves off the system of record keeping copies consistent is the hard part

In practice

  • A database's own internals are derived data: Secondary Indexes, Materialized Views and replicas, all maintained from its log.
  • Search, caches, warehouses and feature stores fed by CDC or an event log are derived data at the scale of a whole company.

Pitfalls

  • Updating derived stores with dual writes from the application: races and partial failures leave copies permanently different.
  • Non-deterministic derivations (wall-clock time, calls to mutable services) that give a different result when rebuilt.

Recall

These cards are also in your review deck, scheduled with spaced repetition.