Notesstream🌱 seedling
Derived Data
Also: Derived Dataset · Derived View · Secondary Data
Data computed from a [[System of Record]] by some transformation: caches, indexes, materialized views, replicas, aggregates, models. It's redundant by design, and if it's lost you can rebuild it from the source.
Why it exists
No single store answers every query well. Search needs an inverted index, dashboards need column stores, hot keys need a cache. Each is the same facts, reshaped for one kind of read.
How it works
- A derivation function maps the source (or its change stream) to the derived form.
- Kept current by one of: recomputation in batch, incremental updates from a Change Data Capture stream or event log, or application dual writes (fragile).
- If the function is deterministic and the input is retained, the derived data can be thrown away and recomputed. That turns bugs into reprocessing jobs.
- It always lags the source by some amount. Readers of derived data are reading the past.
Trade-offs
| Gains | Costs |
|---|---|
| each read pattern gets a store shaped for it | storage and pipelines for every copy |
| rebuildable: corruption is recoverable | lag, so stale reads must be acceptable |
| load moves off the system of record | keeping copies consistent is the hard part |
In practice
- A database's own internals are derived data: Secondary Indexes, Materialized Views and replicas, all maintained from its log.
- Search, caches, warehouses and feature stores fed by CDC or an event log are derived data at the scale of a whole company.
Pitfalls
- Updating derived stores with dual writes from the application: races and partial failures leave copies permanently different.
- Non-deterministic derivations (wall-clock time, calls to mutable services) that give a different result when rebuilt.
Connections
- Part of
- Stream Processing
- Contains
- CQRSMaterialized View
- Learn first
- System of Record
- Builds on this
- Change Data CaptureEvent SourcingKappa Architecture
Recall
These cards are also in your review deck, scheduled with spaced repetition.