Notesencoding🌱 seedling
Schema Evolution
Also: Schema Change · Schema Migration · Reserved Fields · Unknown Fields
Schema evolution is changing a data format while old and new code, and old and new data, keep working together. The safe moves are additive: new optional fields, never a reused identifier, and old code that preserves what it doesn't understand.
Why it exists
Deploys are gradual and data is permanent. Without evolution rules, every format change would need every writer, reader and stored byte to change at once.
How it works
- Add fields as optional, with defaults and (in Protocol Buffers) new numbers.
- Remove only optional fields, and reserve the number and name so they can never be reused.
- Never change or reuse a field number. In Apache Avro, renames need aliases.
- Type changes only between wire-compatible types, accepting truncation risk (
int64→int32). - Preserve unknown fields when old code reads, modifies and writes back, or a Rolling Upgrade silently deletes new data.
- Check changes mechanically:
buf breakingin CI, or a Schema Registry compatibility mode.
Trade-offs
| Gains | Costs |
|---|---|
| independent deploys, safe rollbacks | fields accumulate; nothing is ever truly deleted |
| old data stays readable forever | optional everything means more null handling |
| breaking changes need a new type, topic or version |
In practice
- Databases do evolution too: since PostgreSQL 11 and MySQL 8.0, adding a column with a constant default updates metadata only, and old rows read the default.
- Knight Capital (2012): new code repurposed a flag that old code on one un-updated server still read with its old meaning. More than $460 million lost in about 45 minutes.
Pitfalls
- Old code round-tripping data through a class or JSON and dropping fields it didn't know.
- New enum values reaching old
switchstatements with no default branch.
Connections
- Part of
- Encoding
Recall
These cards are also in your review deck, scheduled with spaced repetition.