Notesencoding🌱 seedling
Schema Registry
Also: Schema ID · Compatibility Mode · Subject
A schema registry stores every version of every message schema, gives each one a compact ID that messages carry instead of the schema, and refuses new versions that break the configured compatibility mode.
Why it exists
In a log or stream, readers need the writer's schema for every message, possibly years later, and nothing else stops a producer from publishing an incompatible change. A registry makes both a service.
How it works
- Schemas are versioned per subject (by default one per topic for keys and one for values).
- The producer registers a schema once and gets an ID; each message is magic byte
0+ 4-byte big-endian ID + payload (Protobuf adds message indexes). Newer versions can carry a schema GUID in a record header instead. - Consumers fetch the schema by ID once and cache it.
- Registration is checked against a mode:
BACKWARD(default),FORWARD,FULL, their_TRANSITIVEvariants, orNONE. Incompatible schemas are rejected with HTTP 409.
Trade-offs
| Gains | Costs |
|---|---|
| 5-byte overhead instead of a schema per message | another critical service in the produce/consume path (mitigated by caching) |
| compatibility enforced before data is written | mode choice dictates upgrade order |
| a history of every schema ever used | IDs are meaningless outside that registry |
In practice
- Schema registries support Avro, Protobuf and JSON Schema; a common recommendation is
BACKWARD_TRANSITIVEfor Protobuf, since adding message types isn't forward compatible. - Implementations: Apicurio Registry, AWS Glue Schema Registry, Buf Schema Registry (Protobuf).
Pitfalls
- Setting
NONEto unblock a deploy, then discovering consumers can't read the topic. - Losing or recreating the registry: every stored message references IDs only it knows.
Connections
- Part of
- Schema Evolution
Recall
These cards are also in your review deck, scheduled with spaced repetition.