Notesencoding🌱 seedling

Schema Registry

Also: Schema ID · Compatibility Mode · Subject

A schema registry stores every version of every message schema, gives each one a compact ID that messages carry instead of the schema, and refuses new versions that break the configured compatibility mode.

Why it exists

In a log or stream, readers need the writer's schema for every message, possibly years later, and nothing else stops a producer from publishing an incompatible change. A registry makes both a service.

How it works

  • Schemas are versioned per subject (by default one per topic for keys and one for values).
  • The producer registers a schema once and gets an ID; each message is magic byte 0 + 4-byte big-endian ID + payload (Protobuf adds message indexes). Newer versions can carry a schema GUID in a record header instead.
  • Consumers fetch the schema by ID once and cache it.
  • Registration is checked against a mode: BACKWARD (default), FORWARD, FULL, their _TRANSITIVE variants, or NONE. Incompatible schemas are rejected with HTTP 409.

Trade-offs

Gains Costs
5-byte overhead instead of a schema per message another critical service in the produce/consume path (mitigated by caching)
compatibility enforced before data is written mode choice dictates upgrade order
a history of every schema ever used IDs are meaningless outside that registry

In practice

  • Schema registries support Avro, Protobuf and JSON Schema; a common recommendation is BACKWARD_TRANSITIVE for Protobuf, since adding message types isn't forward compatible.
  • Implementations: Apicurio Registry, AWS Glue Schema Registry, Buf Schema Registry (Protobuf).

Pitfalls

  • Setting NONE to unblock a deploy, then discovering consumers can't read the topic.
  • Losing or recreating the registry: every stored message references IDs only it knows.

Recall

These cards are also in your review deck, scheduled with spaced repetition.