Notesencoding🌱 seedling

Apache Avro

Also: Avro · Writer's Schema · Reader's Schema · Schema Resolution

Avro is a binary format with no field tags: values are written in schema order, so decoding requires the exact writer's schema. At read time Avro resolves the writer's schema into the reader's, matching fields by name and filling in defaults.

Why it exists

Tag numbers are a burden when schemas are generated, such as one per database table. Avro (started in Hadoop in 2009) lets schemas evolve by name, and is compact because the bytes hold nothing but values.

How it works

  • Schema in JSON (or Avro IDL). Encoding: ints and longs as zig-zag varints, strings as length + UTF-8, unions as branch index + value.
  • Resolution rules: fields matched by name (order may differ); writer-only fields are ignored; reader-only fields take the reader's default, and with no default reading fails; aliases handle renames; int promotes to long, float or double.
  • So add or remove a field only if it has a default. Optional = ["null", T] with default null.
  • Finding the writer's schema: a container file header (Obj + version byte, schema JSON), a per-message schema ID or fingerprint (single-object encoding: C3 01 + 8-byte CRC-64), or negotiation per connection.

Trade-offs

Gains Costs
smallest encoding; no tag bookkeeping unreadable without the writer's schema
friendly to generated schemas every reader needs a schema source (file header, registry)
explicit defaults make compatibility checkable missing defaults break compatibility

In practice

  • Kafka + Schema Registry: each message carries a schema ID; consumers fetch and cache the schema.
  • Change Data Capture tools such as Debezium commonly emit Avro, generating a schema from each table.

Pitfalls

  • Adding a field without a default (breaks backward) or removing one without a default (breaks forward).
  • Losing track of old writer schemas: their data becomes undecodable.

Recall

These cards are also in your review deck, scheduled with spaced repetition.