Notesencoding🌱 seedling
Apache Avro
Also: Avro · Writer's Schema · Reader's Schema · Schema Resolution
Avro is a binary format with no field tags: values are written in schema order, so decoding requires the exact writer's schema. At read time Avro resolves the writer's schema into the reader's, matching fields by name and filling in defaults.
Why it exists
Tag numbers are a burden when schemas are generated, such as one per database table. Avro (started in Hadoop in 2009) lets schemas evolve by name, and is compact because the bytes hold nothing but values.
How it works
- Schema in JSON (or Avro IDL). Encoding: ints and longs as zig-zag varints, strings as length + UTF-8, unions as branch index + value.
- Resolution rules: fields matched by name (order may differ); writer-only fields are ignored; reader-only fields take the reader's default, and with no default reading fails; aliases handle renames;
intpromotes tolong,floatordouble. - So add or remove a field only if it has a default. Optional =
["null", T]with defaultnull. - Finding the writer's schema: a container file header (
Obj+ version byte, schema JSON), a per-message schema ID or fingerprint (single-object encoding:C3 01+ 8-byte CRC-64), or negotiation per connection.
Trade-offs
| Gains | Costs |
|---|---|
| smallest encoding; no tag bookkeeping | unreadable without the writer's schema |
| friendly to generated schemas | every reader needs a schema source (file header, registry) |
| explicit defaults make compatibility checkable | missing defaults break compatibility |
In practice
- Kafka + Schema Registry: each message carries a schema ID; consumers fetch and cache the schema.
- Change Data Capture tools such as Debezium commonly emit Avro, generating a schema from each table.
Pitfalls
- Adding a field without a default (breaks backward) or removing one without a default (breaks forward).
- Losing track of old writer schemas: their data becomes undecodable.
Connections
- Part of
- Serialization
- Contrasts with
- Protocol Buffers
Recall
These cards are also in your review deck, scheduled with spaced repetition.