Notesencoding🌱 seedling

Protocol Buffers

Also: Protobuf · protobuf · proto3 · proto2 · Protobuf Editions

Protocol Buffers is a schema-driven binary format in which every field has a permanent number. On the wire each field is a tag (field number plus wire type) and a value, so parsers can skip fields they don't know, and the number, not the name, is the field's identity.

Why it exists

Google needed a compact, typed, language-neutral format whose schemas could change without coordinated deploys. It ran internally from the early 2000s and was open-sourced in 2008.

How it works

  • Schema: int64 id = 1; string name = 2;. Names never reach the binary wire.
  • Each field: tag = (field_number << 3) | wire_type, then the value. Wire types: VARINT (0), I64 (1), LEN (2), I32 (5).
  • Integers are base-128 varints: 7 bits per byte, top bit = "more follows", least significant group first. 150 → 96 01; the record {id 150, name "Ada"} is 08 96 01 12 03 41 64 61.
  • Negative int32/int64 always take 10 bytes; sint32/sint64 use zig-zag (−1 → 1, 1 → 2).
  • Unknown field numbers are skipped using the wire type, and preserved on re-serialization (proto3 since 3.5).
  • A non-repeated scalar that appears twice: the last value wins. Concatenated messages merge.

Trade-offs

Gains Costs
compact, fast, typed codegen in many languages field numbers must be managed forever
forward compatible by construction (skippable tags) needs the .proto to interpret meaning
self-delimiting fields, mergeable messages schema changes need discipline or tooling (buf breaking)

In practice

  • gRPC uses Protobuf as its default message format.
  • Editions (first shipped 2024) replace proto2/proto3 syntax with per-feature settings such as field presence; the wire format is unchanged.

Pitfalls

  • Reusing a deleted field number: old code decodes the new field as the old one. Always reserved numbers and names.
  • Converting to JSON or copying into a domain object drops unknown fields.

Recall

These cards are also in your review deck, scheduled with spaced repetition.