Encoding & Evolution
Data outlives the code that wrote it. How a record becomes bytes, why JSON, Protobuf and Avro make such different bets, and the rules that let old and new code read each other's data for years.
The whole idea
Data outlives code
Inside a process, data is objects, structs, lists and hash maps, held together by pointers. None of that survives leaving the process. To write a record to disk or send it to another machine, you translate it into a self-contained sequence of bytes. That translation is encoding (or serialization, or marshalling), and the way back is decoding.
If code never changed, any format would do. But code changes all the time, and it never changes everywhere at once. A server-side deploy is a rolling upgrade: for minutes or hours, old and new versions run side by side. Mobile and desktop clients upgrade whenever their users feel like it. And a database row written five years ago is still there, waiting for today’s code to read it.
So every encoding has to cope with two directions of mismatch:
- Backward compatibility: newer code can read data written by older code. This one is relatively easy, because when you write the new code you know exactly what the old format looked like.
- Forward compatibility: older code can read data written by newer code. This is harder: the old code was written before anyone knew what the new format would contain, so it has to ignore additions gracefully and, ideally, not destroy them.
Serialization
Most languages ship a built-in way to turn objects into bytes: Java’s Serializable, Python’s pickle,
Ruby’s Marshal. They’re convenient and they’re a trap for anything that leaves the process.
- They tie you to one language. Another team, or your own future rewrite, can’t read the data.
- They’re a security hole. To restore arbitrary objects, the decoder must be able to instantiate
arbitrary classes. Python’s documentation warns that
pickleis not secure and that you should only unpickle data you trust. In 2015, researchers showed that ordinary libraries on a Java classpath, Apache Commons Collections among them, could be chained into remote code execution through deserialization of untrusted bytes. - Versioning is an afterthought, and efficiency often is too.
They’re fine for a cache that dies with the process. For anything stored or sent, use a format designed to be read by code you didn’t write, possibly years later.
JSON and binary encodings
The obvious alternative is a textual format everyone can read: JSON, XML or CSV. JSON in particular is the default language of web APIs. It is readable, universal and needs no shared schema. It also has real problems:
- Numbers are vague. JSON doesn’t distinguish integers from floats or say how precise they are. Many
parsers, JavaScript’s included, read every number as a 64-bit float, which holds integers exactly only up
to 253. In October 2010, weeks before its new Snowflake tweet IDs were due to pass that, Twitter
began sending every ID twice: as a number (
id) and as a string (id_str). - No binary strings. Bytes must be smuggled in as Base64 text, which is 33% bigger.
- The schema is optional. JSON Schema exists, but nothing forces a writer to follow one, so every reader must hard-code its own idea of the format.
- Field names travel with every record.
One record, three encodings
Binary dialects of JSON, like MessagePack and BSON, shave some of this off (the record above is 15 bytes in MessagePack), but they keep the field names, because they keep JSON’s data model: no schema, so every record must describe itself.
The big savings come from schema-driven binary formats: Apache Thrift (built at Facebook, open-sourced in 2007), Protocol Buffers (used inside Google since the early 2000s and open-sourced in 2008) and Apache Avro (started in the Hadoop project in 2009). Writer and reader agree on a schema in advance, so the bytes only need to carry values. Thrift’s compact protocol encodes the record above in 9 bytes, Protobuf in 8 and Avro in 6.Size is rarely the whole argument for a few-byte record. It matters for terabytes of events, and the schema matters everywhere.
Size is the smaller win. A schema is also documentation that can’t drift from the code. It lets statically typed languages generate classes. And it lets a tool check, before you deploy, whether a change is compatible with everything already written. JSON remains the right choice at the edges, for public APIs and browsers. Between your own services and in your own storage, the schema is worth having.
Protocol Buffers
A Protobuf schema gives every field a type, a name and a field number:
message User {
int64 id = 1;
string name = 2;
}
On the wire, names never appear. Each field is written as a tag followed by a value. The tag packs the
field number and a three-bit wire type into one integer, (field_number << 3) | wire_type. The wire type
says how to find the end of the value: VARINT (0) for integers and booleans, I64 (1) and I32 (5) for
fixed-width numbers, and LEN (2), a length prefix followed by that many bytes, for strings, bytes, nested
messages and packed lists.
Integers, and the tags themselves, are varints: seven bits of value per byte, with the top bit saying whether another byte follows.
How 150 becomes 96 01
A few consequences are worth remembering:
- Small numbers are cheap; negative ones aren’t. A negative
int32orint64is always ten bytes, because its sign bit is the top bit. Thesint32andsint64types use zig-zag encoding instead (0 → 0, −1 → 1, 1 → 2, −2 → 3…), so small negative numbers stay small. - Order and repetition are flexible. If a scalar field appears twice, the last value wins, and concatenating two encoded messages is the same as merging them.
- A reader can skip anything. Meeting a field number it doesn’t know, a parser reads the wire type, works out how many bytes to skip, and carries on. That single property is what makes Protobuf forward compatible.
Thrift works the same way, with field IDs and type codes in its binary and compact protocols. Protobuf is also the default message format of gRPC, which the next lesson covers.
Schema evolution
Because the field number is the field’s identity on the wire, the rules for changing a Protobuf schema follow directly:
- Adding a field is safe, as long as it gets a new number. Old code skips it (forward compatible). New code reading old data finds it missing and sees the field’s default value: zero, empty string, or “not set” for fields with explicit presence (backward compatible).
- Removing a field is safe, but you must reserve its number, and its name too, so nobody can reuse them.
- Never change a field’s number, and never reuse one. Changing a number is the same as deleting the field and adding a new one.
- Changing a type is risky.
int32,uint32,int64,uint64andboolare interchangeable on the wire, but values are truncated as if cast in C++.stringandbytesare compatible only if the bytes are valid UTF-8. A singularstring,bytesor message field can becomerepeated; numeric fields can’t. - Required fields are forever. proto2’s
requiredmeant a reader rejected any message missing the field, so it could never be removed. proto3 dropped it.Protobuf Editions, first shipped in 2024, replacesyntax = "proto2"/"proto3"with per-feature switches such asfield_presence. The wire format doesn’t change.
Forward compatibility needs one more thing from old code: it must not just tolerate unknown fields but keep them. This is the classic way a rolling upgrade loses data:
Old code, new data
Protobuf’s binary format preserves unknown fields by default. Proto3 originally dropped them, and Google restored preservation in version 3.5 (2017). The bytes are still lost if your code converts the message to JSON, or copies its fields one by one into a domain object of its own. Whatever the format, the rule is the same: read, modify and write back without throwing away what you didn’t understand.
The rule about numbers matters even more. A reused field number doesn’t fail loudly; old code decodes the new field as the old one, with complete confidence.
Reusing a field number
Apache Avro
Avro takes the opposite bet. Its schema, written in JSON, has no field numbers at all:
{ "type": "record", "name": "User", "fields": [
{ "name": "id", "type": "long" },
{ "name": "name", "type": "string" } ] }
The encoding is just the values, concatenated in schema order: integers as zig-zag varints, strings as a length and then UTF-8 bytes, and a union as the index of its branch followed by the value. Nothing in the bytes says which field is which, or even where one field ends unless you know its type. So to read Avro you need the exact schema it was written with, the writer’s schema. The reader also has its own reader’s schema, the shape its code expects, and Avro translates from one to the other.
Writer’s schema meets reader’s schema
The resolution rules define what changes are compatible:
- Fields are matched by name, so they can be reordered.
- A field only the writer has is ignored.
- A field only the reader has takes the reader’s default. If it has none, reading fails.
- So you may add or remove a field only if it has a default. An optional field is a union with
null, like["null", "string"], withnullas its default. - Renames use aliases in the reader’s schema: a new reader can read old data, but an old reader can’t read the new name.
- Some type changes are promoted automatically:
inttolong,floatordouble, for example.
Where does the reader find the writer’s schema? Avro offers three answers. A container file puts it
once in the header (the file starts with Obj and a version byte), ahead of millions of records, which is
why Avro is at home in data lakes. A message stream puts a schema ID or fingerprint in front of each
record: Avro’s single-object encoding uses a two-byte marker, C3 01, plus an 8-byte schema fingerprint.
And two parties on a long-lived connection can agree on a schema when the connection opens.
Leaving out field numbers makes Avro friendly to generated schemas. Dump a database table and each column becomes a field; when the table changes, generate a new schema from it. Nobody has to hand out tag numbers, and resolution by name does the rest. That’s why change-data-capture tools like Debezium commonly emit Avro with a schema registry.
Backward and forward compatibility
Now the two directions from the start, made precise. With two versions of a schema there are four pairings of reader code and stored data, and compatibility is a claim about the two that mix versions.
Who can read what
Every change is compatible in one direction, both or neither:
| Change | Backward (new code reads old data) | Forward (old code reads new data) |
|---|---|---|
| Add a field with a default | ✓ the default fills in | ✓ old code skips it |
| Add a field with no default | ✗ old data lacks it | ✓ |
| Remove a field that had a default | ✓ new code ignores it | ✓ old code uses its default |
| Remove a field old code requires | ✓ | ✗ |
| Rename (Avro alias in the new reader) | ✓ | ✗ |
| Reuse a Protobuf field number | ✗ silently wrong | ✗ silently wrong |
Databases need backward compatibility for as long as old rows exist, which usually means forever, because rewriting every row is expensive. So many don’t: since PostgreSQL 11 and MySQL 8.0, adding a column with a constant default changes only metadata, and old rows report the default when read. That is schema resolution, done by the database. A rolling deploy of a service that both reads and writes shared data needs both directions, because old and new nodes read each other’s writes. So does a rollback: once new code has written new data, rolling back means old code must read it.
Schema registries
In a message stream, someone has to keep track of every schema that was ever written, hand out IDs, and refuse changes that break compatibility. That’s a schema registry. For Kafka, Apicurio, AWS Glue Schema Registry and the Buf Schema Registry for Protobuf are the well-known ones.
A registry in the write path
How a registry works:
- Schemas are versioned per subject, by default one per topic for keys and one for values. Avro, Protobuf and JSON Schema are supported.
- Each message is a magic byte
0, a 4-byte big-endian schema ID and then the payload. Protobuf adds a short list of message indexes, saying which message type in the file it is. Newer versions can carry a schema GUID in a record header instead. - Every new version is checked against a compatibility mode:
BACKWARD,FORWARD,FULL, their_TRANSITIVEvariants, orNONE. The default isBACKWARD, so that consumers can always rewind to the start of a topic and read everything with the newest schema. The trade: underBACKWARD, consumers must upgrade before producers. - For Protobuf,
BACKWARD_TRANSITIVEis the common recommendation, because adding a new message type isn’t forward compatible.
Choosing a format
| JSON | Protocol Buffers | Avro | |
|---|---|---|---|
| Our record | 23 bytes | 8 bytes | 6 bytes |
| Schema | optional (JSON Schema) | required, .proto |
required, JSON |
| Field identity on the wire | name | number (tag) | position, resolved by name |
| Unknown fields | kept only by generic parsers | skipped, preserved in binary | dropped by resolution |
| Reading needs | nothing | the .proto for meaning, not for parsing |
the exact writer’s schema |
| Typical home | public APIs, browsers, config | RPC (gRPC), service contracts | Kafka, data files, generated schemas |
| Main risk | number precision, drifting formats | reused field numbers | missing defaults, lost writer schema |
Analytics adds a fourth shape: columnar files like Parquet, which store each field of millions of records together. That’s a different trade, covered in the analytics lesson.
In the real world
- Knight Capital, August 1, 2012. New trading code reused a flag that had once activated an obsolete feature called Power Peg. A technician didn’t copy the new code to one of the eight servers, so orders carrying the flag reached a server where it still meant “run Power Peg”. In about 45 minutes, while trying to fill 212 customer orders, the router sent millions of child orders that drew more than 4 million executions, and Knight lost more than $460 million. It wasn’t Protobuf, but it was exactly the reused-field-number failure: identical bits, a new meaning, and one node still on old code.
- CrowdStrike, July 19, 2024. A content update for the Falcon sensor, Channel File 291, used a template type that defined 21 input fields, while the sensor code that fed it supplied only 20. The new content was the first to actually read the 21st field. The read went out of bounds inside a kernel driver, and about 8.5 million Windows machines (Microsoft’s estimate) crashed. The content validator had checked against 21 fields too. Writer and reader disagreed about the schema, and nothing at runtime checked.
- Twitter’s
id_str, 2010. Twitter’s new Snowflake tweet IDs were scheduled to pass 253 on 26 November 2010, after which JavaScript would silently round them. So from October the API sent every tweet, user and message ID as a string as well as a number. X’s current API sends IDs as strings by default. - Google built gRPC’s predecessor, Stubby, on Protobuf and ran it across its data centres for over a
decade. Its guidance on field numbers (“never reuse”) and on
required(“required is forever”) is earned. - Kafka shops lean on a schema registry’s
BACKWARDdefault so consumers can replay a topic from the beginning, and many teams addbuf breakingor registry compatibility checks to CI so a bad schema fails the build, not production.
Pitfalls
- Treating JSON numbers as 64-bit integers. Send large IDs as strings. Any JavaScript client, and many others, will round them.
- Old code that drops unknown fields. Round-tripping through a domain object, an ORM or JSON loses the bytes. Preserve unknown fields wherever old code writes back data it read.
- Reusing or renumbering a field. Always
reserveddeleted numbers and names. The failure is silent. - “Wire-compatible” type changes that lose data.
int64toint32parses fine and truncates. - New enum values. Old code will meet values it doesn’t know; give every
switcha default branch. - Avro fields without defaults. Adding one breaks backward compatibility; removing one breaks forward.
- Checking compatibility only against the latest version while data from much older versions still exists. Use a transitive mode.
- Forgetting rollbacks. After new code writes new data, rolling back needs forward compatibility.
Recall
Go deeper
- Protocol Buffers: Encoding: tags, wire types, varints, zig-zag and “last one wins”, with worked byte examples.
- Protocol Buffers: Updating a message type: the official list of wire-safe, wire-compatible and unsafe changes, and why to reserve.
- Apache Avro 1.12 specification: binary encoding, container files, single-object encoding and the schema-resolution rules.
- Confluent: Schema Evolution and Compatibility: every compatibility mode, what it allows and who upgrades first.
- Martin Kleppmann, “Schema evolution in Avro, Protocol Buffers and Thrift” (2012): the three formats compared byte by byte.
- Apache Thrift: Compact protocol spec: a third design, with field-ID deltas packed into nibbles.
- CrowdStrike, Channel File 291 root cause analysis (2024): a writer/reader schema mismatch at planetary scale.
- SEC order on Knight Capital (2013): the repurposed flag, the missed server and the 45 minutes.
Keep going
The atomic notes for this lesson live in the knowledge graph:
Next on the map: Modes of Dataflow (how encoded data moves between processes: databases, REST and RPC, messages and durable workflows) and Message Brokers & Logs.