NotesSystem

Apache Parquet

Also: Parquet · Parquet File Format

An open columnar file format, announced by Twitter and Cloudera in 2013, that nearly every analytics engine reads and writes. Its defining choice is a footer with the schema, chunk offsets and statistics, written after the data, so readers fetch only what a query needs.

Design choices

  • Column-Oriented Storage per row group: rows split into row groups (parquet-java targets 128 MB; the spec suggests 512 MB–1 GB), each holding one contiguous column chunk per column.
  • Pages are the unit of encoding and compression (1 MB by default in parquet-java); a chunk has an optional dictionary page, then data pages.
  • Columnar Compression: dictionary encoding (RLE/bit-packed codes) by default, with fallback to PLAIN when the dictionary grows too big; also DELTA_BINARY_PACKED, BYTE_STREAM_SPLIT and others, then Snappy, Zstandard, LZ4 or gzip.
  • Nested data is shredded into columns with repetition and definition levels, the encoding from Google's Dremel.
  • Skipping: min/max statistics per chunk, an optional page index with per-page min/max, and optional per-chunk Bloom Filters for high-cardinality columns.
  • Footer last: the file ends with the metadata, its 4-byte length and the magic PAR1, so writers stream in one pass and readers start from the end.

Notable trade-offs

  • Files are immutable: no in-place updates. Table formats (Data Lakehouse) add commits, deletes and snapshots on top.
  • Many small files mean many footers and little skipping; files need compacting into large row groups.
  • Optimised for storage, not computation: engines decode it into an in-memory format such as Apache Arrow.

Used by

  • Apache Spark, Trino, DuckDB, Polars, Snowflake, BigQuery and most other engines; the default data-file format of Iceberg and Delta Lake.