Parquet, Explained

Schema & Types

The first thing in the footer is the schema: a tree of fields, stored depth-first. Every leaf is a column with a physical type (how the bytes are laid out) and, optionally, a logical type (what they mean). Every field also says how often it may appear.

Physical vs. logical

Parquet has only eight physical types. species is a BYTE_ARRAY, a length-prefixed run of bytes; the STRING logical type on top says "treat them as UTF-8". A date is an INT32 with a DATE annotation, a timestamp an INT64 with TIMESTAMP, a decimal an integer or fixed-length byte array with a precision and scale. Encodings and statistics work on the physical type; logical types only tell the reader how to interpret the result.

required, optional, repeated

The third word on each schema line is the field's repetition. A required field has exactly one value per record. An optional field may be null. A repeated field may appear zero or more times, which is how lists are built. Both columns in this file are optional, so each carries a one-level "is this value present?" marker alongside its data. Those markers are the definition levels, and they are the whole trick behind nested data later on.

Row Groups — Horizontal Slices

The 344 rows are not stored as one block. The writer cut them into row groups of 172 rows, and each row group is self-contained: it holds all columns for its rows, with its own statistics in the footer. Readers can process one row group at a time, hand different row groups to different workers, or skip a row group entirely when its statistics rule it out.

Switch the group size and watch the trade-off. Smaller groups give a filter more chances to skip: with groups of 43, a query for Gentoo penguins touches only the last few. But each group adds metadata to the footer and breaks the column into shorter runs that encode less efficiently. Real writers default to far larger groups than this toy file: pyarrow to about a million rows, Spark to 128 MB, and both are commonly tuned down for selective workloads.

The rows are sorted by species and then by flipper length, so the boundary between the two row groups falls inside the Chinstrap block:

Column Chunks — Vertical Slices

Inside a row group the data is not stored row by row. Each column becomes a column chunk: one contiguous run of bytes holding every value of that column for the row group. Row group 0 has two chunks, one of 172 species values and one of 172 flipper lengths.

These eight rows sit at the point in row group 0 where one species ends and the next begins. Press the button to see how the row group actually lays them out.

This is the payoff of columnar storage. A query such as SELECT species reads the two species chunks and nothing else: 245 of 3,000 bytes, 8% of the file. A row-oriented format would have to read every row to pick one field out of it.

Why columnar storage compresses better

Values in one column are alike in type and usually in value, and the encodings in the next sections feed on exactly that:

• Repeated strings like species names become a tiny dictionary plus small integer indices.
• Sorted or slowly changing numbers like our flipper lengths have small deltas and long runs.
• Floats of similar magnitude share exponent bytes, which byte-stream-split exposes.

Interleaving columns row by row destroys all three patterns. Storing a chunk at a time keeps them.

Pages — The Unit of Encoding

A column chunk is itself a sequence of pages. The page is where encoding and compression happen: each one is encoded independently, compressed independently, and carries a small Thrift header saying what is inside. penguins.parquet has 14 pages, because the writer was asked to start a new page every 64 rows.

• Dictionary page — optional, at most one per chunk, always first: the distinct values.
• Data page — the values (or dictionary indices) for a run of rows, preceded by their definition and repetition levels.

Pick a chunk and click a page to see its bytes.

Notice the shape. species opens with a dictionary page holding its two distinct strings, and its data pages are a few bytes each: 1-bit indices plus a run-length header. flipper_length_mm has no dictionary; every data page is a 4-byte float per value, and the definition levels in front of them record which of the 64 slots hold a value at all.

There are two data page formats. In V1 (used here) the levels and values are written together and compressed as one block. In V2 the levels stay uncompressed and only the values are compressed, so a reader can count nulls and rows without decompressing anything. The spec also lists an INDEX_PAGE type, but no writer uses it: per-page statistics live in the page index instead.

Page size is a writer setting, typically 1 MB by default. Smaller pages make skipping finer-grained and cost a header each; larger pages encode and compress better.

Encodings — How Values Become Bytes

Every page is written with an encoding: a rule for turning a run of typed values into bytes. The footer records which ones each chunk used. In penguins.parquet, species is PLAIN + RLE + RLE_DICTIONARY and flipper_length_mm is RLE + PLAIN. RLE appears in both because definition levels are always RLE-encoded.

Each tab below is one encoding. The first three are the ones in this file, shown on its actual bytes; the last two are what the writer would have used for sorted integers and for floats meant to be compressed.

Compression — After Encoding, Per Page

Encoding and compression are two different steps and people confuse them constantly. An encoding knows the type: it turns 64 floats into bytes, or 172 species into two strings and 172 bits. A codec knows nothing about types: it takes the encoded bytes of one page and squeezes repeated byte patterns out of them. Encoding first, then compression, one page at a time.

What each codec buys

The same penguin data written six times, with the same encodings and only the codec changed. Toggle between the 344-row file and a 100,000-row resample in random order, which is closer to how data arrives in practice.

Encoding and codec together

Neither step replaces the other. The table below writes the resampled data with different encodings, each uncompressed and then with Zstandard. A good encoding is worth more than any codec, and the two combine.

Choosing

• Zstandard is the default answer today: near-Gzip ratios at near-Snappy speed, supported by every current reader.
• Snappy remains the safe choice for very old readers and for write-heavy pipelines where CPU is the bottleneck.
• Gzip or Brotli when the file is written once and read rarely, and bytes on disk are what you pay for.
• Uncompressed only for tiny files, for debugging, or for storage that compresses on its own.

The codec is recorded per column chunk, so a single file can mix them, and decompression happens a page at a time, which is what keeps memory bounded no matter how large the chunk is.

Nulls & Nested Data — Definition and Repetition Levels

A data page stores only the values that exist. To put nulls back, and to rebuild lists and structs, every value is paired with two small integers from the Dremel paper:

• Definition level (D): how many of the optional or repeated fields on the path to this value are actually present. Less than the maximum means "null somewhere on the way".
• Repetition level (R): at which repeated field this value continues the previous one. 0 means a new record.

Start with the simplest case. flipper_length_mm is a flat optional column, so its maximum definition level is 1 and it needs no repetition levels at all. Row group 0 has exactly one null, and it lands in the last data page:

The same idea, nested

Now a schema with a list of structs. Parquet stores lists as three levels: an optional wrapper (so the list itself can be null), a repeated group (one repetition per element), and the element. Every optional or repeated field adds one to the maximum definition level of the leaves beneath it; every repeated field adds one to the repetition level.

This is why definition levels are counters rather than a null bit: one column has to distinguish "no list", "empty list", "element with a null field" and "value present", and it does so with a single small integer that RLE-compresses to almost nothing. A reader replays the levels to rebuild the exact nesting, which is what makes Parquet a faithful store for JSON-shaped, Avro and Protobuf data rather than only flat tables.

Statistics & Predicate Pushdown

For every column chunk the footer records a min_value, a max_value and a null_count. A query engine reads those before it reads any data. If a filter cannot be satisfied by anything between a chunk's min and max, the whole row group is skipped without a single data byte being fetched. The filter has been pushed down into the file layout.

These are the real statistics of penguins.parquet. Change the filter and watch what gets skipped.

The rows are sorted by species, so the species statistics are tight: Adelie and Gentoo equality filters each skip one group, while Chinstrap occurs in both and skips neither. flipper_length_mm is only sorted within each species, so its two ranges overlap from 193 to 210 and only filters outside that band skip anything. Sort order at write time is the single biggest lever on how much pushdown can do.

A few details engines have to get right: nulls never satisfy a comparison, so the null count matters for IS NULL filters; string min and max may be truncated for long values, which the footer flags with is_min_value_exact; and floating-point NaN breaks ordering, so writers leave it out of the range.

Bloom Filters — Skipping on Equality

Min/max statistics are useless for a high-cardinality column: a chunk of user IDs spans nearly the whole range, so no equality filter ever skips it. A Bloom filter fixes that. It is a bit array that can answer "is this value definitely not in the chunk?" in a few memory reads, with no false negatives and a small, tunable rate of false positives.

Insert a value: hash it a few ways and set those bits. Query a value: hash it the same ways and check the bits. Any 0 means it is certainly absent.

Try querying Gentoo after inserting row group 0's two species: it is absent, and the filter says so. Then keep inserting made-up names and watch the false positive rate climb as the array fills. Real filters are sized from the expected number of distinct values and a target false positive rate, usually around 1%.

In the file

Parquet's flavour is the split-block Bloom filter: values are hashed with xxHash64, the hash picks one 256-bit block, and 8 bits inside that block are set, so a lookup touches a single cache line. Each column chunk may carry one; the footer's bloom_filter_offset and bloom_filter_length say where. Readers fetch it only when a query has an equality predicate on that column.

penguins.parquet has none: its writer settings did not create them, and with three distinct species the min/max statistics already do the job. Spark and DuckDB write them on request for high-cardinality columns such as IDs and hashes.

Probe a real stored filter → The bundled DuckDB file has 6,144 rows in three groups. Probe 43 in group 0: all eight bits are set, but that group contains no 43. The selective scanner still checks decoded values and finds matches only in group 1.

Original Parquet file · 48 native probes and exact matching row IDs · Pinned reproduction recipe. The example deliberately uses a high target false-positive rate of 25% to expose this case.

Page Index — Skipping Inside a Chunk

Run actual page-selective scans across eight writer layouts →

Row-group statistics are coarse: a row group is usually hundreds of thousands of rows. Since parquet-format 2.5 (2018) writers can add a page index, two small structures per column chunk that sit between the last row group and the footer:

• ColumnIndex — min, max and null count for every page, plus whether the pages are sorted.
• OffsetIndex — for every page, its byte offset, size and the row it starts at.

Together they let an engine pick out only the pages that can match, and fetch exactly those byte ranges. Each ColumnChunk in the footer records where its two index structures are, so the reader picks them up alongside the footer and before any data.

Row group 0's flipper pages are unordered because the last page starts over with the Chinstrap penguins; row group 1's happen to be ascending. Pick species to see the same structures for a string column: the min and max are strings, and the third page of row group 0 is the only one whose range spans two species.

On a real file with millions of rows per row group and thousands of pages, this is what turns a selective query from "read the row group" into "read three pages". Engines that support it (Spark, Trino, DuckDB, arrow) read the index once, plan the byte ranges, and issue one request per run of pages.

The Footer — Table of Contents

Everything above was found by reading the footer first. It is one Thrift struct, FileMetaData, serialized with the compact protocol: the schema, the row groups, each column chunk's location, sizes, encodings, codec and statistics, the page index offsets, and a free-form list of key/value pairs. Below is the actual footer of penguins.parquet, decoded.

Thrift's compact protocol is why the footer is small: field ids are deltas from the previous field, integers are variable-length and zigzag-encoded, and absent optional fields cost nothing. The ARROW:schema entry is pyarrow stashing its own schema so that round-trips preserve Arrow types Parquet cannot express; pandas, Spark and DuckDB leave similar notes under their own keys.

Follow the ↗ links and you are back where the guide started: the footer names the row groups, the row groups name their chunks, the chunks name their pages, and the pages hold the bytes. That chain of offsets is the whole design.

Now Read a File Yourself

Every number on this page came out of penguins.parquet, parsed in your browser by the same TypeScript reader that powers the explorer. The explorer is the unopinionated version of this guide: the raw schema tree, every row group and page, statistics and encodings for each chunk, a hex view of any value, and the data itself.