Parquet / Writer choices

Same records. Different work.

These eight real files contain the same 4,096 synthetic records. Sorting changes which values share a page. Page size, row-group size, dictionary encoding and compression change the physical layout.

PyArrow 25.0.1 wrote and read back every file. Sorting each readback by its unique id reproduces the same complete table, including nulls, Unicode text and integers above JavaScript’s safe number range.

Predict, then run.

For id ≥ 4000, which layout should exclude more pages: sorted or shuffled? Why might a compressed file be smaller but still require more decoding than a larger sorted file?

Minimum id Use page indexes

Each run opens a fresh local reader after downloading the bundled file. Request bytes measure that reader’s Blob reads, excluding the download. Opening reads cache up to 64 KiB from the tail; cached page and index requests add no source bytes. Page counts show decoding work even when bytes were cached. This is a count-only scan of id, not a timing benchmark or a measurement of disk traffic.

Explain the result

Sorted pages group nearby ids, so a high threshold can exclude most of them from their bounds. Shuffled pages span a wider range and need residual checks. A larger page can include more unwanted rows around the threshold. The file without indexes must decode its full id chunks. Dictionary and compression choices also change storage, but the dictionary-v2 variant changes several settings together; it does not isolate one cause.

Inspect the original files.

Layout	Order	Rows / group	Write batch	Dictionary	Codec / pages	File size	Open
sorted-small	Sorted by id	2048	128	Off	NONE / V1.0	154.1 KB	Download
shuffled-small	Shuffled	2048	128	Off	NONE / V1.0	153.9 KB	Download
sorted-large	Sorted by id	4096	128	Off	NONE / V1.0	153.4 KB	Download
wide-pages	Sorted by id	2048	1024	Off	NONE / V1.0	145.5 KB	Download
dictionary-v1	Sorted by id	2048	128	On	NONE / V1.0	114.5 KB	Download
snappy-v1	Sorted by id	2048	128	Off	snappy / V1.0	77.7 KB	Download
dictionary-v2	Sorted by id	2048	128	On	snappy / V2.0	81.3 KB	Download
without-index	Sorted by id	2048	128	Off	NONE / V1.0	154.0 KB	Download

All variants request a 256-byte data-page target. Writers check that target at batch boundaries, so it is not an exact page size. Relative to sorted-small, sorted-large changes only the row-group size; wide-pages changes only the write batch; dictionary-v1 changes only dictionary use; snappy-v1 changes only compression. The combined dictionary-v2 case also exercises V2 pages. All files include page CRC32 checksums, which the browser reader verifies when fetching a page.

Inspect sizes, settings and SHA-256 hashes · Read the Parquet page-index specification