Columnar Data, from the inside out Compare the layers →

Columnar / Evidence

Follow the evidence.

Each native example starts with original files or recorded engine output. Find its bytes, recorded versions and reproduction recipe here, then return to the lesson to interpret them.

23 families · 515 artifacts · registry version 1. Download the full registry

Published native evidence families. Private parser corpora and purely illustrative UI state are not catalogued here. Hashes identify bytes; format semantics remain with each family verifier.

Repository commands require the project checkout and its documented runtime. Public recipe downloads may also require companion files from their family and source families. A matching hash establishes byte identity, not correctness by itself. The native verifiers check each example’s claims.

23 evidence families match.

parquet · 9 artifacts

Eight physical layouts

Identical synthetic records across sorting, group/page size, encoding, codec and index variants.

Limits: Browser counters measure application reads; they are not disk or general CPU benchmarks.

Open the lesson →

Versions and reproduction

Recipes

  • parquet-ts/test_files/scan/generate.py · source in the project checkout

    Declared pins: pyarrow==25.0.1

    SHA-256 6dce74ff7d7ff0227af181796f5adf5e793e725a4e0b426e4da5f425599e89d6

Verify in the project checkout

npm run verify:scan
Original files (9)

parquet · 3 artifacts

Native Bloom filters

Original split-block filters, independent XXH64/probe values and exact native query answers.

Limits: A possible match still needs residual decoding. Typed logical extensions are outside the probe adapter.

Open the lesson →

Versions and reproduction

Recipes

  • parquet-ts/test_files/bloom/generate.py
    Download this recipe

    Declared pins: duckdb==1.4.4, pyarrow==25.0.1, xxhash==3.5.0

    SHA-256 85095086cf82a171d763f5691c889b2c950abd19ea4afc1437edd1aa9192b876

Verify in the project checkout

npm run verify:bloom
Original files (3)

arrow · 7 artifacts

Arrow file, stream and buffers

Native file/stream batches, nested offsets, dictionary values, exact primitives and matching Parquet.

Limits: The bounded inspector uses Arrow JS 17.0.0; unsupported extension semantics remain explicit.

Open the lesson →

Versions and reproduction

Recipes

  • explorer/scripts/arrow/generate.py
    Download this recipe

    Declared pins: pyarrow==25.0.1

    SHA-256 cdca3d75df4855b5754772f336323cd1af299945939efbfa46d50836b14ca679

Verify in the project checkout

npm run verify:arrow
Original files (7)
  • /arrow/LICENSE.txt 110,423 bytes SHA-256 229dbe29e4e1344e95172f6f3a4277ec0da5c7850cb03897c47f3a24b7afce08
  • /arrow/NOTICE.txt 3,032 bytes SHA-256 b62ea242642d3a18f1e29b0c1fe082415a6943fe158ee22b48cff3762f354009
  • /arrow/buffers.json 6,414 bytes SHA-256 aea47cc9e60e315e71a93580532f14225f476ac946adb406126b899d85bbf06f
  • /arrow/reproduce.py 3,906 bytes SHA-256 cdca3d75df4855b5754772f336323cd1af299945939efbfa46d50836b14ca679
  • /arrow/sample.arrow 2,650 bytes SHA-256 7f4477100d6fc8b37b1d4bb494f1351f0c68a2a8857e5b3e34d09866e8d3f32c
  • /arrow/sample.arrows 2,056 bytes SHA-256 c22c228f8927027d0aa97d017bf2391e02952a544070961a84091ef9cbd988bc
  • /arrow/sample.parquet 3,508 bytes SHA-256 d3776c1610be5d08bc0500866a3a14c0d1f8f28d6379851a04ec4895f2d7f3c4

iceberg · 21 artifacts

Weather table history

Original metadata, manifests and files for appends, deletion and field rename.

Limits: This curated reader is not a general table or catalog client.

Open the lesson →

Versions and reproduction

Recipes

  • explorer/scripts/iceberg/generate.py · source in the project checkout

    Declared pins: pyiceberg[sql-sqlite,pyarrow]==0.10.0, pyarrow==25.0.0, duckdb==1.4.4, fastavro==1.12.1

    SHA-256 d4b2a356f2f6ed88cb378f821de179bee1de8a20e0a6017b55184c6636beea0b

  • explorer/scripts/iceberg/verify.py · source in the project checkout

    Declared pins: duckdb==1.4.4

    SHA-256 533300856fef12a832d42f48780069e3c5fb4e2c821f99d9f51ae1e3de72a3a0

Verify in the project checkout

uv run explorer/scripts/iceberg/verify.py
Original files (21)

iceberg · 61 artifacts

Equality and position deletes

Four native histories verify delete applicability and every logical result.

Limits: Native reader limitations are retained; opening an original Parquet file does not apply table deletes.

Open the lesson →

Versions and reproduction
  • writer: PyArrow 25.0.1 files; Apache Iceberg Java 1.9.2 commits · recorded source
  • readers: ["Spark 3.5.6 / Iceberg 1.9.2","DuckDB 1.4.4 / iceberg extension"] · recorded source

Recipes

  • explorer/scripts/iceberg/native-deletes.py
    Download this recipe

    Declared pins: pyspark==3.5.6, pyarrow==25.0.1, duckdb==1.4.4, fastavro==1.12.1

    SHA-256 afe038e19f94c151245f964375d0007ee32768bacaf19da9e0de242afd08091d

Verify in the project checkout

uv run explorer/scripts/iceberg/native-deletes.py --verify
Original files (61)

iceberg · 125 artifacts

Writers, cleanup and cumulative vectors

Native conflicts, composite/null keys, compaction, expiration and v3 vector bytes.

Limits: Recorded disposable-table experiments do not grant browser support for arbitrary v3 tables.

Open the lesson →

Companion evidence: Weather table history.

Versions and reproduction

Recipes

  • explorer/scripts/iceberg/native-lifecycle.py
    Download this recipe

    Declared pins: pyspark==3.5.6, pyarrow==25.0.1, fastavro==1.12.1

    SHA-256 7092e4a9ae95b41bdbd8e5a5e3ca744332128b4af9ff7667400932505ef4bb2a

  • explorer/scripts/iceberg/NativeVectors.java
    Download this recipe

    SHA-256 9765933e2cb96fa89b0c5222b124b52a6965084807267b222f5a791945439f82

Verify in the project checkout

uv run explorer/scripts/iceberg/native-lifecycle.py --verify
Original files (125)

iceberg · 1 artifacts

Mixed partition and schema identity

Native mixed day/month specs, field-ID-preserving evolution and residual plans.

Limits: This is a recorded native experiment with the shared weather rows.

Open the lesson →

Companion evidence: Weather table history.

Versions and reproduction

Recipes

  • explorer/scripts/depth/iceberg-evolution.py · source in the project checkout

    Declared pins: pyiceberg[sql-sqlite,pyarrow]==0.10.0, pyarrow==25.0.1, duckdb==1.4.4

    SHA-256 6b3be3e2feffbd67293229c8073ff73b0c449b10f1431de50e092e2df2b0d5ad

Verify in the project checkout

uv run explorer/scripts/depth/iceberg-evolution.py --verify
Original files (1)

delta · 9 artifacts

Original Delta commits and checkpoint

Independent writer/reader proof for three versions and a classic checkpoint.

Limits: The curated browser replay implements reader 1 / writer 2 only.

Open the lesson →

Versions and reproduction

Recipes

  • explorer/scripts/delta/package.py · source in the project checkout

    Declared pins: pyarrow==25.0.1, deltalake==1.6.3

    SHA-256 bf6315cf4867c65e7972c04b76d861d4241715447d32eea00fe35e10c2fe80c1

  • explorer/scripts/delta/proof.py · source in the project checkout

    Declared pins: deltalake==1.6.3, pyarrow==25.0.1

    SHA-256 9494809d1169bac3a1736b32964f3cff79107f471fe0bea82c9ee17588348804

  • explorer/scripts/delta/verify-spark.py · source in the project checkout

    Declared pins: pyspark==3.5.6, delta-spark==3.3.2

    SHA-256 283ff9ffdb3c29e5a4c9c15590fe1f5326b5b59ddc059ca2d37a39633a9062cd

Verify in the project checkout

npm run verify:delta
Original files (9)

delta · 42 artifacts

Vectors and conflicting writers

Original deletion vectors, exact logical rows and three real commit schedules.

Limits: Captured native execution does not broaden the general browser replay protocol.

Open the lesson →

Companion evidence: Original Delta commits and checkpoint.

Versions and reproduction

Recipes

  • explorer/scripts/delta/native-depth.py
    Download this recipe

    Declared pins: pyspark==3.5.6, delta-spark==3.3.2, deltalake==1.6.3, pyarrow==25.0.1, pyroaring==1.0.3, pyzmq==27.1.0

    SHA-256 53d1b0524b0a768ce9caead6757b9d3c48c99b10f5916e5cc25ddf7e4021da30

Verify in the project checkout

uv run explorer/scripts/delta/native-depth.py --verify
Original files (42)

delta · 56 artifacts

Materialization and actual historical absence

REORG and VACUUM preserve the logical answer while removing old physical dependencies.

Limits: Dot-prefixed originals are available through exact-byte aliases and the complete archive. Cleanup runs only in a disposable copy.

Open the lesson →

Companion evidence: Vectors and conflicting writers.

Versions and reproduction

Recipes

  • explorer/scripts/delta/native-lifecycle.py
    Download this recipe

    Declared pins: pyspark==3.5.6, delta-spark==3.3.2, pyarrow==25.0.1

    SHA-256 545829a570c864bfe0043cf29c1aa94cd3cf55da95057cfc0f121ef39d169664

Verify in the project checkout

uv run explorer/scripts/delta/native-lifecycle.py --verify
Original files (56)

duckdb · 4 artifacts

SQL answers and vector lessons

Recorded query answers and native nullable-prefix/filter states.

Limits: Browser playback explains mechanisms; it is not a trace of each instruction executed by DuckDB.

Open the lesson →

Versions and reproduction

Recipes

  • explorer/scripts/duckdb/generate.py · source in the project checkout

    Declared pins: duckdb==1.4.3

    SHA-256 c506cdeaff12083938294dbbabd065592ffd571fa1100694f06929d7c81a02d8

  • explorer/scripts/duckdb/lesson.py · source in the project checkout

    Declared pins: duckdb==1.4.3

    SHA-256 b0e7ad4db5c3cb3c65997ca2fbb8151fdb53fca212b62e2edc65453dd798cfba

  • explorer/static/duckdb/reproduce.py
    Download this recipe

    Declared pins: duckdb==1.4.3

    SHA-256 543bddb25353a35d98266ea16aa95e8d5b7fca7b57ddfd77e4bf09c5e54440ca

Verify in the project checkout

uv run explorer/scripts/duckdb/verify.py
uv run explorer/scripts/duckdb/lesson.py --verify
Original files (4)
  • /duckdb/lesson.json 9,910 bytes SHA-256 f5acec680f01e7cbab35d0c8d80fabd314717c33ebccd3f1d10537f7bed0a92c
  • /duckdb/queries.json 58,184 bytes SHA-256 7aa63559ec13d994c5678818a5db950c67912d4a7a4b0d7b2553cfc8c55d0f9d
  • /duckdb/reproduce.py 952 bytes SHA-256 543bddb25353a35d98266ea16aa95e8d5b7fca7b57ddfd77e4bf09c5e54440ca
  • /duckdb/weather.parquet 1,190 bytes SHA-256 12e54f3643325c44da99b5508fc79bc83e92ee2a0da78ddf603d6d090064a4b5

duckdb · 8 artifacts

Pinned browser SQL runtime

Self-hosted, hash-checked distribution of the pinned Wasm runtime.

Limits: Runtime bytes are not query-result evidence. Live SQL retains its file and execution limits.

Open the lesson →

Versions and reproduction

Recipes

  • explorer/scripts/duckdb/assets.mjs · source in the project checkout

    SHA-256 8ff624788a5371795f600d7415904cf9834f68a3f6f7e93e717f23f3f90db33d

Verify in the project checkout

node scripts/check-duckdb.mjs
Original files (8)

engines · 3 artifacts

Two native query engines

Equal native answers and original plans from DuckDB and DataFusion.

Limits: Only workspace paths are normalized; these captures are not a universal engine ranking.

Open the lesson →

Companion evidence: SQL answers and vector lessons.

Versions and reproduction

Recipes

  • explorer/scripts/engines/generate.py · source in the project checkout

    Declared pins: datafusion==54.0.0, duckdb==1.4.3, pyarrow==25.0.1

    SHA-256 ea1da2ac095aee05c5f108a3ca8d081c94cabc2573362d688158a20ee57bb436

  • explorer/static/engines/reproduce.py
    Download this recipe

    Declared pins: datafusion==54.0.0, duckdb==1.4.3, pyarrow==25.0.1

    SHA-256 0e91337f294967546befb599baf5b618ebd0a295df8b12bb504d9f28283d5a2f

Verify in the project checkout

uv run explorer/scripts/engines/generate.py --verify
Original files (3)
  • /engines/plans.json 25,619 bytes SHA-256 87c5203d72e7adeda4a379f86a3e82f99d9876661cf78bf220fd682a93c63560
  • /engines/raw-plans.json 24,771 bytes SHA-256 3ee44637cac28c679a8da53b8bf4250ea2386d493759f20779ca60e33fd235f1
  • /engines/reproduce.py 1,494 bytes SHA-256 0e91337f294967546befb599baf5b618ebd0a295df8b12bb504d9f28283d5a2f

engines · 1 artifacts

Native execution counterexamples

Native nullable aggregates, composed selections, join multiplicity and layout profiles.

Limits: Small illustrative state models are identified separately from the captured native results.

Open the lesson →

Companion evidence: SQL answers and vector lessons.

Versions and reproduction

Recipes

  • explorer/scripts/depth/sql-proof.py · source in the project checkout

    Declared pins: duckdb==1.4.3, pyarrow==25.0.1

    SHA-256 ef849955fae905e8f3012cca9600db8ce0935129f6b64ee9b806d2c9a6d8f0ab

Verify in the project checkout

uv run explorer/scripts/depth/sql-proof.py --verify
Original files (1)

engines · 1 artifacts

Exact SQL result buffers

Native buffer/value evidence for the typed SQL result inspection contract.

Limits: Live previews retain at most 32 columns and 1,000 rows, with separate typed/text budgets.

Open the lesson →

Companion evidence: SQL answers and vector lessons.

Versions and reproduction

Recipes

  • explorer/scripts/engines/result-buffers.py · source in the project checkout

    Declared pins: datafusion==54.0.0, pyarrow==25.0.1

    SHA-256 58297fbe821c6a6f7e257213ded5ed7f80d68f03e824dc732c4b57cbef34289f

Verify in the project checkout

uv run explorer/scripts/engines/result-buffers.py --verify
Original files (1)

engines · 11 artifacts

Memory, spill and LIMIT

Ten native runs: nine successful profiles and one recorded out-of-memory outcome. Complete successful results match independent Polars queries.

Limits: Buffer peaks are not process RSS; scan rows are not I/O bytes; actual OOM is preserved.

Open the lesson →

Versions and reproduction

Recipes

  • explorer/scripts/engines/memory-work.py
    Download this recipe

    Declared pins: duckdb==1.4.4, polars==1.44.1, pyarrow==25.0.1

    SHA-256 6e15367f3aa7722cf6ee807527d3b2b8c26355550c9a94bf290b1a97b6f4024f

Verify in the project checkout

uv run explorer/scripts/engines/memory-work.py --verify
Original files (11)

lance · 8 artifacts

Lance fragments and original buffers

Original manifest/fragments and native exact search results.

Limits: The decoder covers the explicitly demonstrated 2.1 layout, not arbitrary Lance datasets.

Open the lesson →

Versions and reproduction

Recipes

  • explorer/scripts/lance/package.py · source in the project checkout

    Declared pins: pylance==11.0.0, pyarrow==25.0.1, numpy==2.4.3, protobuf==6.33.5

    SHA-256 7bd9c70a219bb234bbd6c571508bd409d33d859c29dcbb45233bee62e59f703f

  • explorer/static/lance/reproduce.py
    Download this recipe

    Declared pins: pylance==11.0.0, numpy==2.4.3

    SHA-256 28da59c3842d3987a04e4f0ea5b940921ee64b5ae8f3d4b0fa2a942db1f5726b

Verify in the project checkout

npm run verify:lance
Original files (8)

lance · 2 artifacts

Measured IVF_FLAT candidates

Native candidate sets and recall against independent NumPy exact neighbors.

Limits: Exact scoring within visited partitions does not prove complete recall; PQ and graph search remain separate scope.

Open the lesson →

Versions and reproduction

Recipes

  • explorer/scripts/depth/lance-index.py
    Download this recipe

    Declared pins: pylance==11.0.0, pyarrow==25.0.1, numpy==2.4.3

    SHA-256 8d7ea73827058cf27dc210b21d820fe070966e3347e9216abf95ac73ab19202f

Verify in the project checkout

uv run explorer/scripts/depth/lance-index.py --verify
Original files (2)

tables · 23 artifacts

Twenty logical snapshot exports

Derived Parquet results preserve native logical snapshot rows and source provenance.

Limits: Querying these exports is not live Iceberg/Delta table execution.

Open the lesson →

Companion evidence: Weather table history,Equality and position deletes,Original Delta commits and checkpoint,Vectors and conflicting writers.

Versions and reproduction

Recipes

  • explorer/scripts/tables/snapshots.py
    Download this recipe

    Declared pins: pyarrow==25.0.1, duckdb==1.4.4

    SHA-256 2cec7ebbae06e8c21d7d536e941e5b47db9404d1060e353a95c6d7ca735a1d5a

Verify in the project checkout

npm run verify:tables
Original files (23)

engines · 17 artifacts

Estimates, statistics, casts and join order

Twelve native executions with independently checked results, preserved profiles and original Parquet inputs.

Limits: Estimates and work counters describe the pinned examples. Byte counters are not physical disk traffic, and no timing ranking is asserted.

Open the lesson →

Versions and reproduction

Recipes

  • explorer/scripts/engines/counterexamples.py
    Download this recipe

    Declared pins: duckdb==1.4.4, polars==1.44.1, pyarrow==25.0.1

    SHA-256 9f874820c8a996b486381d3c6ebe10b38ee98564a527c20a2c0faeb4a4bd60f5

Verify in the project checkout

npm run verify:counterexamples
Original files (17)

tables · 50 artifacts

One logical history in two formats

Paired native append, update/delete, compaction, identity-preserving rename and retained-version reads, checked against original metadata and data files.

Limits: Local Hadoop Iceberg and path-based Delta with name mapping. These copy-on-write captures do not implement a generic browser table reader or REST catalog.

Open the lesson →

Versions and reproduction

Recipes

  • explorer/scripts/tables/history.py
    Download this recipe

    Declared pins: pyspark==3.5.6, delta-spark==3.3.2, pyarrow==25.0.1, polars==1.44.1, fastavro==1.12.1

    SHA-256 923f48821513ac17b693eee101f679c675c1825987dd75ac0b414a9e3db4de77

Verify in the project checkout

npm run verify:history
Original files (50)

lance · 32 artifacts

Lance index coverage across versions

Five native retained versions and twenty exact NumPy queries; every logical row and physical address checked against original fragment data.

Limits: Stable row IDs are disabled. These native captures do not implement a generic table reader or establish ANN performance.

Open the lesson →

Companion evidence: Lance fragments and original buffers,Measured IVF_FLAT candidates.

Versions and reproduction

Recipes

  • explorer/scripts/lance/lifecycle.py
    Download this recipe

    Declared pins: pylance==11.0.0, pyarrow==25.0.1, numpy==2.4.3

    SHA-256 837f3e2ab6b8b9a184b1825b57ddc8b17b65d521d547624560e0e583656832aa

Verify in the project checkout

npm run verify:lance-lifecycle
Original files (32)

lance · 21 artifacts

Lance pages, nulls and compression

Two native multi-page layouts with complete source-value parity, pinned protobuf metadata, original Zstd layers and bounded browser INT64 decoding.

Limits: The browser decodes only the demonstrated all-valid FullZip flat INT64 column. Nullable string values remain native SDK results, not browser-decoded strings.

Open the lesson →

Companion evidence: Lance fragments and original buffers.

Versions and reproduction

Recipes

  • explorer/scripts/lance/physical.py
    Download this recipe

    Declared pins: pylance==11.0.0, pyarrow==25.0.1, numpy==2.4.3, protobuf==6.33.5

    SHA-256 49eb905d70ec943b56f0b0a79fd27135e8313b37bd5ae79b84551b97a2ca36fb

Verify in the project checkout

npm run verify:lance-physical
Original files (21)
Preserved filenames and download aliases

These Hadoop checksum companions retain their original paths in the reproduction archive. Their public downloads use content hashes because the static host omits dotfiles. Each alias has exactly the original bytes.

  • delta/lifecycle/after-reorg/.part-00000-8ecc143a-9fea-4088-9005-d85a4f1d3e46-c000.snappy.parquet.crc → 20 bytes
  • delta/lifecycle/after-reorg/_delta_log/.00000000000000000005.crc.crc → 28 bytes
  • delta/lifecycle/after-reorg/_delta_log/.00000000000000000005.json.crc → 20 bytes
  • delta/lifecycle/after-vacuum/.part-00000-8ecc143a-9fea-4088-9005-d85a4f1d3e46-c000.snappy.parquet.crc → 20 bytes
  • delta/lifecycle/after-vacuum/_delta_log/.00000000000000000005.crc.crc → 28 bytes
  • delta/lifecycle/after-vacuum/_delta_log/.00000000000000000005.json.crc → 20 bytes
  • delta/lifecycle/after-vacuum/_delta_log/.00000000000000000006.crc.crc → 28 bytes
  • delta/lifecycle/after-vacuum/_delta_log/.00000000000000000006.json.crc → 12 bytes
  • delta/lifecycle/after-vacuum/_delta_log/.00000000000000000007.crc.crc → 28 bytes
  • delta/lifecycle/after-vacuum/_delta_log/.00000000000000000007.json.crc → 12 bytes
  • delta/lifecycle/after-vacuum/_delta_log/._last_vacuum_info.crc → 12 bytes

Find a concept →