{"schemaVersion":1,"scope":"Authored lessons and prerequisites. Companion native examples do not imply general reader support.","evidenceRegistry":"/evidence/registry.json","lessons":[{"id":"file-overview","title":"File Structure","level":"foundation","summary":"Locate the magic bytes, column data and footer before following a byte offset.","prerequisites":[],"evidence":["parquet-layouts"],"key":"parquet:file-overview","topic":"parquet","href":"/parquet/explained#file-overview"},{"id":"schema-types","title":"Schema & Types","level":"foundation","summary":"Map physical leaf types to logical values and optional or repeated fields.","prerequisites":["parquet:file-overview"],"evidence":["parquet-layouts"],"key":"parquet:schema-types","topic":"parquet","href":"/parquet/explained#schema-types"},{"id":"row-groups","title":"Row Groups","level":"foundation","summary":"Follow horizontal record groups into independently stored leaf columns.","prerequisites":["parquet:file-overview"],"evidence":["parquet-layouts"],"key":"parquet:row-groups","topic":"parquet","href":"/parquet/explained#row-groups"},{"id":"column-chunks","title":"Column Chunks","level":"foundation","summary":"Find the pages belonging to one leaf column in one row group.","prerequisites":["parquet:row-groups"],"evidence":["parquet-layouts"],"key":"parquet:column-chunks","topic":"parquet","href":"/parquet/explained#column-chunks"},{"id":"pages","title":"Pages","level":"foundation","summary":"Separate page headers, encoded bodies and optional CRC integrity checks.","prerequisites":["parquet:column-chunks"],"evidence":["parquet-layouts"],"key":"parquet:pages","topic":"parquet","href":"/parquet/explained#pages"},{"id":"encodings","title":"Encodings","level":"deep","summary":"Compare plain values, dictionary indices, runs and delta representations before compression.","prerequisites":["parquet:pages"],"evidence":["parquet-layouts"],"key":"parquet:encodings","topic":"parquet","href":"/parquet/explained#encodings"},{"id":"compression","title":"Compression","level":"deep","summary":"Distinguish encoded bytes, compressed page bodies and decoded memory.","prerequisites":["parquet:encodings"],"evidence":["parquet-layouts"],"key":"parquet:compression","topic":"parquet","href":"/parquet/explained#compression"},{"id":"nulls-nesting","title":"Nulls & Nesting","level":"foundation","summary":"Reconstruct null lists, empty lists and repeated children from definition and repetition levels.","prerequisites":["parquet:schema-types"],"evidence":[],"key":"parquet:nulls-nesting","topic":"parquet","href":"/parquet/explained#nulls-nesting"},{"id":"statistics","title":"Statistics","level":"foundation","summary":"Use trusted min/max bounds to exclude candidates without asserting that a value exists.","prerequisites":["parquet:row-groups"],"evidence":["parquet-layouts"],"key":"parquet:statistics","topic":"parquet","href":"/parquet/explained#statistics"},{"id":"bloom-filters","title":"Bloom Filters","level":"deep","summary":"Probe split-block filters: absence excludes work; a possible match needs residual evaluation.","prerequisites":["parquet:statistics"],"evidence":["parquet-bloom"],"key":"parquet:bloom-filters","topic":"parquet","href":"/parquet/explained#bloom-filters"},{"id":"page-index","title":"Page Index","level":"deep","summary":"Combine page bounds and row locations to plan selective reads and conservative fallbacks.","prerequisites":["parquet:pages","parquet:statistics"],"evidence":["parquet-layouts"],"key":"parquet:page-index","topic":"parquet","href":"/parquet/explained#page-index"},{"id":"footer","title":"Footer","level":"deep","summary":"Follow schema, row-group metadata and offsets from the file trailer.","prerequisites":["parquet:file-overview","parquet:column-chunks"],"evidence":["parquet-layouts"],"key":"parquet:footer","topic":"parquet","href":"/parquet/explained#footer"},{"id":"closing","title":"Your Turn","level":"foundation","summary":"Open a local file and connect metadata, decoded records and application read traces.","prerequisites":["parquet:footer"],"evidence":["parquet-layouts"],"key":"parquet:closing","topic":"parquet","href":"/parquet/explained#closing"},{"id":"what-is","title":"What Arrow is","level":"foundation","summary":"Place Arrow in the path from stored columns to computed answers.","prerequisites":[],"evidence":[],"key":"arrow:what-is","topic":"arrow","href":"/arrow/explained#what-is"},{"id":"slots","title":"A record batch","level":"foundation","summary":"Follow a logical row into typed values, validity, offsets and children.","prerequisites":[],"evidence":["arrow-buffers"],"key":"arrow:slots","topic":"arrow","href":"/arrow/explained#slots"},{"id":"validity","title":"Validity","level":"foundation","summary":"Distinguish the physical slot from its logical null value using validity bits.","prerequisites":["arrow:slots"],"evidence":["arrow-buffers"],"key":"arrow:validity","topic":"arrow","href":"/arrow/explained#validity"},{"id":"fixed-width","title":"Fixed-width values","level":"foundation","summary":"Address original integer slots without losing precision.","prerequisites":["arrow:slots"],"evidence":[],"key":"arrow:fixed-width","topic":"arrow","href":"/arrow/explained#fixed-width"},{"id":"offsets","title":"Variable-width strings","level":"foundation","summary":"Use byte offsets for strings and child-slot offsets for lists, including empty and null lists.","prerequisites":["arrow:validity"],"evidence":["arrow-buffers"],"key":"arrow:offsets","topic":"arrow","href":"/arrow/explained#offsets"},{"id":"lists","title":"Lists and child arrays","level":"foundation","summary":"Follow parent offsets into child arrays and their validity.","prerequisites":["arrow:slots"],"evidence":[],"key":"arrow:lists","topic":"arrow","href":"/arrow/explained#lists"},{"id":"dictionary","title":"Dictionary arrays","level":"deep","summary":"Resolve dictionary indices against a separate values array.","prerequisites":["arrow:slots"],"evidence":["arrow-buffers"],"key":"arrow:dictionary","topic":"arrow","href":"/arrow/explained#dictionary"},{"id":"slicing","title":"Slicing shared buffers","level":"foundation","summary":"Apply a logical slice offset without confusing parent positions and child offsets.","prerequisites":["arrow:offsets"],"evidence":["arrow-buffers"],"key":"arrow:slicing","topic":"arrow","href":"/arrow/explained#slicing"},{"id":"precision","title":"Decimal and timestamp","level":"deep","summary":"Keep exact 64-bit integers when a display or JavaScript number would lose precision.","prerequisites":["arrow:slots"],"evidence":["arrow-buffers"],"key":"arrow:precision","topic":"arrow","href":"/arrow/explained#precision"},{"id":"ipc","title":"IPC files and streams","level":"foundation","summary":"Compare IPC file and stream framing around schemas, dictionaries and record batches.","prerequisites":["arrow:slots","arrow:dictionary"],"evidence":["arrow-buffers"],"key":"arrow:ipc","topic":"arrow","href":"/arrow/explained#ipc"},{"id":"storage-memory","title":"Arrow next to Parquet","level":"foundation","summary":"Compare the storage and memory representations of the same values.","prerequisites":["arrow:slots"],"evidence":[],"key":"arrow:storage-memory","topic":"arrow","href":"/arrow/explained#storage-memory"},{"id":"evidence","title":"Original buffers","level":"foundation","summary":"Inspect original IPC files and reproduce the pinned native buffer observations.","prerequisites":["arrow:ipc"],"evidence":["arrow-buffers"],"key":"arrow:evidence","topic":"arrow","href":"/arrow/explained#evidence"},{"id":"your-turn","title":"Your turn","level":"foundation","summary":"Apply the Arrow walkthrough to an original artefact in the local tools.","prerequisites":["arrow:what-is"],"evidence":[],"key":"arrow:your-turn","topic":"arrow","href":"/arrow/explained#your-turn"},{"id":"what-is","title":"What Iceberg is","level":"foundation","summary":"Place Iceberg in the path from stored columns to computed answers.","prerequisites":[],"evidence":[],"key":"iceberg:what-is","topic":"iceberg","href":"/iceberg/explained#what-is"},{"id":"membership","title":"The table is a selection.","level":"foundation","summary":"Select table rows through a committed snapshot rather than every file in a directory.","prerequisites":[],"evidence":["iceberg-weather"],"key":"iceberg:membership","topic":"iceberg","href":"/iceberg/explained#membership"},{"id":"references","title":"Every arrow has evidence.","level":"foundation","summary":"Trace metadata JSON to a manifest list, manifests and original Parquet data.","prerequisites":["iceberg:membership"],"evidence":["iceberg-weather"],"key":"iceberg:references","topic":"iceberg","href":"/iceberg/explained#references"},{"id":"manifests","title":"A manifest is a file inventory with meaning.","level":"foundation","summary":"Read file entries with partition values, statistics and data or delete content.","prerequisites":["iceberg:references"],"evidence":["iceberg-weather"],"key":"iceberg:manifests","topic":"iceberg","href":"/iceberg/explained#manifests"},{"id":"manifest-status","title":"A manifest entry is not automatically a live file.","level":"deep","summary":"Interpret existing, added and deleted entries in their snapshot context.","prerequisites":["iceberg:manifests"],"evidence":["iceberg-weather"],"key":"iceberg:manifest-status","topic":"iceberg","href":"/iceberg/explained#manifest-status"},{"id":"append","title":"New files join. Old files keep their identity.","level":"foundation","summary":"Follow a new commit that adds files while preserving earlier file identity.","prerequisites":["iceberg:membership"],"evidence":["iceberg-weather"],"key":"iceberg:append","topic":"iceberg","href":"/iceberg/explained#append"},{"id":"history","title":"Change the snapshot, change the table.","level":"foundation","summary":"Compare committed memberships and answers across snapshots.","prerequisites":["iceberg:append"],"evidence":["iceberg-weather"],"key":"iceberg:history","topic":"iceberg","href":"/iceberg/explained#history"},{"id":"delete","title":"Remove one row. Rewrite its file.","level":"foundation","summary":"Follow a copy-on-write deletion that replaces a data file.","prerequisites":["iceberg:history"],"evidence":["iceberg-weather"],"key":"iceberg:delete","topic":"iceberg","href":"/iceberg/explained#delete"},{"id":"delete-files","title":"Two kinds of delete file, two different identities.","level":"deep","summary":"Separate equality keys from file-path and physical-position deletes.","prerequisites":["iceberg:delete"],"evidence":["iceberg-deletes"],"key":"iceberg:delete-files","topic":"iceberg","href":"/iceberg/explained#delete-files"},{"id":"delete-artifacts","title":"Open the delete, down to its value bytes.","level":"deep","summary":"Inspect original delete files and connect their values to affected data rows.","prerequisites":["iceberg:delete-files","parquet:column-chunks"],"evidence":["iceberg-deletes"],"key":"iceberg:delete-artifacts","topic":"iceberg","href":"/iceberg/explained#delete-artifacts"},{"id":"delete-prediction","title":"Can the new key survive and satisfy a predicate?","level":"deep","summary":"Reason about delete applicability before evaluating a remaining-row predicate.","prerequisites":["iceberg:delete-files"],"evidence":["iceberg-deletes"],"key":"iceberg:delete-prediction","topic":"iceberg","href":"/iceberg/explained#delete-prediction"},{"id":"iceberg-delete-boundaries","title":"A key, a partition specification, and a rewrite all change applicability.","level":"deep","summary":"Test sequence, field identity, partition specification and rewrite boundaries for deletes.","prerequisites":["iceberg:delete-files","iceberg:mixed-specs"],"evidence":["iceberg-lifecycle"],"key":"iceberg:iceberg-delete-boundaries","topic":"iceberg","href":"/iceberg/explained#iceberg-delete-boundaries"},{"id":"iceberg-v3-vectors","title":"The next delete must include the previous positions.","level":"deep","summary":"Follow cumulative v3 deletion vectors and the positions retained from earlier deletes.","prerequisites":["iceberg:delete-files"],"evidence":["iceberg-lifecycle"],"key":"iceberg:iceberg-v3-vectors","topic":"iceberg","href":"/iceberg/explained#iceberg-v3-vectors"},{"id":"pruning","title":"Rule out files before opening them.","level":"foundation","summary":"Project predicates into partition and file bounds without inventing matches.","prerequisites":["iceberg:manifests","parquet:statistics"],"evidence":["iceberg-weather"],"key":"iceberg:pruning","topic":"iceberg","href":"/iceberg/explained#pruning"},{"id":"mixed-specs","title":"Partition evolution changes new writes, not old files.","level":"deep","summary":"Read old and new partition layouts using each file’s own specification.","prerequisites":["iceberg:pruning"],"evidence":["iceberg-evolution"],"key":"iceberg:mixed-specs","topic":"iceberg","href":"/iceberg/explained#mixed-specs"},{"id":"schema","title":"A field’s name can change. Its identity stays.","level":"foundation","summary":"Track stable field IDs through a rename while preserving the meaning of historical data.","prerequisites":["iceberg:membership"],"evidence":["iceberg-weather"],"key":"iceberg:schema","topic":"iceberg","href":"/iceberg/explained#schema"},{"id":"field-identity","title":"Reusing an old name must not resurrect an old column.","level":"deep","summary":"Distinguish a renamed field from a new field that reuses an old name.","prerequisites":["iceberg:schema"],"evidence":["iceberg-evolution"],"key":"iceberg:field-identity","topic":"iceberg","href":"/iceberg/explained#field-identity"},{"id":"commit","title":"The pointer update is the commit.","level":"deep","summary":"Separate staged files from the atomic metadata-pointer update that publishes a snapshot.","prerequisites":["iceberg:append"],"evidence":["iceberg-weather"],"key":"iceberg:commit","topic":"iceberg","href":"/iceberg/explained#commit"},{"id":"snapshot-reachability","title":"Removing a snapshot reference does not make every old file garbage.","level":"deep","summary":"Trace retained references before deciding whether an old file is unreachable.","prerequisites":["iceberg:history"],"evidence":["iceberg-lifecycle"],"key":"iceberg:snapshot-reachability","topic":"iceberg","href":"/iceberg/explained#snapshot-reachability"},{"id":"native-iceberg-writers","title":"A stale snapshot can be safe for one operation and unsafe for another.","level":"deep","summary":"Compare native append and conflicting rewrite commits against a stale snapshot.","prerequisites":["iceberg:commit"],"evidence":["iceberg-lifecycle"],"key":"iceberg:native-iceberg-writers","topic":"iceberg","href":"/iceberg/explained#native-iceberg-writers"},{"id":"native-iceberg-cleanup","title":"Expiration and orphan cleanup remove different things.","level":"deep","summary":"Distinguish snapshot expiration from orphan cleanup using recorded native outcomes.","prerequisites":["iceberg:snapshot-reachability"],"evidence":["iceberg-lifecycle"],"key":"iceberg:native-iceberg-cleanup","topic":"iceberg","href":"/iceberg/explained#native-iceberg-cleanup"},{"id":"files","title":"Iceberg selects it. Parquet stores it.","level":"foundation","summary":"Open an original physical Parquet file while preserving its table-membership context.","prerequisites":["iceberg:membership","parquet:row-groups"],"evidence":["iceberg-weather","table-snapshots"],"key":"iceberg:files","topic":"iceberg","href":"/iceberg/explained#files"},{"id":"evidence","title":"Take the evidence with you.","level":"reference","summary":"Download original metadata and data, pinned recipes and independent observations.","prerequisites":["iceberg:references"],"evidence":["iceberg-weather","iceberg-deletes","iceberg-lifecycle","iceberg-evolution"],"key":"iceberg:evidence","topic":"iceberg","href":"/iceberg/explained#evidence"},{"id":"your-turn","title":"Your turn","level":"foundation","summary":"Apply the Iceberg walkthrough to an original artefact in the local tools.","prerequisites":["iceberg:what-is"],"evidence":[],"key":"iceberg:your-turn","topic":"iceberg","href":"/iceberg/explained#your-turn"},{"id":"what-is","title":"What Delta Lake is","level":"foundation","summary":"Place Delta Lake in the path from stored columns to computed answers.","prerequisites":[],"evidence":[],"key":"delta:what-is","topic":"delta","href":"/delta/explained#what-is"},{"id":"directory","title":"Three files. Twenty-three rows. A seventeen-row table.","level":"foundation","summary":"Explain why the files on disk contain more rows than the active table.","prerequisites":[],"evidence":["delta-weather"],"key":"delta:directory","topic":"delta","href":"/delta/explained#directory"},{"id":"replay","title":"Read the log. Watch the table take shape.","level":"foundation","summary":"Apply ordered transaction-log actions to reconstruct active membership.","prerequisites":["delta:directory"],"evidence":["delta-weather"],"key":"delta:replay","topic":"delta","href":"/delta/explained#replay"},{"id":"delete","title":"Delete one row. Replace one file.","level":"foundation","summary":"Follow a replacement file and remove action for a copy-on-write delete.","prerequisites":["delta:replay"],"evidence":["delta-weather"],"key":"delta:delete","topic":"delta","href":"/delta/explained#delete"},{"id":"snapshot","title":"Three commits, three answers.","level":"foundation","summary":"Compare logical answers at three committed versions.","prerequisites":["delta:replay"],"evidence":["delta-weather"],"key":"delta:snapshot","topic":"delta","href":"/delta/explained#snapshot"},{"id":"actions","title":"Actions change membership.","level":"foundation","summary":"Interpret add and remove actions without treating removed data as immediately erased.","prerequisites":["delta:replay"],"evidence":["delta-weather"],"key":"delta:actions","topic":"delta","href":"/delta/explained#actions"},{"id":"files","title":"Active and historical files.","level":"foundation","summary":"Separate active data files from historical files still present on disk.","prerequisites":["delta:actions"],"evidence":["delta-weather"],"key":"delta:files","topic":"delta","href":"/delta/explained#files"},{"id":"checkpoint","title":"A checkpoint changes where replay begins.","level":"foundation","summary":"Start replay from materialized state and apply subsequent log actions.","prerequisites":["delta:replay"],"evidence":["delta-weather"],"key":"delta:checkpoint","topic":"delta","href":"/delta/explained#checkpoint"},{"id":"pruning","title":"File bounds can rule work out.","level":"foundation","summary":"Use active-file statistics to exclude work while retaining uncertain candidates.","prerequisites":["delta:actions","parquet:statistics"],"evidence":["delta-weather"],"key":"delta:pruning","topic":"delta","href":"/delta/explained#pruning"},{"id":"delta-native-vectors","title":"Original deletion vectors and cumulative masks","level":"deep","summary":"Inspect native deletion-vector bytes and cumulative masks over original physical rows.","prerequisites":["delta:delete"],"evidence":["delta-native"],"key":"delta:delta-native-vectors","topic":"delta","href":"/delta/explained#delta-native-vectors"},{"id":"delta-native-writers","title":"Native writer conflicts and recomputation","level":"deep","summary":"Follow native conflicts, retry validation and recomputation before publication.","prerequisites":["delta:actions"],"evidence":["delta-native"],"key":"delta:delta-native-writers","topic":"delta","href":"/delta/explained#delta-native-writers"},{"id":"delta-native-cleanup","title":"Remove the mask, then remove the old file.","level":"deep","summary":"Observe native mask materialization and cleanup of superseded physical files.","prerequisites":["delta:delta-native-vectors"],"evidence":["delta-lifecycle"],"key":"delta:delta-native-cleanup","topic":"delta","href":"/delta/explained#delta-native-cleanup"},{"id":"delta-writers","title":"Two writers can stage files. Only a valid commit publishes them.","level":"deep","summary":"Model the difference between staging files and publishing a valid optimistic commit.","prerequisites":["delta:actions"],"evidence":["delta-native"],"key":"delta:delta-writers","topic":"delta","href":"/delta/explained#delta-writers"},{"id":"delta-retention","title":"A version can be known yet no longer readable.","level":"deep","summary":"Separate reconstructable historical membership from the continued availability of its data files.","prerequisites":["delta:snapshot","delta:checkpoint"],"evidence":["delta-lifecycle"],"key":"delta:delta-retention","topic":"delta","href":"/delta/explained#delta-retention"},{"id":"delta-protocol","title":"Recognizing Parquet is not enough to read a Delta table.","level":"deep","summary":"Check reader feature requirements before claiming support for a Delta table.","prerequisites":["delta:replay","delta:delta-native-vectors"],"evidence":["delta-native"],"key":"delta:delta-protocol","topic":"delta","href":"/delta/explained#delta-protocol"},{"id":"comparison","title":"Similar histories, different metadata.","level":"foundation","summary":"Compare log replay and manifest-based membership across table formats.","prerequisites":["delta:replay","iceberg:membership"],"evidence":["delta-weather","iceberg-weather"],"key":"delta:comparison","topic":"delta","href":"/delta/explained#comparison"},{"id":"evidence","title":"Inspect the evidence.","level":"reference","summary":"Reproduce the original log, checkpoint, data files and native lifecycle observations.","prerequisites":["delta:checkpoint"],"evidence":["delta-weather","delta-native","delta-lifecycle"],"key":"delta:evidence","topic":"delta","href":"/delta/explained#evidence"},{"id":"your-turn","title":"Your turn","level":"foundation","summary":"Apply the Delta Lake walkthrough to an original artefact in the local tools.","prerequisites":["delta:what-is"],"evidence":[],"key":"delta:your-turn","topic":"delta","href":"/delta/explained#your-turn"},{"id":"what-is","title":"What DuckDB is","level":"foundation","summary":"Place DuckDB in the path from stored columns to computed answers.","prerequisites":[],"evidence":[],"key":"duckdb:what-is","topic":"duckdb","href":"/duckdb/explained#what-is"},{"id":"query-overview","title":"One question, a few operators.","level":"foundation","summary":"Follow a SQL question through scan, filter, projection and aggregation.","prerequisites":[],"evidence":["duckdb-captures"],"key":"duckdb:query-overview","topic":"duckdb","href":"/duckdb/explained#query-overview"},{"id":"vectors","title":"Eighteen values enter. Which five leave?","level":"foundation","summary":"Map a predicate’s selected positions back to an input vector.","prerequisites":["duckdb:query-overview"],"evidence":["duckdb-captures"],"key":"duckdb:vectors","topic":"duckdb","href":"/duckdb/explained#vectors"},{"id":"vector-representations","title":"A vector is a logical column, not always a flat array.","level":"deep","summary":"Compare flat, constant and dictionary vector representations.","prerequisites":["duckdb:vectors"],"evidence":["duckdb-captures"],"key":"duckdb:vector-representations","topic":"duckdb","href":"/duckdb/explained#vector-representations"},{"id":"selection-composition","title":"A second filter addresses the first filter’s result.","level":"deep","summary":"Compose a second selection through the first selection’s row mapping.","prerequisites":["duckdb:vectors"],"evidence":["duckdb-captures"],"key":"duckdb:selection-composition","topic":"duckdb","href":"/duckdb/explained#selection-composition"},{"id":"output-buffers","title":"Batches carry the result.","level":"foundation","summary":"Inspect result batches and their Arrow memory buffers.","prerequisites":["duckdb:vectors","arrow:slots"],"evidence":["duckdb-captures"],"key":"duckdb:output-buffers","topic":"duckdb","href":"/duckdb/explained#output-buffers"},{"id":"query","title":"Choose a real query.","level":"foundation","summary":"Execute a bundled query in the browser and distinguish it from recorded native results.","prerequisites":["duckdb:query-overview"],"evidence":["duckdb-runtime","duckdb-captures"],"key":"duckdb:query","topic":"duckdb","href":"/duckdb/explained#query"},{"id":"plan","title":"The actual operator tree.","level":"foundation","summary":"Read the recorded physical operator tree and its measured cardinalities.","prerequisites":["duckdb:query-overview"],"evidence":["duckdb-captures"],"key":"duckdb:plan","topic":"duckdb","href":"/duckdb/explained#plan"},{"id":"query-join","title":"A join attaches values through a key.","level":"foundation","summary":"Attach values through key equality while preserving duplicate matches.","prerequisites":["duckdb:vectors"],"evidence":["duckdb-captures"],"key":"duckdb:query-join","topic":"duckdb","href":"/duckdb/explained#query-join"},{"id":"groups","title":"An average is a sum and a count in progress.","level":"foundation","summary":"Maintain sum and count state across batches to compute a final average.","prerequisites":["duckdb:vectors"],"evidence":["duckdb-captures"],"key":"duckdb:groups","topic":"duckdb","href":"/duckdb/explained#groups"},{"id":"pipeline-state","title":"Batches can finish while an aggregate is still incomplete.","level":"deep","summary":"Explain why LIMIT caps output while a filter, sort or aggregate may consume more input.","prerequisites":["duckdb:groups"],"evidence":["engine-memory"],"key":"duckdb:pipeline-state","topic":"duckdb","href":"/duckdb/explained#pipeline-state"},{"id":"physical-locality","title":"The same values can produce very different pruning opportunities.","level":"deep","summary":"Compare pruning for identical values written in different physical orders.","prerequisites":["parquet:statistics","duckdb:plan"],"evidence":["parquet-layouts"],"key":"duckdb:physical-locality","topic":"duckdb","href":"/duckdb/explained#physical-locality"},{"id":"evidence","title":"Reproduce the evidence.","level":"reference","summary":"Reproduce the pinned native query results, plans and buffer observations.","prerequisites":["duckdb:plan"],"evidence":["duckdb-captures"],"key":"duckdb:evidence","topic":"duckdb","href":"/duckdb/explained#evidence"},{"id":"your-turn","title":"Your turn","level":"foundation","summary":"Apply the DuckDB walkthrough to an original artefact in the local tools.","prerequisites":["duckdb:what-is"],"evidence":[],"key":"duckdb:your-turn","topic":"duckdb","href":"/duckdb/explained#your-turn"},{"id":"what-is","title":"What a query engine is","level":"foundation","summary":"Place a query engine in the path from stored columns to computed answers.","prerequisites":[],"evidence":[],"key":"engines:what-is","topic":"engines","href":"/engines/explained#what-is"},{"id":"execution-overview","title":"Follow the join.","level":"foundation","summary":"Follow a join query from input rows to an ordered result.","prerequisites":[],"evidence":["engine-plans"],"key":"engines:execution-overview","topic":"engines","href":"/engines/explained#execution-overview"},{"id":"rewrite","title":"Why attach a description to a row we will discard?","level":"foundation","summary":"Move a legal filter before a join to reduce intermediate work.","prerequisites":["engines:execution-overview"],"evidence":["engine-plans"],"key":"engines:rewrite","topic":"engines","href":"/engines/explained#rewrite"},{"id":"join-lab","title":"Build and probe","level":"foundation","summary":"Build a hash lookup and probe candidate keys, preserving duplicate multiplicity.","prerequisites":["engines:execution-overview"],"evidence":["engine-plans"],"key":"engines:join-lab","topic":"engines","href":"/engines/explained#join-lab"},{"id":"hash-equality","title":"A hash locates candidates. Equality decides matches.","level":"deep","summary":"Resolve hash collisions with equality checks, including duplicate and null keys.","prerequisites":["engines:join-lab"],"evidence":["engine-depth"],"key":"engines:hash-equality","topic":"engines","href":"/engines/explained#hash-equality"},{"id":"outer-join-rewrite","title":"The same-looking filter move can change an outer join.","level":"deep","summary":"Predict when moving a filter across an outer join changes null-extended rows.","prerequisites":["engines:rewrite","engines:join-lab"],"evidence":["engine-depth"],"key":"engines:outer-join-rewrite","topic":"engines","href":"/engines/explained#outer-join-rewrite"},{"id":"operator-dependencies","title":"A pipeline is constrained by what each operator must know.","level":"deep","summary":"Distinguish streaming output from blocking state that needs more input.","prerequisites":["engines:execution-overview"],"evidence":["engine-depth"],"key":"engines:operator-dependencies","topic":"engines","href":"/engines/explained#operator-dependencies"},{"id":"sort-state","title":"Keeping the smallest three still requires seeing the last row.","level":"deep","summary":"Explain why keeping the smallest three values can still require scanning the last row.","prerequisites":["engines:operator-dependencies"],"evidence":["engine-memory"],"key":"engines:sort-state","topic":"engines","href":"/engines/explained#sort-state"},{"id":"merge-lab","title":"Local order is not global order.","level":"foundation","summary":"Merge sorted local runs to establish global order.","prerequisites":["engines:sort-state"],"evidence":["engine-plans"],"key":"engines:merge-lab","topic":"engines","href":"/engines/explained#merge-lab"},{"id":"result-batch","title":"The ordered result buffers","level":"foundation","summary":"Connect a final ordered result to its typed Arrow buffers.","prerequisites":["engines:merge-lab","arrow:slots"],"evidence":["engine-results"],"key":"engines:result-batch","topic":"engines","href":"/engines/explained#result-batch"},{"id":"arrow-lab","title":"A null has a slot, but no value.","level":"foundation","summary":"Locate a null’s physical slot and validity bit in an output batch.","prerequisites":["arrow:validity"],"evidence":["engine-results"],"key":"engines:arrow-lab","topic":"engines","href":"/engines/explained#arrow-lab"},{"id":"actual-plans","title":"Compare the actual plans.","level":"foundation","summary":"Compare recorded DuckDB and DataFusion operator plans for the same query.","prerequisites":["engines:execution-overview"],"evidence":["engine-plans"],"key":"engines:actual-plans","topic":"engines","href":"/engines/explained#actual-plans"},{"id":"optimization","title":"Optimization moves work toward its input.","level":"deep","summary":"Reason about moving projections and filters while preserving semantics.","prerequisites":["engines:rewrite","engines:actual-plans"],"evidence":["engine-plans"],"key":"engines:optimization","topic":"engines","href":"/engines/explained#optimization"},{"id":"memory-work","title":"Memory limits, spill and a million-row LIMIT","level":"deep","summary":"Inspect native memory-limit failures, spill evidence and LIMIT input work.","prerequisites":["engines:operator-dependencies"],"evidence":["engine-memory"],"key":"engines:memory-work","topic":"engines","href":"/engines/explained#memory-work"},{"id":"optimization-counterexamples","title":"Challenge estimates, casts and join order","level":"deep","summary":"Challenge cardinality estimates, casts and join order with recorded counterexamples.","prerequisites":["engines:optimization"],"evidence":["engine-counterexamples"],"key":"engines:optimization-counterexamples","topic":"engines","href":"/engines/explained#optimization-counterexamples"},{"id":"evidence","title":"Reproduce, then inspect.","level":"reference","summary":"Download pinned native plans, results and reproduction scripts.","prerequisites":["engines:actual-plans"],"evidence":["engine-plans","engine-depth","engine-memory","engine-counterexamples"],"key":"engines:evidence","topic":"engines","href":"/engines/explained#evidence"},{"id":"your-turn","title":"Your turn","level":"foundation","summary":"Apply the query-engine comparison to an original artefact in the local tools.","prerequisites":["engines:what-is"],"evidence":[],"key":"engines:your-turn","topic":"engines","href":"/engines/explained#your-turn"},{"id":"what-is","title":"What Lance is","level":"foundation","summary":"Place Lance in the path from stored columns to computed answers.","prerequisites":[],"evidence":[],"key":"lance:what-is","topic":"lance","href":"/lance/explained#what-is"},{"id":"exact-search","title":"Which vectors are closest?","level":"foundation","summary":"Compute exact neighbors under a specified distance rule.","prerequisites":[],"evidence":["lance-files"],"key":"lance:exact-search","topic":"lance","href":"/lance/explained#exact-search"},{"id":"distance-lab","title":"Make every candidate account for its distance.","level":"foundation","summary":"Account for each candidate’s distance before ranking neighbors.","prerequisites":["lance:exact-search"],"evidence":["lance-files"],"key":"lance:distance-lab","topic":"lance","href":"/lance/explained#distance-lab"},{"id":"ivf-index","title":"An index chooses which distances will never be evaluated.","level":"deep","summary":"Compare exact search with a native IVF_FLAT index and measure missed neighbors as probes change.","prerequisites":["lance:distance-lab"],"evidence":["lance-index"],"key":"lance:ivf-index","topic":"lance","href":"/lance/explained#ivf-index"},{"id":"access-plan","title":"Three nearby vectors can share the work of one.","level":"foundation","summary":"Group selected row requests to see when physical reads can be shared.","prerequisites":["lance:exact-search"],"evidence":["lance-files"],"key":"lance:access-plan","topic":"lance","href":"/lance/explained#access-plan"},{"id":"manifest","title":"The manifest connects the table.","level":"foundation","summary":"Follow a dataset version’s fragment and data-file references.","prerequisites":["lance:exact-search"],"evidence":["lance-files"],"key":"lance:manifest","topic":"lance","href":"/lance/explained#manifest"},{"id":"lance-footer-walk","title":"The footer tells a reader where to ask the next question.","level":"foundation","summary":"Decode footer locations and follow the next range into column metadata.","prerequisites":["lance:manifest"],"evidence":["lance-physical"],"key":"lance:lance-footer-walk","topic":"lance","href":"/lance/explained#lance-footer-walk"},{"id":"byte-ranges","title":"Follow one file's byte ranges.","level":"foundation","summary":"Connect known file regions to original bytes in the unindexed format-2.1 fixture.","prerequisites":["lance:manifest"],"evidence":["lance-files"],"key":"lance:byte-ranges","topic":"lance","href":"/lance/explained#byte-ranges"},{"id":"address-lab","title":"From a row to its bytes","level":"foundation","summary":"Translate a selected row address into the fixture’s physical byte range.","prerequisites":["lance:byte-ranges"],"evidence":["lance-files"],"key":"lance:address-lab","topic":"lance","href":"/lance/explained#address-lab"},{"id":"search-storage","title":"Search selects IDs; storage supplies values.","level":"foundation","summary":"Separate the IDs chosen by search from storage work needed to fetch their values.","prerequisites":["lance:access-plan","lance:address-lab"],"evidence":["lance-files","lance-index"],"key":"lance:search-storage","topic":"lance","href":"/lance/explained#search-storage"},{"id":"index-lifecycle","title":"An application ID survives a rewritten address.","level":"deep","summary":"Track logical IDs, physical addresses and index coverage through append, delete and compaction.","prerequisites":["lance:ivf-index","lance:address-lab"],"evidence":["lance-lifecycle"],"key":"lance:index-lifecycle","topic":"lance","href":"/lance/explained#index-lifecycle"},{"id":"physical-layouts","title":"The chosen row crosses two different page boundaries.","level":"deep","summary":"Follow fixed-width and variable-width values across different page boundaries.","prerequisites":["lance:lance-footer-walk"],"evidence":["lance-physical"],"key":"lance:physical-layouts","topic":"lance","href":"/lance/explained#physical-layouts"},{"id":"layers","title":"Keep the layers distinct.","level":"foundation","summary":"Distinguish nearest-neighbor selection, dataset membership and physical file layout.","prerequisites":["lance:search-storage"],"evidence":["lance-files","lance-index"],"key":"lance:layers","topic":"lance","href":"/lance/explained#layers"},{"id":"evidence","title":"Download the evidence.","level":"reference","summary":"Download the original unindexed file separately from native index experiments and recipes.","prerequisites":["lance:layers"],"evidence":["lance-files","lance-index","lance-lifecycle","lance-physical"],"key":"lance:evidence","topic":"lance","href":"/lance/explained#evidence"},{"id":"your-turn","title":"Your turn","level":"foundation","summary":"Apply the Lance walkthrough to an original artefact in the local tools.","prerequisites":["lance:what-is"],"evidence":[],"key":"lance:your-turn","topic":"lance","href":"/lance/explained#your-turn"}],"concepts":[{"term":"Deleted row still in a file","definition":"A table can apply delete metadata while retaining a data file’s original rows. Physical file contents and visible table rows differ.","href":"/delta/explained#delta-native-vectors","aliases":["deleted row remains","physical rows visible table tombstone soft delete"]},{"term":"Null versus empty list","definition":"A missing list is null; a present list with no elements is empty. Both can have no leaf values while representing different records.","href":"/parquet/explained#nulls-nesting","aliases":["missing list present empty array null child"]},{"term":"Why LIMIT still scans","definition":"A result limit caps returned rows. A filter, sort or aggregation may still need substantial input work to determine them.","href":"/duckdb/explained#pipeline-state","aliases":["top k blocking operator full scan input output early termination"]},{"term":"Dictionary encoding","definition":"Repeated values can be represented by small dictionary indices. Dictionary contents and index streams are distinct.","href":"/parquet/explained#encodings","aliases":["dictionary codes indices repeated strings"]},{"term":"Compression","definition":"A codec transforms encoded bytes. Its compressed size differs from encoded size, decoded arrays and process memory.","href":"/parquet/explained#compression","aliases":["codec snappy zstd gzip compressed size"]},{"term":"CRC checksum","definition":"An integrity check over a Parquet page body. A missing checksum makes no integrity assertion.","href":"/parquet/explained#pages","aliases":["crc32 corruption integrity damaged page"]},{"term":"Bloom filter","definition":"A probabilistic membership structure whose positive answer permits false positives. A correct negative can exclude a candidate.","href":"/parquet/explained#bloom-filters","aliases":["probabilistic membership false positive definite absence"]},{"term":"Page index","definition":"Optional per-page statistics and locations that can help plan selective reads. Inspecting an index alone does not execute pruning.","href":"/parquet/explained#page-index","aliases":["column index offset index page pruning selective scan"]},{"term":"Field identity","definition":"A stable identifier can preserve the meaning of a field across renaming. Reusing a name does not necessarily reuse identity.","href":"/iceberg/explained#field-identity","aliases":["field id rename reused column name schema evolution"]},{"term":"Manifest","definition":"In Iceberg, metadata that records data or delete file entries with associated statistics and partition information.","href":"/iceberg/explained#manifests","aliases":["file inventory manifest list metadata pointer"]},{"term":"Partition evolution","definition":"Changing a table’s partition specification while historical files can retain earlier partition layouts.","href":"/iceberg/explained#mixed-specs","aliases":["mixed partition specs partition transform old new layout"]},{"term":"Optimistic commit","definition":"Writers prepare work against a known table state and validate before publishing a new state. Conflicts can require recomputation.","href":"/delta/explained#delta-native-writers","aliases":["concurrent writer conflict retry stale snapshot recompute"]},{"term":"Physical row position","definition":"The location of a row within a data file, distinct from a business key or its position in a filtered result.","href":"/iceberg/explained#delete-files","aliases":["position delete row address row index equality delete"]},{"term":"Hash collision","definition":"Different join keys can map to the same hash bucket. The engine still needs equality checks to establish a match.","href":"/engines/explained#hash-equality","aliases":["same bucket different keys duplicate join key equality"]},{"term":"Outer join","definition":"A join that can retain unmatched input rows. Later filters can remove those rows, so predicate placement affects semantics.","href":"/engines/explained#outer-join-rewrite","aliases":["left join null extended rows filter placement"]},{"term":"IVF partition","definition":"A coarse vector-search partition used to restrict candidates. Unprobed partitions can contain exact neighbors.","href":"/lance/explained#ivf-index","aliases":["approximate nearest neighbor ann centroid nprobes vector index"]},{"term":"Row group","definition":"A horizontal group of Parquet records, with a column chunk for each leaf column.","href":"/parquet/explained#row-groups","aliases":["horizontal records group size"]},{"term":"Column chunk","definition":"The pages for one leaf column in one Parquet row group.","href":"/parquet/explained#column-chunks","aliases":["leaf column contiguous pages"]},{"term":"Definition and repetition levels","definition":"Streams that encode missing nested values and repeated-record boundaries.","href":"/parquet/explained#nulls-nesting","aliases":["dremel nesting nested records repeated fields"]},{"term":"Predicate pushdown","definition":"Using a predicate earlier in a read or execution plan to avoid unnecessary work while preserving the result.","href":"/parquet/explained#statistics","aliases":["min max bounds statistics pruning skip files"]},{"term":"Snapshot","definition":"A committed view of a table whose referenced data and applicable deletes determine its rows.","href":"/iceberg/explained#membership","aliases":["table membership committed version time travel"]},{"term":"Deletion vector","definition":"A representation of deleted row positions associated with data files. Its storage and applicability rules depend on the table format.","href":"/delta/explained#delta-native-vectors","aliases":["bitmap deleted positions cumulative mask"]},{"term":"Checkpoint","definition":"A materialized state that provides a starting point for replay. It does not make historical data files permanent.","href":"/delta/explained#checkpoint","aliases":["transaction log replay materialized state"]},{"term":"Selection vector","definition":"A mapping of selected row positions that can change which values an operator sees without copying every underlying value.","href":"/duckdb/explained#selection-composition","aliases":["indirection selected row mapping compose filters"]},{"term":"Arrow buffers","definition":"Typed memory buffers for values and, where required, validity, offsets and child arrays. They connect physical storage to query results.","href":"/arrow/explained#slots","aliases":["validity bitmap offsets typed arrays zero copy ipc"]},{"term":"Build and probe","definition":"Hash-join phases that construct a lookup from one input and use the other input to find candidate matches.","href":"/engines/explained#join-lab","aliases":["hash join lookup duplicates cardinality"]},{"term":"Recall at k","definition":"The fraction of the exact top-k neighbors present in a retrieved top-k set for a specified query and distance rule.","href":"/lance/explained#ivf-index","aliases":["missed neighbors exact top k approximate search quality"]}],"topics":[{"topic":"parquet","scope":"Real bytes, encodings, nested values, statistics and local file inspection.","boundary":"The reader supports documented encodings and codecs. Encryption and external column chunks are rejected.","examples":[{"label":"Compare writer layouts","href":"/parquet/layouts","mode":"Live browser execution"},{"label":"From pages to values","href":"/parquet/explained#pages","mode":"Original bytes"},{"label":"Inspect your own file","href":"/parquet/explorer","mode":"Live browser execution"}]},{"topic":"iceberg","scope":"Snapshots, field identity, native equality/position deletes, conflicting writers, cleanup and cumulative v3 deletion vectors.","boundary":"Curated tables and native experiments. The browser is not a general Iceberg reader or a remote catalog client.","examples":[{"label":"Follow table references","href":"/iceberg/explained#references","mode":"Original bytes"},{"label":"Delete applicability","href":"/iceberg/explained#delete-files","mode":"Recorded native execution"},{"label":"Native cleanup","href":"/iceberg/explained#native-iceberg-cleanup","mode":"Recorded native execution"}]},{"topic":"delta","scope":"Legacy log/checkpoint replay plus native deletion vectors, writer conflicts, delete materialization and physical retention cleanup.","boundary":"The general lesson replayer accepts reader 1 / writer 2. Advanced native examples do not imply arbitrary Delta feature support.","examples":[{"label":"Replay committed actions","href":"/delta/explained#replay","mode":"Original bytes"},{"label":"Native deletion vectors","href":"/delta/explained#delta-native-vectors","mode":"Original bytes"},{"label":"Native writer conflicts","href":"/delta/explained#delta-native-writers","mode":"Recorded native execution"},{"label":"Native materialization and cleanup","href":"/delta/explained#delta-native-cleanup","mode":"Recorded native execution"},{"label":"Retention counterexample","href":"/delta/explained#delta-retention","mode":"Explanatory model"}]},{"topic":"duckdb","scope":"Live local SELECT/CTE and EXPLAIN; native plans, typed result buffers, joins, partial aggregation and physical locality.","boundary":"The guide uses recorded examples. The SQL tool runs separately with explicit input, time and result limits.","examples":[{"label":"Actual operator tree","href":"/duckdb/explained#plan","mode":"Recorded native execution"},{"label":"Run the matching SQL","href":"/parquet/explorer?queryExample=filter","mode":"Live browser execution"}]},{"topic":"engines","scope":"DuckDB/DataFusion plans and exact results, build/probe, join legality, ordering, skew, Arrow buffers and native memory/spill profiles.","boundary":"Plan captures and explanatory playback do not measure current browser scheduling, device I/O or comparative performance.","examples":[{"label":"Follow build and probe","href":"/engines/explained#join-lab","mode":"Explanatory model"},{"label":"Inspect memory and spill","href":"/engines/explained#memory-work","mode":"Recorded native execution"},{"label":"Inspect native plans","href":"/engines/explained#actual-plans","mode":"Recorded native execution"}]},{"topic":"lance","scope":"Original format-2.1 fragments and buffers, exact and IVF_FLAT search, native index lifecycle, and multi-page nullable/compressed layouts with bounded ID decoding.","boundary":"The examples are separate datasets. Nullable strings are native SDK results; bounded decoders and recorded queries do not constitute a general Lance reader or live ANN service.","examples":[{"label":"Exact candidate distances","href":"/lance/explained#distance-lab","mode":"Recorded native execution"},{"label":"IVF candidates and misses","href":"/lance/explained#ivf-index","mode":"Recorded native execution"},{"label":"Original footer and buffers","href":"/lance/explained#lance-footer-walk","mode":"Original bytes"},{"label":"Index coverage and changing addresses","href":"/lance/explained#index-lifecycle","mode":"Recorded native execution"},{"label":"Multiple pages and compressed strings","href":"/lance/explained#physical-layouts","mode":"Original bytes"}]},{"topic":"arrow","scope":"Native typed arrays, validity and offsets, shared-buffer slices, IPC file/stream batches and a bounded local inspector.","boundary":"The guide decodes a pinned fixture. The IPC tool uses Apache Arrow JavaScript 17.0.0; it is not a table reader or a constant-memory scanner.","examples":[{"label":"Trace a logical slot","href":"/arrow/explained#slots","mode":"Original bytes"},{"label":"Inspect local IPC batches","href":"/arrow/explorer","mode":"Live browser execution"}]}]}