Columnar Data, from the inside out All topics →

Columnar / Compare

Different layers.
Connected ideas.

These technologies answer different questions. Start with the responsibility you need to understand, then follow the concrete example.

Three questions to separate.

STORAGE

How are values laid out?

A file format organizes bytes, columns, pages and encodings. Parquet and Lance make this layer visible.

MEMBERSHIP

Which files belong?

A table format records versions and references. Iceberg, Delta and Lance use different metadata structures.

EXECUTION

How is an answer computed?

An engine plans scans, joins and aggregates. DuckDB and DataFusion illustrate this work.

Responsibilities, side by side.

A technology can span layers. Lance combines a file container, table metadata and optional search structures; DuckDB includes both an execution engine and storage. This table describes responsibilities, not a ranking.

On a narrow screen, scroll the comparison sideways to see every layer.

TechnologyFile/storage layerTable membershipExecution/search
ParquetColumnar file layoutNo table membership layerRead by query engines
ArrowTyped memory buffers and IPC containersNo snapshot or commit layerBatches consumed and produced by engines
IcebergReferences data filesSnapshots and manifestsRead by query engines
Delta LakeParquet data filesCommit log and checkpointsRead by query engines
DuckDBScans files and has its own storageQueries tables; separate from the examples’ membership metadataPlans and executes SQL
DataFusionScans through data sourcesProviders expose tables to the enginePlans and executes SQL
LanceColumn pages and buffersVersioned manifests and fragmentsSDK scans and search; optional index structures

Follow one learning path.

  1. Start with Parquet's bytes: locate metadata, pages and encoded values.
  2. Trace Arrow’s typed buffers: distinguish memory layouts from encoded storage and IPC.
  3. Follow Iceberg's references or replay Delta's log: see how many files become one versioned table.
  4. Ask the weather file a SQL question: inspect the scan and run the query locally.
  5. Follow the execution plan: compare real operators and understand batches.
  6. Explore selective and vector access: connect search results to fragments and column buffers.

What the examples actually demonstrate.

GuideDemonstrated scope
ParquetReal bytes, encodings, nested values, statistics and local file inspection.
ArrowNative typed arrays, validity and offsets, shared-buffer slices, IPC file/stream batches and a bounded local inspector.
IcebergSnapshots, field identity, native equality/position deletes, conflicting writers, cleanup and cumulative v3 deletion vectors.
Delta LakeLegacy log/checkpoint replay plus native deletion vectors, writer conflicts, delete materialization and physical retention cleanup.
DuckDBLive local SELECT/CTE and EXPLAIN; native plans, typed result buffers, joins, partial aggregation and physical locality.
DataFusionDuckDB/DataFusion plans and exact results, build/probe, join legality, ordering, skew, Arrow buffers and native memory/spill profiles.
LanceOriginal format-2.1 fragments and buffers, exact and IVF_FLAT search, native index lifecycle, and multi-page nullable/compressed layouts with bounded ID decoding.

The Iceberg and Delta lessons start from the same weather records and both append and delete rows, but their deletes remove different observation IDs. It does not make their metadata interchangeable. The DuckDB/DataFusion queries read the same bounded weather file; they do not benchmark table-format performance.

Use each kind of evidence for its question.

File hashes and decoded ranges substantiate structure. Independent engine or reader results substantiate the demonstrated values. Recorded plans show selected operators. Conceptual playback explains a mechanism. None of these alone establishes a general speed or cost ranking.

Arrow has a dedicated buffer guide and IPC inspector, connected to the native result buffers in the engine guide. It is neither a table catalog nor a synonym for any on-disk format. The Lance guide compares exact search with a separate native IVF_FLAT index. Its original byte fixture remains unindexed; hosted LanceDB services are a separate product.

Follow one native table history in Iceberg and Delta →