Columnar Data, from the inside out Arrow, Explained →

Columnar / Memory and interchange

Arrow

Arrow specifies how typed columns are laid out in memory so that languages and engines can exchange arrays without rebuilding each value. Parquet encodes and compresses columns for storage; a reader decodes them into arrays that an engine can compute on. Arrow describes the validity, offsets and value buffers of those arrays. PyArrow, DuckDB and DataFusion produce and consume record batches. Arrow IPC carries these layouts between processes or stores them in files.

ParquetColumns stored as bytesIceberg · Delta LakeFiles selected by a versionArrowTyped arrays in memoryDuckDB · DataFusionOperators turn values into answers
Table metadata selects files. Readers decode stored columns into arrays. Engines compute on those arrays.
Read Arrow, Explained →Inspect an Arrow IPC file →