Iceberg, Explained

Every arrow has evidence.

The catalog identifies a metadata file. That JSON records snapshots; each snapshot names a manifest list. The Avro list names manifests, and those manifests describe the data files.

0 original artifact bytes fetched in this session (cached per file). Loading a snapshot reads its complete tiny table to verify the demonstration; the pruning model below describes potential savings separately.

The table is a selection.

These are all 5 Parquet files retained in the bundled history. Only the files referenced by the selected snapshot contribute rows. The directory listing below is a comparison; it is never used to decide membership.

12 table rows come from 2 of 5 data files, containing 3,678 stored bytes.

The source CSV has 24 synthetic observations; this story uses the first 18. Timestamps are Iceberg timestamps without a time zone. ISO serialization provides a consistent display for those synthetic timestamps.

A manifest is a file inventory with meaning.

Each manifest entry carries a status: 0 existing, 1 added, 2 deleted. The reader keeps 0 and 1 and skips 2. In the final snapshot, File 3 has status 2. Its replacement preserves the surviving observations. A later snapshot can reuse a manifest, so “added” describes when a file entered that manifest.

Partition values come from the manifest’s partition struct. Bounds are typed values keyed by field ID: field 4 is the temperature, stored here as little-endian doubles.

A manifest entry is not automatically a live file.

Return to the original weather table and the snapshot selected at the top. A manifest entry records a file and its status: existing (0), added (1), or deleted (2). Existing and added entries can supply this snapshot’s rows; deleted entries describe removal and must not be counted as active data.

New files join. Old files keep their identity.

The first write contains 12 rows across two days. Appending six observations adds 2 files and reuses all 2 original files. Observation 13 arrives late for January 1; the other five observations introduce January 3.

The new snapshot names its own manifest list. A reused manifest keeps its references to the same data files. Uploading new files alone would not change the table.

Remove one row. Rewrite its file.

Observation 3 shares a file with five other rows. Removing it writes a replacement containing those five survivors. The snapshot operation is overwrite, and the current table now has 17 rows.

The original six-row file still exists for time travel. A deleted manifest entry drops a file reference. A delete file instead records which rows to remove while retaining their data file. Iceberg v2 also supports position and equality delete files; the delete-files chapter follows their sequence and applicability rules through actual files.

Change the snapshot, change the table.

The selector above drives the file graph, inventory, rows and pruning together. Snapshot identifiers are exact 64-bit values; they are never rounded to JavaScript numbers.

An old snapshot remains readable while its required files are retained. Snapshot expiration and physical file cleanup are separate maintenance operations; this fixture keeps the complete history.

A field’s name can change. Its identity stays.

After the deletion, the writer renames temperature_c to air_temperature_c. Both names identify field 4. The data files are unchanged, and the current snapshot ID is unchanged.

Checking the box selects the latest metadata context, which can still reference every retained snapshot. The row header changes; its values still come from the original physical column. Historical reads may use the snapshot’s original schema.

Partition evolution is another metadata capability. The mixed-specification lesson in the pruning chapter uses a second real table to demonstrate it; this original fixture has one day partition specification.

Reusing an old name must not resurrect an old column.

Use the alternate layout’s weather observations to see how a rename preserves values. Rename temperature_c to air_temperature_c, move it to the first display position, and add a new nullable temperature_c field. No data file changes. The renamed field keeps ID 4, matching the main story; the newly added field receives ID 5.

{
  "type": "struct",
  "fields": [
    {
      "id": 4,
      "name": "air_temperature_c",
      "type": "double",
      "required": false
    },
    {
      "id": 1,
      "name": "observation_id",
      "type": "long",
      "required": false
    },
    {
      "id": 2,
      "name": "station",
      "type": "string",
      "required": false
    },
    {
      "id": 3,
      "name": "observed_at",
      "type": "timestamp",
      "required": false
    },
    {
      "id": 5,
      "name": "temperature_c",
      "type": "double",
      "required": false
    }
  ],
  "schema-id": 3,
  "identifier-field-ids": []
}

Rule out files before opening them.

First, the manifest list’s day bounds can rule out manifests. Inside those retained manifests, the day partition and upper temperature bound can prove a data file has no matches. A surviving file remains a candidate until its rows are evaluated. Missing bounds keep the file.

1 / 1 candidate manifests after the day filter.

The filter retains 2 of 2 files. Decoded rows yield 12 matches; excluded files contain 0 bytes.

Matching observation IDs: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12.

The first pass checks manifest-list partition summaries. The second checks each file’s partition and temperature bounds. Decoded rows decide which candidates actually match.

Partition evolution changes new writes, not old files.

Keep the same seventeen observations from the delete snapshot. Change only how their files are partitioned: write the eleven surviving observations from the first two days under the day specification, then write the six appended observations, IDs 13–18, under a month specification. ID 13 is a late observation from January 1; IDs 14–18 belong to January 3. This is a separate native layout of the same logical answer, with observation 3 still absent. Both specifications remain in metadata. Each manifest identifies the specification needed to interpret its files.

Hidden partitioning lets queries use the source timestamp. Planning translates that condition through each file’s partition transform. Previous files retain their own partition specification when the current spec changes. PyIceberg’s actual task residuals are shown above; DuckDB independently verifies each final result.

The pointer update is the commit.

Conceptual simulation: two writers start from the same metadata. Their files can coexist in storage; only a successful catalog commit publishes a new table state.

A catalog provides atomic publication and conflict detection. Before retrying, the writer validates its operation against the newly committed state.

Removing a snapshot reference does not make every old file garbage.

Snapshots share unchanged data files. In this reachability model, retain or expire historical snapshots and optionally pin the first snapshot with a branch. The current snapshot always remains. A file stays reachable if any retained reference needs it.

The catalog’s atomic metadata-pointer update is the publication boundary. Data and metadata files can be staged beforehand, but readers resolve one committed root and follow its references. A competing writer validates against the current root, then atomically publishes its new metadata pointer.

Download the mixed-spec and field-identity evidence · Iceberg specification

Iceberg selects it. Parquet stores it.

Open an exact bundled file in the existing explorer. The return link preserves this snapshot selection, including when you inspect a historical file.

Take the evidence with you.

Written with PyIceberg 0.10.0 / PyArrow 25.0.0. Every snapshot’s exact rows and values were independently checked with DuckDB 1.4.4 + iceberg extension (extension 1095c1fa). The downloadable bundle contains the original artifacts, source CSV and a hash index.

Read the Apache Iceberg specification, schema evolution guide and reliability model.