The log defines the table.
Warehouse directory
3 files, 23 physical rows.
We wrote 12 weather observations, appended six, then deleted observation 13. All three Parquet files still exist. The table’s answer depends on which files its log selects.
17rows selected by the logA + C. Observation 13 is absent; each remaining ID occurs once.
A, B and C are reading aliases assigned in first-add order. The next section exposes their exact original filenames. The directory view uses the verified contents of each original file.
How does the reader know to exclude B? ↓ Step 1 of 13
Source / commit 0
_delta_log/00000000000000000000.json - 1commitInfo
- 2protocol
- 3metaData
- 4add
No line selected.
Reader working state
0referenced rows Reconstructing the next complete version.
Last complete snapshot
0no table reconstructed yet Atomic publication covers the complete commit. Intermediate working sets become visible to readers only after every action has been reconciled.
File A · 12 physical rows · part-00000-bfdf29f6-7be0-4378-b1a2-346f035d0445-c000.snappy.parquet
123456789101112
A survives every version unchanged. The delete rewrites B, which holds observation 13.
Download original file A → Start with the artefact
Three files, one table.
The directory retains original and replacement Parquet files. Begin with the first commit and follow the file references.
What Delta Lake is
Delta Lake is an open table format that tracks a table through an ordered transaction log. The _delta_log folder sits beside the Parquet files. Commits add and remove file references; replaying them reconstructs a version, even while older files remain on disk. A checkpoint gives readers a shorter starting point. Databricks created Delta Lake; Spark and delta-rs produce and consume its tables.
About this fixture
Written by delta-rs 1.6.3; checked with Spark 3.5.6 / Delta Spark 3.3.2. Replay and checkpoint decoding run in this browser over original commits. Deletion-vector and transaction chapters use separate native captures. Intermediate replay states show a reader working through a complete commit.
Actions change membership.
protocol declares required reader/writer capabilities. metaData
declares the schema and table settings. An add action makes a data file active;
remove excludes it from later versions. The latest relevant action wins when replaying
the log.
commitInfoCommit or table metadata {
"commitInfo": {
"timestamp": 1789067035249,
"operation": "WRITE",
"operationParameters": {
"mode": "ErrorIfExists"
},
"engineInfo": "delta-rs:py-1.6.3",
"clientVersion": "delta-rs.py-1.6.3",
"operationMetrics": {
"num_added_files": 1,
"num_removed_files": 0,
"num_partitions": 0,
"num_added_rows": 12,
"execution_time_ms": 1
}
}
} protocolCommit or table metadata {
"protocol": {
"minReaderVersion": 1,
"minWriterVersion": 2
}
} metaDataCommit or table metadata {
"metaData": {
"id": "4f149ece-9a97-45b4-94d8-60a51b8104f1",
"name": null,
"description": null,
"format": {
"provider": "parquet",
"options": {}
},
"schemaString": "{\"type\":\"struct\",\"fields\":[{\"name\":\"observation_id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"station\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"observed_at\",\"type\":\"timestamp\",\"nullable\":true,\"metadata\":{}},{\"name\":\"temperature_c\",\"type\":\"double\",\"nullable\":true,\"metadata\":{}}]}",
"partitionColumns": [],
"createdTime": 1789067035247,
"configuration": {
"delta.checkpointPolicy": "classic"
}
}
} addFile enters this version {
"add": {
"path": "part-00000-bfdf29f6-7be0-4378-b1a2-346f035d0445-c000.snappy.parquet",
"partitionValues": {},
"size": 1716,
"modificationTime": 1789067035249,
"dataChange": true,
"stats": "{\"numRecords\":12,\"minValues\":{\"temperature_c\":5.0,\"observed_at\":\"2026-01-01T00:00:00Z\",\"observation_id\":1,\"station\":\"North\"},\"maxValues\":{\"observation_id\":12,\"temperature_c\":16.0,\"observed_at\":\"2026-01-02T15:00:00Z\",\"station\":\"South\"},\"nullCount\":{\"station\":0,\"observed_at\":0,\"temperature_c\":0,\"observation_id\":0}}",
"tags": null,
"baseRowId": null,
"defaultRowCommitVersion": null,
"clusteringProvider": null
}
} Active and historical files.
1 active files supply 12 physical rows. Removed files remain
available here so older versions can still be read; this fixture has not vacuumed its history.
File B · Historical / not yet addedpart-00000-62341c08-96d8-43a9-bcf1-5f1f6bd91945-c000.snappy.parquet6 physical rowsFile inspection becomes available when the selected version is loaded.
File A · Activepart-00000-bfdf29f6-7be0-4378-b1a2-346f035d0445-c000.snappy.parquet12 physical rowsFile inspection becomes available when the selected version is loaded.
File C · Historical / not yet addedpart-00000-ea75b463-cd35-4a58-8a1e-d9623429ed07-c000.zstd.parquet5 physical rowsFile inspection becomes available when the selected version is loaded.
Version 2 removes the six-row appended file and adds a replacement with five rows. Observation
13 is absent from the new file. This is copy-on-write deletion: no deletion vector is needed.
Delete one row. Replace one file.
The writer read version 1, found observation 13 in B, and wrote C containing the other five observations. A did not change. The new file uses Zstandard while B uses Snappy; each Parquet file carries its own encoding and compression metadata.
The actual DELETE commit contains an add for C and a remove for B. The file actions replace B as a whole; its original bytes remain intact. The five surviving observation IDs stay the same even though their physical file changes.
B · IDs 13–18
6 old rows, still on disk→C · IDs 14–18
5 copied rows, new file
When does another reader see it?
Writing C is preparation. The new table version is defined by atomic publication of the log entry containing both file actions. A reader pinned to version 1 continues to use A + B. A reader choosing version 2 uses A + C. Neither has to mix the two versions.
If publication fails, the uploaded file stays outside the table. The committed log establishes membership. This original fixture records a successful delete. The native transaction chapter below also includes executed failures and their uncommitted replacement files.
What if two writers start from version 1?
Publication reserves the next log version for one successful writer. A writer that loses publication must inspect intervening changes and validate whether its operation remains compatible. Conflicting work requires a fresh computation or rejection, followed by publication against the current state. The precise publication mechanism depends on the storage implementation.
Why keep B?
Version 1 needs its original bytes. Time travel requires both reconstructible metadata and the data files it references. Retention cleanup can eventually remove obsolete files. Reading those historical rows then requires recovering their data bytes as well as their log. This base fixture retains its old files. The native cleanup chapter below executes materialization and retention on a separate copy.
Three commits, three answers.
JSON commit names are zero-padded version numbers. Reading version 0 means applying commits
0 through 0, in order. The resulting active set determines which Parquet files to read.
Read the complete selected commit
{"commitInfo":{"timestamp":1789067035249,"operation":"WRITE","operationParameters":{"mode":"ErrorIfExists"},"engineInfo":"delta-rs:py-1.6.3","clientVersion":"delta-rs.py-1.6.3","operationMetrics":{"num_added_files":1,"num_removed_files":0,"num_partitions":0,"num_added_rows":12,"execution_time_ms":1}}}
{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}
{"metaData":{"id":"4f149ece-9a97-45b4-94d8-60a51b8104f1","name":null,"description":null,"format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"observation_id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"station\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"observed_at\",\"type\":\"timestamp\",\"nullable\":true,\"metadata\":{}},{\"name\":\"temperature_c\",\"type\":\"double\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"createdTime":1789067035247,"configuration":{"delta.checkpointPolicy":"classic"}}}
{"add":{"path":"part-00000-bfdf29f6-7be0-4378-b1a2-346f035d0445-c000.snappy.parquet","partitionValues":{},"size":1716,"modificationTime":1789067035249,"dataChange":true,"stats":"{\"numRecords\":12,\"minValues\":{\"temperature_c\":5.0,\"observed_at\":\"2026-01-01T00:00:00Z\",\"observation_id\":1,\"station\":\"North\"},\"maxValues\":{\"observation_id\":12,\"temperature_c\":16.0,\"observed_at\":\"2026-01-02T15:00:00Z\",\"station\":\"South\"},\"nullCount\":{\"station\":0,\"observed_at\":0,\"temperature_c\":0,\"observation_id\":0}}","tags":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}A checkpoint changes where replay begins.
The real classic checkpoint represents version 1. It stores table state as Parquet rows:
protocol, metadata and active file actions. To reach version 2, load that state and apply
commit 2. Checkpoints accelerate reconstruction of an existing version.
Choose version 1 or 2: checkpoint 1 starts after version 0.
The button fetches the original Parquet checkpoint, verifies its hash and decodes it with the
TypeScript reader. It compares exact active filenames with replay from commit 0.
Replay from the beginning
0.json → 1.json → 2.json
3,707 stored bytes9 JSON actions. Reconstruct protocol, metadata and membership through all three versions.
Start from checkpoint 1
checkpoint 1 → 2.json
15,741 stored bytes4 checkpoint rows plus 3 JSON actions. Begin with A + B, then replace B with C.
Both routes end at A + C and 17 rows. In this tiny fixture the checkpoint route is larger in stored bytes. The totals add the listed metadata files. A checkpoint saves replay work by consolidating actions; its encoded size is a separate question.
A checkpoint consolidates table state at an existing version. Its rows contain action fields; here they preserve protocol, metadata and two active file references. A later checkpoint can also retain unexpired remove tombstones. To read version 0, checkpoint 1 is too new.
File bounds can rule work out.
1 of 1 active files might match; 12 recorded
rows actually match. A file whose maximum is below the threshold can be excluded. A remaining file
still needs row filtering.
File A · Read candidate Maximum temperature: 16 °C
Maximum is at least 0 °C; inspect rows to decide.
1,716 stored bytes Candidate file bytes: 1,716. Matching observation IDs: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12.
A matching maximum retains the file for evaluation. Its individual rows still face the predicate.
Missing bounds keep a file in the candidate set. Decoded row values decide the final matches.
The same rows, two deeper questions
Can we delete without rewriting—and publish safely under contention?
The following native experiments fork the eighteen-row table above. Spark produces real deletion vectors; delta-rs executes overlapping transactions. Download the verified results and artifact hashes, or the complete reproduction bundle. These captures are separate from the original three-version history.
Change how the same delete is stored
Keep file B. Remove its first row from the answer.
Return to the original version 1: A + B, eighteen observations. Fork that exact table and enable
deletion vectors. This branch's version 2 changes the protocol; version 3 deletes observation
13; version 4 also deletes observation 16. These branch versions are separate from the
copy-on-write history above.
All steps below come from Delta Spark 3.3.2 / Spark 3.5.6. File B has the same path and SHA-256 as the
original. Its six physical rows never change. The logical file changes when a new
deletion-vector descriptor is published with it.
DV branch version
Branch version 3: B contributes 5 rows; A contributes twelve; the table
has 17 rows. Visible B IDs: 14, 15, 16, 17, 18.
Find the descriptor
The add names the same Parquet path as the remove. The removed
logical identity includes its old DV, if any; the added identity includes the new one.
Replaying only by filename would lose this distinction. The reader reconciles both logical identities before publishing the new snapshot.
{
"storageType": "u",
"pathOrInlineDv": "54*e638{sfL$%/NaJsv)",
"offset": 1,
"sizeInBytes": 34,
"cardinality": 1
} storageType: u means the last twenty characters encode a UUID in Z85. That
resolves to deletion_vector_0fbbcf6b-09a9-446d-954f-44b220c5ff66.bin.
The descriptor points to byte 1, requests a 34-byte payload,
and promises 1 deleted position.
Inspect byte field
0100000022d1d339640100000000000000000000003a3000000100000000000000100000000000f7a6b4b5
Bytes [0, 1): File version → 1
A replacement file and a bitmap buy different work.
Original copy-on-write delete Five surviving rows copied into C; 1633 new Parquet bytes. Read A + C for seventeen rows.
This DV delete Zero new Parquet files. 43 bytes in the new DV sidecar, including framing
and checksum; 34 payload bytes. Read A + unchanged B, applying 1
deleted position.
These are artifact sizes for these captures, excluding commit JSON, reads, requests and
runtime overhead. The version-3 comparison has the same seventeen-row answer; version 4
deletes another row.
Old versions still need their original descriptors and data. The next native chapter
continues version 4 through a rewrite and physical cleanup.
Statistics can describe rows you must not return.
The native add action still records numRecords: 6 and
tightBounds: false. Its minimum observation ID remains 13. That is a safe lower
bound, not proof that ID 13 survives. Logical count is six minus DV cardinality; filtering
and aggregate answers must respect the mask.
Predict a new query
At branch version 4, the DV contains positions [0, 3]. Now ask for temperature_c ≥ 20. Use the physical temperatures in the diagram: which observation IDs should the reader
return?
Your prediction
Inspect the original atomic commit and native metrics
[
{
"commitInfo": {
"timestamp": 1789141334417,
"operation": "DELETE",
"operationParameters": {
"predicate": "[\"(observation_id#770L = 13)\"]"
},
"readVersion": 2,
"isolationLevel": "Serializable",
"isBlindAppend": false,
"operationMetrics": {
"numRemovedFiles": "0",
"numRemovedBytes": "0",
"numCopiedRows": "0",
"numDeletionVectorsAdded": "1",
"numDeletionVectorsRemoved": "0",
"numAddedChangeFiles": "0",
"executionTimeMs": "1538",
"numDeletionVectorsUpdated": "0",
"numDeletedRows": "1",
"scanTimeMs": "0",
"numAddedFiles": "0",
"numAddedBytes": "0",
"rewriteTimeMs": "0"
},
"engineInfo": "Apache-Spark/3.5.6 Delta-Lake/3.3.2",
"txnId": "101fef7b-770f-4a79-b3d2-a33c97a7a612"
}
},
{
"add": {
"path": "part-00000-62341c08-96d8-43a9-bcf1-5f1f6bd91945-c000.snappy.parquet",
"partitionValues": {},
"size": 1604,
"modificationTime": 1789067035253,
"dataChange": true,
"stats": "{\"numRecords\":6,\"minValues\":{\"observation_id\":13,\"station\":\"North\",\"observed_at\":\"2026-01-01T21:00:00.000Z\",\"temperature_c\":17.0},\"maxValues\":{\"observation_id\":18,\"station\":\"South\",\"observed_at\":\"2026-01-03T15:00:00.000Z\",\"temperature_c\":22.0},\"nullCount\":{\"observation_id\":0,\"station\":0,\"observed_at\":0,\"temperature_c\":0},\"tightBounds\":false}",
"deletionVector": {
"storageType": "u",
"pathOrInlineDv": "54*e638{sfL$%/NaJsv)",
"offset": 1,
"sizeInBytes": 34,
"cardinality": 1
}
}
},
{
"remove": {
"path": "part-00000-62341c08-96d8-43a9-bcf1-5f1f6bd91945-c000.snappy.parquet",
"deletionTimestamp": 1789141334196,
"dataChange": true,
"extendedFileMetadata": true,
"partitionValues": {},
"size": 1604,
"stats": "{\"numRecords\":6}"
}
}
] Download original commit 3
Native values were checked against the original source and the copy-on-write result. Python's
independent Roaring library and this bounded browser decoder agree on the stored positions. Pinned representation specification.
Keep the read version. Add another writer.
A stale result can be valid Parquet and still be wrong to publish.
These are executed delta-rs 1.6.3 transactions on copies of the original version 1,
without deletion vectors. Both table handles open before A commits. A finishes first; B then attempts
work using its old snapshot. The controlled schedule makes that overlap reproducible.
Native writer schedule
1. Both writers read version 1
Both handles read version 1. The pinned reader sees eighteen rows.
Inspect the two successful commits and the failed writer’s artifacts
Commit 2
[
{
"commitInfo": {
"timestamp": 1789141340320,
"operation": "DELETE",
"operationParameters": {
"predicate": "observation_id = 13"
},
"readVersion": 1,
"engineInfo": "delta-rs:py-1.6.3",
"clientVersion": "delta-rs.py-1.6.3",
"operationMetrics": {
"num_added_files": 1,
"num_removed_files": 1,
"num_deleted_rows": 1,
"num_copied_rows": 5,
"execution_time_ms": 8,
"scan_time_ms": 4,
"rewrite_time_ms": 0
}
}
},
{
"add": {
"path": "part-00000-b2eb0cea-e0a9-4636-825c-fca4e0db0bc4-c000.zstd.parquet",
"partitionValues": {},
"size": 1633,
"modificationTime": 1789141340320,
"dataChange": true,
"stats": "{\"numRecords\":5,\"minValues\":{\"observation_id\":14,\"temperature_c\":18.0,\"station\":\"North\",\"observed_at\":\"2026-01-03T03:00:00Z\"},\"maxValues\":{\"observed_at\":\"2026-01-03T15:00:00Z\",\"temperature_c\":22.0,\"observation_id\":18,\"station\":\"South\"},\"nullCount\":{\"observed_at\":0,\"observation_id\":0,\"temperature_c\":0,\"station\":0}}",
"tags": null,
"baseRowId": null,
"defaultRowCommitVersion": null,
"clusteringProvider": null
}
},
{
"remove": {
"path": "part-00000-62341c08-96d8-43a9-bcf1-5f1f6bd91945-c000.snappy.parquet",
"dataChange": true,
"deletionTimestamp": 1789141340317,
"extendedFileMetadata": true,
"partitionValues": {},
"size": 1604
}
}
]Commit 3
[
{
"commitInfo": {
"timestamp": 1789141340336,
"operation": "DELETE",
"operationParameters": {
"predicate": "observation_id = 14"
},
"readVersion": 2,
"engineInfo": "delta-rs:py-1.6.3",
"operationMetrics": {
"num_added_files": 1,
"num_removed_files": 1,
"num_deleted_rows": 1,
"num_copied_rows": 4,
"execution_time_ms": 2,
"scan_time_ms": 1,
"rewrite_time_ms": 0
},
"clientVersion": "delta-rs.py-1.6.3"
}
},
{
"add": {
"path": "part-00000-0bcd863c-86d2-46ab-9b7f-41dcdb2f064a-c000.zstd.parquet",
"partitionValues": {},
"size": 1620,
"modificationTime": 1789141340336,
"dataChange": true,
"stats": "{\"numRecords\":4,\"minValues\":{\"observed_at\":\"2026-01-03T06:00:00Z\",\"station\":\"North\",\"observation_id\":15,\"temperature_c\":19.0},\"maxValues\":{\"temperature_c\":22.0,\"station\":\"South\",\"observed_at\":\"2026-01-03T15:00:00Z\",\"observation_id\":18},\"nullCount\":{\"temperature_c\":0,\"station\":0,\"observed_at\":0,\"observation_id\":0}}",
"tags": null,
"baseRowId": null,
"defaultRowCommitVersion": null,
"clusteringProvider": null
}
},
{
"remove": {
"path": "part-00000-b2eb0cea-e0a9-4636-825c-fca4e0db0bc4-c000.zstd.parquet",
"dataChange": true,
"deletionTimestamp": 1789141340335,
"extendedFileMetadata": true,
"partitionValues": {},
"size": 1633
}
}
]overlap/_delta_log/00000000000000000000.json · 1597 bytes
overlap/_delta_log/00000000000000000001.json · 917 bytes
overlap/_delta_log/00000000000000000002.json · 1193 bytes
overlap/_delta_log/00000000000000000003.json · 1191 bytes
overlap/part-00000-0bcd863c-86d2-46ab-9b7f-41dcdb2f064a-c000.zstd.parquet · 1620 bytes
overlap/part-00000-62341c08-96d8-43a9-bcf1-5f1f6bd91945-c000.snappy.parquet · 1604 bytes
overlap/part-00000-b2eb0cea-e0a9-4636-825c-fca4e0db0bc4-c000.zstd.parquet · 1633 bytes
overlap/part-00000-bfdf29f6-7be0-4378-b1a2-346f035d0445-c000.snappy.parquet · 1716 bytes
overlap/part-00000-da1ec8d1-8f3e-4cbb-8f0d-8979b6e97741-c000.zstd.parquet · 1633 bytes
Predict a retry
Would renaming B's failed output and choosing commit 3 repair the overlap case? No: the bytes
still include deleted ID 13. A retry must re-evaluate the operation on current input. Changing a
commit number repairs neither stale data nor a stale read dependency.
Successful commits and failed replacement artifacts are original engine output. The rejected operation supplies the displayed error. Compare its staged files with the log: publication left the successful writer’s snapshot intact.
Recorded native execution · Spark 3.5.6 / Delta Spark 3.3.2
Remove the mask, then remove the old file.
The deletion-vector example ends at version 4 with sixteen visible rows in files containing eighteen physical rows. IDs 13 and 16 are masked. This experiment continues that table in a disposable copy: first materialize its deletes with REORG TABLE APPLY (PURGE), then clean up obsolete files with VACUUM.
REORG keeps the sixteen-row answer. It rewrites the masked file with only its survivors and removes the vector from active membership. The unchanged file stays active. The original masked file remains on disk, so version 4 still reads successfully. Running REORG again creates no new version in this experiment.
VACUUM changes historical availability. It removes the obsolete data and vector files. This pinned engine records VACUUM START and VACUUM END at versions 6 and 7; the latest answer still has sixteen rows. The old log can reconstruct version 4’s membership, but Spark reports a missing data file when asked to read it.
Zero retention with the duration check disabled only for this disposable experiment. It is not a production recommendation. The recipe copies the source table into a fresh temporary directory before executing cleanup.
Before materialization · version 4 · 16 logical rows
Result IDs: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 17, 18.
Complete native rows and file actions
{
"rows": [
{
"observation_id": 1,
"station": "North",
"observed_at": "2026-01-01T00:00:00.000000",
"temperature_c": 5
},
{
"observation_id": 2,
"station": "South",
"observed_at": "2026-01-01T03:00:00.000000",
"temperature_c": 6
},
{
"observation_id": 3,
"station": "North",
"observed_at": "2026-01-01T06:00:00.000000",
"temperature_c": 7
},
{
"observation_id": 4,
"station": "South",
"observed_at": "2026-01-01T09:00:00.000000",
"temperature_c": 8
},
{
"observation_id": 5,
"station": "North",
"observed_at": "2026-01-01T12:00:00.000000",
"temperature_c": 9
},
{
"observation_id": 6,
"station": "South",
"observed_at": "2026-01-01T15:00:00.000000",
"temperature_c": 10
},
{
"observation_id": 7,
"station": "North",
"observed_at": "2026-01-02T00:00:00.000000",
"temperature_c": 11
},
{
"observation_id": 8,
"station": "South",
"observed_at": "2026-01-02T03:00:00.000000",
"temperature_c": 12
},
{
"observation_id": 9,
"station": "North",
"observed_at": "2026-01-02T06:00:00.000000",
"temperature_c": 13
},
{
"observation_id": 10,
"station": "South",
"observed_at": "2026-01-02T09:00:00.000000",
"temperature_c": 14
},
{
"observation_id": 11,
"station": "North",
"observed_at": "2026-01-02T12:00:00.000000",
"temperature_c": 15
},
{
"observation_id": 12,
"station": "South",
"observed_at": "2026-01-02T15:00:00.000000",
"temperature_c": 16
},
{
"observation_id": 14,
"station": "South",
"observed_at": "2026-01-03T03:00:00.000000",
"temperature_c": 18
},
{
"observation_id": 15,
"station": "North",
"observed_at": "2026-01-03T06:00:00.000000",
"temperature_c": 19
},
{
"observation_id": 17,
"station": "North",
"observed_at": "2026-01-03T12:00:00.000000",
"temperature_c": 21
},
{
"observation_id": 18,
"station": "South",
"observed_at": "2026-01-03T15:00:00.000000",
"temperature_c": 22
}
],
"activeFiles": {
"part-00000-bfdf29f6-7be0-4378-b1a2-346f035d0445-c000.snappy.parquet": {
"path": "part-00000-bfdf29f6-7be0-4378-b1a2-346f035d0445-c000.snappy.parquet",
"partitionValues": {},
"size": 1716,
"modificationTime": 1789067035249,
"dataChange": true,
"stats": "{\"numRecords\":12,\"minValues\":{\"temperature_c\":5.0,\"observed_at\":\"2026-01-01T00:00:00Z\",\"observation_id\":1,\"station\":\"North\"},\"maxValues\":{\"observation_id\":12,\"temperature_c\":16.0,\"observed_at\":\"2026-01-02T15:00:00Z\",\"station\":\"South\"},\"nullCount\":{\"station\":0,\"observed_at\":0,\"temperature_c\":0,\"observation_id\":0}}",
"tags": null,
"baseRowId": null,
"defaultRowCommitVersion": null,
"clusteringProvider": null
},
"part-00000-62341c08-96d8-43a9-bcf1-5f1f6bd91945-c000.snappy.parquet": {
"path": "part-00000-62341c08-96d8-43a9-bcf1-5f1f6bd91945-c000.snappy.parquet",
"partitionValues": {},
"size": 1604,
"modificationTime": 1789067035253,
"dataChange": true,
"stats": "{\"numRecords\":6,\"minValues\":{\"observation_id\":13,\"station\":\"North\",\"observed_at\":\"2026-01-01T21:00:00.000Z\",\"temperature_c\":17.0},\"maxValues\":{\"observation_id\":18,\"station\":\"South\",\"observed_at\":\"2026-01-03T15:00:00.000Z\",\"temperature_c\":22.0},\"nullCount\":{\"observation_id\":0,\"station\":0,\"observed_at\":0,\"temperature_c\":0},\"tightBounds\":false}",
"deletionVector": {
"storageType": "u",
"pathOrInlineDv": ".{8>%1BS{ZOA.Ar+A>}K",
"offset": 1,
"sizeInBytes": 36,
"cardinality": 2
}
}
}
}Download this state’s original files (12)
After REORG · version 5 · 16 logical rows
Result IDs: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 17, 18.
Complete native rows and file actions
{
"rows": [
{
"observation_id": 1,
"station": "North",
"observed_at": "2026-01-01T00:00:00.000000",
"temperature_c": 5
},
{
"observation_id": 2,
"station": "South",
"observed_at": "2026-01-01T03:00:00.000000",
"temperature_c": 6
},
{
"observation_id": 3,
"station": "North",
"observed_at": "2026-01-01T06:00:00.000000",
"temperature_c": 7
},
{
"observation_id": 4,
"station": "South",
"observed_at": "2026-01-01T09:00:00.000000",
"temperature_c": 8
},
{
"observation_id": 5,
"station": "North",
"observed_at": "2026-01-01T12:00:00.000000",
"temperature_c": 9
},
{
"observation_id": 6,
"station": "South",
"observed_at": "2026-01-01T15:00:00.000000",
"temperature_c": 10
},
{
"observation_id": 7,
"station": "North",
"observed_at": "2026-01-02T00:00:00.000000",
"temperature_c": 11
},
{
"observation_id": 8,
"station": "South",
"observed_at": "2026-01-02T03:00:00.000000",
"temperature_c": 12
},
{
"observation_id": 9,
"station": "North",
"observed_at": "2026-01-02T06:00:00.000000",
"temperature_c": 13
},
{
"observation_id": 10,
"station": "South",
"observed_at": "2026-01-02T09:00:00.000000",
"temperature_c": 14
},
{
"observation_id": 11,
"station": "North",
"observed_at": "2026-01-02T12:00:00.000000",
"temperature_c": 15
},
{
"observation_id": 12,
"station": "South",
"observed_at": "2026-01-02T15:00:00.000000",
"temperature_c": 16
},
{
"observation_id": 14,
"station": "South",
"observed_at": "2026-01-03T03:00:00.000000",
"temperature_c": 18
},
{
"observation_id": 15,
"station": "North",
"observed_at": "2026-01-03T06:00:00.000000",
"temperature_c": 19
},
{
"observation_id": 17,
"station": "North",
"observed_at": "2026-01-03T12:00:00.000000",
"temperature_c": 21
},
{
"observation_id": 18,
"station": "South",
"observed_at": "2026-01-03T15:00:00.000000",
"temperature_c": 22
}
],
"activeFiles": {
"part-00000-bfdf29f6-7be0-4378-b1a2-346f035d0445-c000.snappy.parquet": {
"path": "part-00000-bfdf29f6-7be0-4378-b1a2-346f035d0445-c000.snappy.parquet",
"partitionValues": {},
"size": 1716,
"modificationTime": 1789067035249,
"dataChange": true,
"stats": "{\"numRecords\":12,\"minValues\":{\"temperature_c\":5.0,\"observed_at\":\"2026-01-01T00:00:00Z\",\"observation_id\":1,\"station\":\"North\"},\"maxValues\":{\"observation_id\":12,\"temperature_c\":16.0,\"observed_at\":\"2026-01-02T15:00:00Z\",\"station\":\"South\"},\"nullCount\":{\"station\":0,\"observed_at\":0,\"temperature_c\":0,\"observation_id\":0}}",
"tags": null,
"baseRowId": null,
"defaultRowCommitVersion": null,
"clusteringProvider": null
},
"part-00000-8ecc143a-9fea-4088-9005-d85a4f1d3e46-c000.snappy.parquet": {
"path": "part-00000-8ecc143a-9fea-4088-9005-d85a4f1d3e46-c000.snappy.parquet",
"partitionValues": {},
"size": 1376,
"modificationTime": 1789226809028,
"dataChange": false,
"stats": "{\"numRecords\":4,\"minValues\":{\"observation_id\":14,\"station\":\"North\",\"observed_at\":\"2026-01-03T03:00:00.000Z\",\"temperature_c\":18.0},\"maxValues\":{\"observation_id\":18,\"station\":\"South\",\"observed_at\":\"2026-01-03T15:00:00.000Z\",\"temperature_c\":22.0},\"nullCount\":{\"observation_id\":0,\"station\":0,\"observed_at\":0,\"temperature_c\":0},\"tightBounds\":true}"
}
}
}Download this state’s original files (18)
After VACUUM · version 7 · 16 logical rows
Result IDs: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 17, 18.
Complete native rows and file actions
{
"rows": [
{
"observation_id": 1,
"station": "North",
"observed_at": "2026-01-01T00:00:00.000000",
"temperature_c": 5
},
{
"observation_id": 2,
"station": "South",
"observed_at": "2026-01-01T03:00:00.000000",
"temperature_c": 6
},
{
"observation_id": 3,
"station": "North",
"observed_at": "2026-01-01T06:00:00.000000",
"temperature_c": 7
},
{
"observation_id": 4,
"station": "South",
"observed_at": "2026-01-01T09:00:00.000000",
"temperature_c": 8
},
{
"observation_id": 5,
"station": "North",
"observed_at": "2026-01-01T12:00:00.000000",
"temperature_c": 9
},
{
"observation_id": 6,
"station": "South",
"observed_at": "2026-01-01T15:00:00.000000",
"temperature_c": 10
},
{
"observation_id": 7,
"station": "North",
"observed_at": "2026-01-02T00:00:00.000000",
"temperature_c": 11
},
{
"observation_id": 8,
"station": "South",
"observed_at": "2026-01-02T03:00:00.000000",
"temperature_c": 12
},
{
"observation_id": 9,
"station": "North",
"observed_at": "2026-01-02T06:00:00.000000",
"temperature_c": 13
},
{
"observation_id": 10,
"station": "South",
"observed_at": "2026-01-02T09:00:00.000000",
"temperature_c": 14
},
{
"observation_id": 11,
"station": "North",
"observed_at": "2026-01-02T12:00:00.000000",
"temperature_c": 15
},
{
"observation_id": 12,
"station": "South",
"observed_at": "2026-01-02T15:00:00.000000",
"temperature_c": 16
},
{
"observation_id": 14,
"station": "South",
"observed_at": "2026-01-03T03:00:00.000000",
"temperature_c": 18
},
{
"observation_id": 15,
"station": "North",
"observed_at": "2026-01-03T06:00:00.000000",
"temperature_c": 19
},
{
"observation_id": 17,
"station": "North",
"observed_at": "2026-01-03T12:00:00.000000",
"temperature_c": 21
},
{
"observation_id": 18,
"station": "South",
"observed_at": "2026-01-03T15:00:00.000000",
"temperature_c": 22
}
],
"activeFiles": {
"part-00000-bfdf29f6-7be0-4378-b1a2-346f035d0445-c000.snappy.parquet": {
"path": "part-00000-bfdf29f6-7be0-4378-b1a2-346f035d0445-c000.snappy.parquet",
"partitionValues": {},
"size": 1716,
"modificationTime": 1789067035249,
"dataChange": true,
"stats": "{\"numRecords\":12,\"minValues\":{\"temperature_c\":5.0,\"observed_at\":\"2026-01-01T00:00:00Z\",\"observation_id\":1,\"station\":\"North\"},\"maxValues\":{\"observation_id\":12,\"temperature_c\":16.0,\"observed_at\":\"2026-01-02T15:00:00Z\",\"station\":\"South\"},\"nullCount\":{\"station\":0,\"observed_at\":0,\"temperature_c\":0,\"observation_id\":0}}",
"tags": null,
"baseRowId": null,
"defaultRowCommitVersion": null,
"clusteringProvider": null
},
"part-00000-8ecc143a-9fea-4088-9005-d85a4f1d3e46-c000.snappy.parquet": {
"path": "part-00000-8ecc143a-9fea-4088-9005-d85a4f1d3e46-c000.snappy.parquet",
"partitionValues": {},
"size": 1376,
"modificationTime": 1789226809028,
"dataChange": false,
"stats": "{\"numRecords\":4,\"minValues\":{\"observation_id\":14,\"station\":\"North\",\"observed_at\":\"2026-01-03T03:00:00.000Z\",\"temperature_c\":18.0},\"maxValues\":{\"observation_id\":18,\"station\":\"South\",\"observed_at\":\"2026-01-03T15:00:00.000Z\",\"temperature_c\":22.0},\"nullCount\":{\"observation_id\":0,\"station\":0,\"observed_at\":0,\"temperature_c\":0},\"tightBounds\":true}"
}
}
}Download this state’s original files (25)
Inspect the actual historical-read failure
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 168.0 failed 1 times, most recent failure: Lost task 1.0 in stage 168.0 (TID 127) (ayo-mbp.tailb899c1.ts.net executor driver): org.apache.spark.SparkFileNotFoundException: File file:<temporary>/table/part-00000-62341c08-96d8-43a9-bcf1-5f1f6bd91945-c000.snappy.parquet does not exist
It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved.
at org.apache.spark.sql.errors.QueryExecutionErrors$.readCurrentFileNotFoundError(QueryExecutionErrors.scala:781)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:222)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:282)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:131)
at org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:593)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.util.random.SamplingUtils$.reservoirSampleAndCount(SamplingUtils.scala:41)
at org.apache.spark.RangePartitioner$.$anonfun$sketch$1(Partitioner.scala:322)
at org.apache.spark.RangePartitioner$.$anonfun$sketch$1$adapted(Partitioner.scala:320)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2(RDD.scala:910)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2$adapted(RDD.scala:910)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:621)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:624)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Driver stacktrace:
Check your understanding: what survives cleanup?
The version 4 log is still present, and version 7 returns all sixteen expected rows. Can you recover version 4 merely by replaying its log?
Reveal the answer
Replay identifies the old files and their vector. Recovering the old rows requires restoring those files. Current rows, reconstructible membership, and readable historical data are three separate properties.
Download all three tables and the recipe · Complete capture and SHA-256 inventory · Pinned reproduction and verifier · Reproduction instructions. Every captured state is read again with Spark; PyArrow checks physical IDs independently.
After reconstructing and reading the table
What can make that answer unsafe to publish or impossible to read?
Keep files A, B and C and the same delete operation. After the recorded native schedules, explore additional protocol assumptions, then remove retained artifacts. The retention version selector also updates the real snapshot inspection above.
Explore the supplemental protocol model
Two writers can stage files. Only a valid commit publishes them.
Start both writers and a reader at version 1, whose files are A + B. Writer A performs the fixture’s real replacement of B by C. Writer B either plans another replacement of B or a blind append of a new file D. The extra writer and D are an explicit protocol scenario. In the append variant, D has no read dependency on removed B by assumption. Compare this assumption with the native delta-rs capture above: that client tracked a read dependency and rejected the stale append.
Writer B intends to
1. Read version 1
Published table
Version 1: A + B
AB
Reader pinned at version 1
A + B · eighteen rows
The reader stays pinned to its selected version while another writer commits.
Both writers observed version 1. Neither has permission to overwrite the next commit.
Files written by an unsuccessful transaction can remain as orphans. Membership still depends on a successful commit. The eventual cleanup decision must account for retention and in-flight operations; cleanup must distinguish abandoned files from files still being prepared.
Atomic publication and conflict validation answer different questions. Atomicity prevents readers from seeing half a commit. Validation determines whether an operation is still valid after intervening commits. Details vary with operation and isolation rules; the scenarios above isolate file replacement, a blind append, and a metadata conflict rather than claiming a complete conflict detector.
A version can be known yet no longer readable.
Time travel needs two things: enough log or checkpoint information to reconstruct membership, and every data file that membership references. Explore hypothetical removal below. No bundled artifact is actually deleted.
Recognizing Parquet is not enough to read a Delta table.
A reader that ignores required table features can return plausible but wrong rows. For example, a deletion vector can invalidate row positions while leaving the Parquet file in place. A reader must understand the table’s required capabilities before treating physical rows as logical rows.
The real delete replaces B’s six rows with C’s five survivors. It writes 1,633 bytes of replacement data to remove one logical row. The replacement’s size measures copy-on-write amplification for this file and codec. Larger affected files
can copy more unchanged data; deletion vectors trade that rewrite for additional read-time state
and maintenance.
Delta protocol: versions, features, reconciliation and readers
Similar histories, different metadata.
The Iceberg example expresses a similar 12 → 18 → 17 row-count progression through snapshots and manifests. Its delete removes observation 3; this Delta history removes 13, so the final rows differ. This Delta example reconstructs membership from ordered log actions and checkpoints. Both retain historical data files until a separate cleanup makes them unavailable.
This Delta table uses its original schema without column mapping. A rename that preserves physical field identity requires the relevant table feature.
Your turn.
Open your own _delta_log and find an add or remove action. Follow its path to the file whose membership changed.