Same question, less intermediate work
Why attach a description to a row we will discard?
We want warm observations, their station descriptions, and increasing observation IDs. The timestamp is irrelevant to this answer. Keep this query in view as its implementation changes:
SELECT d.observation_id, s.description, d.temperature_c
FROM data d
JOIN stations s USING (station)
WHERE d.temperature_c >= 18
ORDER BY d.observation_id
The unoptimized logical plan places the temperature filter above the inner join. Because that predicate depends only on the weather input, the optimizer can move it below the join. Switch between these equivalent relational orders.
Weather input18 observationsid · station · temperature
↓
temperature ≥ 1818 → 5 rows
↓
Inner join5 weather rows probe the lookup
← North / South
2 retained keys
↓
Order by observation ID14 · 15 · 16 · 17 · 18
ID 14: reaches the join; survives into the ordered result.
5 intermediate joined rows, five final rows. Moving the filter changes how many fixture rows reach the join. DataFusion’s recorded optimized plan chooses the earlier filter.
What makes the rewrite legal?
Every surviving weather row finds the same station matches in either order. The predicate reads no lookup columns and has no side effects. A predicate on the description could not be evaluated from weather alone. Outer joins and predicates involving nulls require their own equivalence checks.
Projection moves too: observed_at is dropped from the scan’s requested columns. station must stay until the join, even though the final output only contains its description. Avoiding a column and rejecting rows are different decisions.
Locate this rewrite in the captured DataFusion plans
Before optimization
Sort: d.observation_id ASC NULLS LAST
Projection: d.observation_id, s.description, d.temperature_c
Filter: d.temperature_c >= Int64(18)
Inner Join: Using d.station = s.station
SubqueryAlias: d
TableScan: data
SubqueryAlias: s
TableScan: stationsAfter optimization
Sort: d.observation_id ASC NULLS LAST
Projection: d.observation_id, s.description, d.temperature_c
Inner Join: d.station = CAST(s.station AS Utf8View)
SubqueryAlias: d
Filter: data.temperature_c >= Float64(18)
TableScan: data projection=[observation_id, station, temperature_c], partial_filters=[data.temperature_c >= Float64(18)]
SubqueryAlias: s
TableScan: stations projection=[station, description]Next, implement the optimizer’s recorded choice: filter before the join. The same five warm rows become probes; the two station descriptions become retained state.
Ten returned rows can still require a million input rows.
This separate experiment uses one million synthetic events and a 500,000-row lookup. DuckDB 1.4.4 executes five queries at two memory settings; Polars 1.44.1 / PyArrow 25.0.1 independently verifies every successful result in full. These are recorded native runs, not browser benchmarks or playback timings.
Each run starts a fresh connection with one DuckDB thread, insertion-order preservation off and at most 1 GB of temporary disk space. The buffer setting is written in decimal MB; measurements below use MiB. Whole-process resident memory also includes allocations outside that managed budget.
The wide aggregation failed at 48 MB. The lower-cardinality aggregation completed. The join and full sort used temporary disk space at the lower limit. Top-10 retained less sorting state and did not spill, but its profile still reports all one million event rows scanned.
The join scans both inputs, so its reported total is 1.5 million rows. Scanned-row counters count rows examined at the inputs. The tables distinguish output cardinality from scanned rows. Parent and child operators can count the same records at different stages.
The successful 256 MB aggregation’s reported peak is below 48 MB, yet the 48 MB run failed. The limit affects execution and allocation decisions; reducing the cap can change the allocations needed to finish.
500,000 aggregate groups · 48MB · out of memory
SELECT key, sum(id)::BIGINT AS total, count(*)::BIGINT AS n FROM events GROUP BY key ORDER BY key
The native execution did not produce a result. This failure is retained as evidence.
Out of Memory Error: could not allocate block of size 256.0 KiB (45.5 MiB/45.7 MiB used)
Possible solutions:
* Reducing the number of threads (SET threads=X)
* Disabling insertion-order preservation (SET preserve_insertion_order=false)
* Increasing the memory limit (SET memory_limit='...GB')
See also https://duckdb.org/docs/stable/guides/performance/how_to_tune_workloads
500,000 aggregate groups · 256MB · completed
SELECT key, sum(id)::BIGINT AS total, count(*)::BIGINT AS n FROM events GROUP BY key ORDER BY key
One captured execution: 0.043 s latency; 0.038 s reported CPU time. Use the pinned recipe to observe the same query under your own conditions.
First 8 of 500,000 exact result rows:
[
{
"key": 0,
"total": 500000,
"n": 2
},
{
"key": 1,
"total": 500002,
"n": 2
},
{
"key": 2,
"total": 500004,
"n": 2
},
{
"key": 3,
"total": 500006,
"n": 2
},
{
"key": 4,
"total": 500008,
"n": 2
},
{
"key": 5,
"total": 500010,
"n": 2
},
{
"key": 6,
"total": 500012,
"n": 2
},
{
"key": 7,
"total": 500014,
"n": 2
}
] Last 8 result rows:
[
{
"key": 499992,
"total": 1499984,
"n": 2
},
{
"key": 499993,
"total": 1499986,
"n": 2
},
{
"key": 499994,
"total": 1499988,
"n": 2
},
{
"key": 499995,
"total": 1499990,
"n": 2
},
{
"key": 499996,
"total": 1499992,
"n": 2
},
{
"key": 499997,
"total": 1499994,
"n": 2
},
{
"key": 499998,
"total": 1499996,
"n": 2
},
{
"key": 499999,
"total": 1499998,
"n": 2
}
] Complete result SHA-256: ca7c525c513a26670afd0845708f3e2956b5012fc4eb61e07f55120dc9bd30f3
The digest covers every row in a canonical Arrow IPC stream with int64 columns. The verifier also compares the full native and Polars tables before computing it. Download the original normalized native profile.
4,096 aggregate groups · 48MB · completed
SELECT key % 4096 AS bucket, sum(id)::BIGINT AS total, count(*)::BIGINT AS n FROM events GROUP BY bucket ORDER BY bucket
One captured execution: 0.013 s latency; 0.012 s reported CPU time. Use the pinned recipe to observe the same query under your own conditions.
First 8 of 4,096 exact result rows:
[
{
"bucket": 0,
"total": 122964576,
"n": 246
},
{
"bucket": 1,
"total": 122964822,
"n": 246
},
{
"bucket": 2,
"total": 122965068,
"n": 246
},
{
"bucket": 3,
"total": 122965314,
"n": 246
},
{
"bucket": 4,
"total": 122965560,
"n": 246
},
{
"bucket": 5,
"total": 122965806,
"n": 246
},
{
"bucket": 6,
"total": 122966052,
"n": 246
},
{
"bucket": 7,
"total": 122966298,
"n": 246
}
] Last 8 result rows:
[
{
"bucket": 4088,
"total": 122462624,
"n": 244
},
{
"bucket": 4089,
"total": 122462868,
"n": 244
},
{
"bucket": 4090,
"total": 122463112,
"n": 244
},
{
"bucket": 4091,
"total": 122463356,
"n": 244
},
{
"bucket": 4092,
"total": 122463600,
"n": 244
},
{
"bucket": 4093,
"total": 122463844,
"n": 244
},
{
"bucket": 4094,
"total": 122464088,
"n": 244
},
{
"bucket": 4095,
"total": 122464332,
"n": 244
}
] Complete result SHA-256: e7b52d37fa6cef35712896f0c03e2e3e38965012400d34ad489cad02345766b3
The digest covers every row in a canonical Arrow IPC stream with int64 columns. The verifier also compares the full native and Polars tables before computing it. Download the original normalized native profile.
4,096 aggregate groups · 256MB · completed
SELECT key % 4096 AS bucket, sum(id)::BIGINT AS total, count(*)::BIGINT AS n FROM events GROUP BY bucket ORDER BY bucket
One captured execution: 0.012 s latency; 0.012 s reported CPU time. Use the pinned recipe to observe the same query under your own conditions.
First 8 of 4,096 exact result rows:
[
{
"bucket": 0,
"total": 122964576,
"n": 246
},
{
"bucket": 1,
"total": 122964822,
"n": 246
},
{
"bucket": 2,
"total": 122965068,
"n": 246
},
{
"bucket": 3,
"total": 122965314,
"n": 246
},
{
"bucket": 4,
"total": 122965560,
"n": 246
},
{
"bucket": 5,
"total": 122965806,
"n": 246
},
{
"bucket": 6,
"total": 122966052,
"n": 246
},
{
"bucket": 7,
"total": 122966298,
"n": 246
}
] Last 8 result rows:
[
{
"bucket": 4088,
"total": 122462624,
"n": 244
},
{
"bucket": 4089,
"total": 122462868,
"n": 244
},
{
"bucket": 4090,
"total": 122463112,
"n": 244
},
{
"bucket": 4091,
"total": 122463356,
"n": 244
},
{
"bucket": 4092,
"total": 122463600,
"n": 244
},
{
"bucket": 4093,
"total": 122463844,
"n": 244
},
{
"bucket": 4094,
"total": 122464088,
"n": 244
},
{
"bucket": 4095,
"total": 122464332,
"n": 244
}
] Complete result SHA-256: e7b52d37fa6cef35712896f0c03e2e3e38965012400d34ad489cad02345766b3
The digest covers every row in a canonical Arrow IPC stream with int64 columns. The verifier also compares the full native and Polars tables before computing it. Download the original normalized native profile.
Wide hash join · 48MB · completed
SELECT sum(length(e.payload || d.payload))::BIGINT AS checksum, count(*)::BIGINT AS n FROM events e JOIN lookup d USING (key)
One captured execution: 0.309 s latency; 0.304 s reported CPU time. Use the pinned recipe to observe the same query under your own conditions.
First 1 of 1 exact result rows:
[
{
"checksum": 128000000,
"n": 1000000
}
] Complete result SHA-256: 5880d049dc5289f1b20cd60ae7197f0c4be6f829098fef406d0675c06735644c
The digest covers every row in a canonical Arrow IPC stream with int64 columns. The verifier also compares the full native and Polars tables before computing it. Download the original normalized native profile.
Wide hash join · 256MB · completed
SELECT sum(length(e.payload || d.payload))::BIGINT AS checksum, count(*)::BIGINT AS n FROM events e JOIN lookup d USING (key)
One captured execution: 0.117 s latency; 0.112 s reported CPU time. Use the pinned recipe to observe the same query under your own conditions.
First 1 of 1 exact result rows:
[
{
"checksum": 128000000,
"n": 1000000
}
] Complete result SHA-256: 5880d049dc5289f1b20cd60ae7197f0c4be6f829098fef406d0675c06735644c
The digest covers every row in a canonical Arrow IPC stream with int64 columns. The verifier also compares the full native and Polars tables before computing it. Download the original normalized native profile.
Full string sort · 48MB · completed
SELECT id FROM events ORDER BY payload, id
One captured execution: 0.350 s latency; 0.316 s reported CPU time. Use the pinned recipe to observe the same query under your own conditions.
First 8 of 1,000,000 exact result rows:
[
{
"id": 848775
},
{
"id": 752491
},
{
"id": 738639
},
{
"id": 5329
},
{
"id": 79042
},
{
"id": 241361
},
{
"id": 552871
},
{
"id": 646434
}
] Last 8 result rows:
[
{
"id": 789215
},
{
"id": 986128
},
{
"id": 616604
},
{
"id": 233443
},
{
"id": 373470
},
{
"id": 95102
},
{
"id": 803622
},
{
"id": 40691
}
] Complete result SHA-256: fafa1331f2fe3437c994c667494b36e7464fdc57f612804df25fd253fb1ae202
The digest covers every row in a canonical Arrow IPC stream with int64 columns. The verifier also compares the full native and Polars tables before computing it. Download the original normalized native profile.
Full string sort · 256MB · completed
SELECT id FROM events ORDER BY payload, id
One captured execution: 0.183 s latency; 0.127 s reported CPU time. Use the pinned recipe to observe the same query under your own conditions.
First 8 of 1,000,000 exact result rows:
[
{
"id": 848775
},
{
"id": 752491
},
{
"id": 738639
},
{
"id": 5329
},
{
"id": 79042
},
{
"id": 241361
},
{
"id": 552871
},
{
"id": 646434
}
] Last 8 result rows:
[
{
"id": 789215
},
{
"id": 986128
},
{
"id": 616604
},
{
"id": 233443
},
{
"id": 373470
},
{
"id": 95102
},
{
"id": 803622
},
{
"id": 40691
}
] Complete result SHA-256: fafa1331f2fe3437c994c667494b36e7464fdc57f612804df25fd253fb1ae202
The digest covers every row in a canonical Arrow IPC stream with int64 columns. The verifier also compares the full native and Polars tables before computing it. Download the original normalized native profile.
Same order, LIMIT 10 · 48MB · completed
SELECT id FROM events ORDER BY payload, id LIMIT 10
One captured execution: 0.033 s latency; 0.033 s reported CPU time. Use the pinned recipe to observe the same query under your own conditions.
First 8 of 10 exact result rows:
[
{
"id": 848775
},
{
"id": 752491
},
{
"id": 738639
},
{
"id": 5329
},
{
"id": 79042
},
{
"id": 241361
},
{
"id": 552871
},
{
"id": 646434
}
] Last 8 result rows:
[
{
"id": 738639
},
{
"id": 5329
},
{
"id": 79042
},
{
"id": 241361
},
{
"id": 552871
},
{
"id": 646434
},
{
"id": 414859
},
{
"id": 194813
}
] Complete result SHA-256: c1196b1878223b1b92d24ae9c5f8537e3c42602d4a3699dc6115a3901291b063
The digest covers every row in a canonical Arrow IPC stream with int64 columns. The verifier also compares the full native and Polars tables before computing it. Download the original normalized native profile.
Same order, LIMIT 10 · 256MB · completed
SELECT id FROM events ORDER BY payload, id LIMIT 10
One captured execution: 0.031 s latency; 0.031 s reported CPU time. Use the pinned recipe to observe the same query under your own conditions.
First 8 of 10 exact result rows:
[
{
"id": 848775
},
{
"id": 752491
},
{
"id": 738639
},
{
"id": 5329
},
{
"id": 79042
},
{
"id": 241361
},
{
"id": 552871
},
{
"id": 646434
}
] Last 8 result rows:
[
{
"id": 738639
},
{
"id": 5329
},
{
"id": 79042
},
{
"id": 241361
},
{
"id": 552871
},
{
"id": 646434
},
{
"id": 414859
},
{
"id": 194813
}
] Complete result SHA-256: c1196b1878223b1b92d24ae9c5f8537e3c42602d4a3699dc6115a3901291b063
The digest covers every row in a canonical Arrow IPC stream with int64 columns. The verifier also compares the full native and Polars tables before computing it. Download the original normalized native profile.
Reproduce the inputs, outputs and profile evidence
Captured 2026-09-12T17:10:41.932640+00:00 on Darwin / arm64. Complete capture, input hashes and oracle results · Pinned reproduction script.
Run uv run reproduce.py --output new-capture in a directory of your choice. The script regenerates both input tables in a temporary workspace, writes deterministic Parquet inputs, verifies them with an independent reader, and records fresh profiles. The reproduction script generates its inputs locally. This page presents the recorded result.
{
"events": {
"rows": 1000000,
"sql": "SELECT i AS id, i % 500000 AS key, (i * 104729) % 1000000 AS score, repeat(md5(i::VARCHAR), 2) AS payload FROM range(1000000) t(i)",
"parquetBytes": 50319922,
"parquetSha256": "10ed782b8fced6891ce674a4f826b6caf345b4e5c851ea5b7b2a09c3adad9604"
},
"lookup": {
"rows": 500000,
"sql": "SELECT i AS key, i * 3 AS amount, repeat(md5(i::VARCHAR), 2) AS payload FROM range(500000) t(i)",
"parquetBytes": 22663755,
"parquetSha256": "4eab6c07075e5a141fa5745aea5bd8c25a72357cce4b48b646646cdeb08c5d1d"
}
}Only the temporary workspace path is replaced. Timings and native metrics are retained. Result digests are canonical uncompressed Arrow IPC streams with int64 columns and batches of 65536 rows.
DuckDB metric definitions · Blocking operators and spill limits. Behavior outside the pinned engine, queries and configuration requires a new capture.
Challenge the estimate. Check the answer.
DuckDB 1.4.4 runs twelve small native experiments. Polars 1.44.1 / PyArrow 25.0.1 checks the complete result of every query and the generated inputs independently. Select a captured execution to compare its plan, estimates and measured result.
Optimization counterexample
6 optimization comparisons shown.
Remove the statistics, keep the records.
Both uncompressed Parquet files contain the same 100,000 sorted records in 25 row groups. One stores column bounds; the other omits them. The query asks for ID 43.
Both answers are 731. Bounds rule out 24 groups in the original file. The native byte counter increases without statistics, while both scan operators report 100,000 rows scanned. Inspect the source and pruning counters separately to understand the work skipped.
Inspect 2 recorded results and plans
stats
SELECT sum(value)::BIGINT AS total FROM read_parquet('stats.parquet') WHERE id=43 Complete result:
[
{
"total": 731
}
] Native total_bytes_read: 7,416. Interpret this as the engine’s reported byte counter. An in-memory input still requires CPU and memory work.
Download stats native profile · Complete result SHA-256: 71e1596bf552a2397b10933234988e08020a02b697cfa719d10751963764afe0
missing
SELECT sum(value)::BIGINT AS total FROM read_parquet('missing.parquet') WHERE id=43 Complete result:
[
{
"total": 731
}
] Native total_bytes_read: 17,404. Interpret this as the engine’s reported byte counter. An in-memory input still requires CPU and memory work.
Download missing native profile · Complete result SHA-256: 71e1596bf552a2397b10933234988e08020a02b697cfa719d10751963764afe0
A pushed filter is not always a usable range.
Compare integer equality with equality after casting the same ID to text. The fixture has nonnegative integer IDs, so these particular predicates select exactly the same row.
Both predicates appear inside READ_PARQUET. The cast case reports more bytes read. Compare the scan’s actual pruning information to see whether integer bounds still eliminate the same work. The predicate’s type conversion determines which bounds the optimizer can use.
Inspect 2 recorded results and plans
stats
SELECT sum(value)::BIGINT AS total FROM read_parquet('stats.parquet') WHERE id=43 Complete result:
[
{
"total": 731
}
] Native total_bytes_read: 7,416. Interpret this as the engine’s reported byte counter. An in-memory input still requires CPU and memory work.
Download stats native profile · Complete result SHA-256: 71e1596bf552a2397b10933234988e08020a02b697cfa719d10751963764afe0
cast
SELECT sum(value)::BIGINT AS total FROM read_parquet('stats.parquet') WHERE CAST(id AS VARCHAR)='43' Complete result:
[
{
"total": 731
}
] Native total_bytes_read: 19,704. Interpret this as the engine’s reported byte counter. An in-memory input still requires CPU and memory work.
Download cast native profile · Complete result SHA-256: 71e1596bf552a2397b10933234988e08020a02b697cfa719d10751963764afe0
The same marginal distributions can hide a relationship.
Each column contains 0–99 exactly 100 times in both tables. The independent table contains every pair once; the correlated table has a=b in every row. Query a<10 AND b>=90.
The independent table returns count 100; the correlated table returns 0. Both scan estimates are 2,000. The joint distribution also depends on how column values occur together in rows.
Inspect 2 recorded results and plans
independent
SELECT count(*) AS n FROM independent WHERE a<10 AND b>=90
Complete result:
[
{
"n": 100
}
] Native total_bytes_read: 0. Interpret this as the engine’s reported byte counter. An in-memory input still requires CPU and memory work.
Download independent native profile · Complete result SHA-256: 4b5975cfbafa512614c1808d5376af0a8fe15937048ab837d73b3f324686a9a4
correlated
SELECT count(*) AS n FROM correlated WHERE a<10 AND b>=90
Complete result:
[
{
"n": 0
}
] Native total_bytes_read: 0. Interpret this as the engine’s reported byte counter. An in-memory input still requires CPU and memory work.
Download correlated native profile · Complete result SHA-256: 9edf26a11e8aad8749dabfe0f58c5b1739d2cfb434fa6b6af0e30777f1314189
Distinct keys do not describe their frequencies.
Both tables have 10,000 rows and keys 0–99. The uniform table repeats each key 100 times. In the skewed table, the first 9,500 rows use key 0; five later rows also use it.
Filtering key=0 produces 100 versus 9,505 rows. Inspect the estimates beside the actual outputs. Join multiplicity depends on each key’s frequency in both inputs.
Inspect 2 recorded results and plans
uniform
SELECT count(*) AS n FROM uniform WHERE key=0
Complete result:
[
{
"n": 100
}
] Native total_bytes_read: 0. Interpret this as the engine’s reported byte counter. An in-memory input still requires CPU and memory work.
Download uniform native profile · Complete result SHA-256: 4b5975cfbafa512614c1808d5376af0a8fe15937048ab837d73b3f324686a9a4
skew
SELECT count(*) AS n FROM skew WHERE key=0
Complete result:
[
{
"n": 9505
}
] Native total_bytes_read: 0. Interpret this as the engine’s reported byte counter. An in-memory input still requires CPU and memory work.
Download skew native profile · Complete result SHA-256: 02516a999d7a18c180c3364777160454a90ee5a2364b289b801aada66b4bc305
Refresh an estimate after changing the distribution.
Start with the uniform table and ANALYZE it. Then UPDATE key=id, changing 100 distinct keys into 10,000. Compare the updated table before and after another ANALYZE.
The answer changes from 100 matching rows to one after the update. Before refreshing, the scan estimate remains 103; afterward it is 2. The refreshed estimate is closer and still not exact. The pinned engine’s estimates let you compare its before-and-after view of the same data.
Inspect 3 recorded results and plans
stats-before
Before the query:
ANALYZE uniform
SELECT count(*) AS n, sum(id)::BIGINT AS total FROM uniform WHERE key=43
Complete result:
[
{
"n": 100,
"total": 499300
}
] Native total_bytes_read: 0. Interpret this as the engine’s reported byte counter. An in-memory input still requires CPU and memory work.
Download stats-before native profile · Complete result SHA-256: 8281a8301710ac3dbc8929c73d2c93a6736a8738351ad19585129345993fc2b1
stats-stale
Before the query:
ANALYZE uniform;
UPDATE uniform SET key=id
SELECT count(*) AS n, sum(id)::BIGINT AS total FROM uniform WHERE key=43
Complete result:
[
{
"n": 1,
"total": 43
}
] Native total_bytes_read: 0. Interpret this as the engine’s reported byte counter. An in-memory input still requires CPU and memory work.
Download stats-stale native profile · Complete result SHA-256: 7a9914d46ab8dc57945c34b4a5f0352f8fe423f06c201d81114a96d942dea2ce
stats-refreshed
Before the query:
ANALYZE uniform;
UPDATE uniform SET key=id;
ANALYZE uniform
SELECT count(*) AS n, sum(id)::BIGINT AS total FROM uniform WHERE key=43
Complete result:
[
{
"n": 1,
"total": 43
}
] Native total_bytes_read: 0. Interpret this as the engine’s reported byte counter. An in-memory input still requires CPU and memory work.
Download stats-refreshed native profile · Complete result SHA-256: 7a9914d46ab8dc57945c34b4a5f0352f8fe423f06c201d81114a96d942dea2ce
The same answer can pass through different intermediate sizes.
Join 100,000 fact rows to a repeated-key table and a selective table. Run the same SQL with normal optimization, then disable join_order and build_side_probe_side to preserve its written join sequence.
Both return n=1,000 and total=45,004,500. The earlier join produces 100 rows in the automatic plan and 1,000 in the written sequence. Dynamic filtering reduces the fact scan in both runs before the join materializes its result.
Inspect 2 recorded results and plans
join-auto
SELECT count(*) AS n, sum(f.id)::BIGINT AS total FROM fact f JOIN wide w USING(b) JOIN selective s USING(a) WHERE s.keep=1
Complete result:
[
{
"n": 1000,
"total": 45004500
}
] Native total_bytes_read: 0. Interpret this as the engine’s reported byte counter. An in-memory input still requires CPU and memory work.
Download join-auto native profile · Complete result SHA-256: 4afb141d95c2b722ac1d8e0f5713d88011c909e93fc71de13eb64115e4b18ea2
join-written
Before the query:
SET disabled_optimizers='join_order,build_side_probe_side'
SELECT count(*) AS n, sum(f.id)::BIGINT AS total FROM fact f JOIN wide w USING(b) JOIN selective s USING(a) WHERE s.keep=1
Complete result:
[
{
"n": 1000,
"total": 45004500
}
] Native total_bytes_read: 0. Interpret this as the engine’s reported byte counter. An in-memory input still requires CPU and memory work.
Download join-written native profile · Complete result SHA-256: 4afb141d95c2b722ac1d8e0f5713d88011c909e93fc71de13eb64115e4b18ea2
Reproduce these observations
Captured 2026-09-12T22:57:39.637430+00:00 on Darwin / arm64. Each query starts a fresh connection with one thread. Only the Parquet comparisons have file-level byte counters. No timing rankings or portable speed thresholds are asserted.
Complete capture and hashes · Pinned recipe · Parquet with statistics · Parquet without statistics
Run uv run reproduce.py --output new-capture to regenerate the inputs and produce fresh native profiles. Run with --verify --output existing-capture to compare exact results, native estimates and cardinalities, preserved hashes and the stated pairwise differences. Timings and exact byte counters may vary; the recorded originals remain unchanged.
Only the temporary workspace path is replaced. Native timings and counters remain in each profile. Input/result digests cover canonical int64 Arrow IPC streams. Byte counters describe this engine execution, not physical disk traffic.
{
"correlated": {
"sql": "SELECT i%100 AS a, i%100 AS b FROM range(10000) t(i)",
"rows": 10000,
"sha256": "175ef45bfb2e7afd8cd5c5e8e5376c357333695868959e1f86b9543047862ced"
},
"independent": {
"sql": "SELECT i%100 AS a, (i//100)%100 AS b FROM range(10000) t(i)",
"rows": 10000,
"sha256": "85879aadd1d33a53c8a0fe8a97d9e38e6a26f42b705c5b4f4e2883882b350ed0"
},
"uniform": {
"sql": "SELECT i AS id, i%100 AS key FROM range(10000) t(i)",
"rows": 10000,
"sha256": "3229e23f047058d6416162c000bf5b413ee47d517b707050b0a5a01d7ce6789c"
},
"skew": {
"sql": "SELECT i AS id, CASE WHEN i<9500 THEN 0 ELSE i%100 END AS key FROM range(10000) t(i)",
"rows": 10000,
"sha256": "92379ba72e19f93babcc05e7d569b3faccfbfd3a22be96ab7bc3317787254309"
},
"fact": {
"sql": "SELECT i AS id, i%10000 AS a, i%100 AS b FROM range(100000) t(i)",
"rows": 100000,
"sha256": "e33c66be2414d6a8e11536242776ddd8ae6925448db66c2b8ff160547c630518"
},
"wide": {
"sql": "SELECT i%100 AS b FROM range(1000) t(i)",
"rows": 1000,
"sha256": "704ba785faba6a4d9bb4ce04d55c0c84c8fa802ddc88b65df469cf671d53c2af"
},
"selective": {
"sql": "SELECT i AS a, CASE WHEN i<10 THEN 1 ELSE 0 END AS keep FROM range(10000) t(i)",
"rows": 10000,
"sha256": "142063796db3493278342f91c9367ae9795a4c64e97a0b784a3e510eb86ec782"
}
} Join-order controls · ANALYZE · Parquet filter pushdown. The profiles here record DuckDB 1.4.4; current documentation can describe newer releases.