A few rows, many layers.
Start with the artefact
Start with a search.
The cross marks (2.25, 1.125). Which 3 stored vectors are closest?
What Lance is
Lance combines a columnar file format, a versioned table format and optional vector indexes. Machine-learning workloads need both selected rows and nearest-neighbor searches from the same store. Parquet groups columns into row groups for analytical scans; Lance pages and row addresses support reaching selected values. Its manifests connect fragments into table versions, while indexes narrow a search. The Lance SDK and LanceDB write these structures and read them back.
About this fixture
Captured with pylance 11.0.0, file format 2.1. Exact answers were checked with NumPy. Original files are fetched and hash-checked for byte inspection. IVF_FLAT, lifecycle and physical-layout experiments have separate native captures. Search diagrams show recorded results and calculated distances.
Which vectors are closest?
Each point is a synthetic vector. Choose a recorded exact Lance query with use_index=False. Independent NumPy squared Euclidean distances produce the same three IDs and distances.
Query point
Prefilter group
The group predicate runs before nearest-neighbor selection. Distance is (x − qx)² + (y − qy)², so this column is squared distance, not its square root. The marked points and lines visualize the recorded answer.
Make every candidate account for its distance.
The point and group controls above drive this calculation too. Every row stays visible, including ineligible rows and those outside the first three. Select any candidate to separate its horizontal and vertical contributions.
ID 4: two differences, one score
x: 3 − 2.25 = 0.75; squared = 0.5625
y: 1 − 1.125 = -0.125; squared = 0.015625
Squared distance = 0.5625 + 0.015625 = 0.578125
Filter, then take three
12 eligible candidates → IDs 4, 2, 5
Take three, then filter
Global top three: 4, 2, 5
After group filter: 4, 2, 5 · 3 rows
These operations answer different questions. Taking three first can discard an eligible fourth-nearest row and leave fewer than three answers after filtering. The recorded SDK query uses prefiltering; all eighteen point/group combinations are checked against independent NumPy distances.
This exact example evaluates the mathematical distance for every eligible candidate. An approximate index adds candidate selection before final ranking and can miss a true nearest neighbor. Compare its returned IDs with the exact neighbors to measure recall.
The manifest connects the table.
The version 1 manifest references two fragments. Each fragment contains six rows in one data file, with field IDs and column indices linking the schema to the physical columns.
Manifest · version 1_versions/18446744073709551614.manifest
Read the version from the manifest’s contents. Its long numeric filename stays a string throughout this guide. The decoded protobuf says version 1.
Decoded manifest from the version-matched protobuf schema
{
"fields": [
{
"name": "id",
"parent_id": -1,
"logical_type": "int64",
"nullable": true,
"encoding": "PLAIN"
},
{
"name": "group",
"id": 1,
"parent_id": -1,
"logical_type": "string",
"nullable": true,
"encoding": "VAR_BINARY"
},
{
"name": "vector",
"id": 2,
"parent_id": -1,
"logical_type": "fixed_size_list:float:2",
"nullable": true,
"encoding": "PLAIN"
}
],
"fragments": [
{
"files": [
{
"path": "011100000111000101111010480cca4ad6982721bc32f9b3b3.lance",
"fields": [
0,
1,
2
],
"column_indices": [
0,
1,
2
],
"file_major_version": 2,
"file_minor_version": 1,
"file_size_bytes": "947"
}
],
"physical_rows": "6"
},
{
"id": "1",
"files": [
{
"path": "111110111010010100111110f9ed7e4bf8ac34a9f93e293104.lance",
"fields": [
0,
1,
2
],
"column_indices": [
0,
1,
2
],
"file_major_version": 2,
"file_minor_version": 1,
"file_size_bytes": "947"
}
],
"physical_rows": "6"
}
],
"version": "1",
"timestamp": "2026-09-10T19:22:59.018815Z",
"max_fragment_id": 1,
"transaction_file": "0-088e28a3-ff90-4023-bdb3-58d75355441b.txn",
"writer_version": {
"library": "lance",
"version": "11.0.0"
},
"data_format": {
"file_format": "lance",
"version": "2.1"
},
"transaction_section": "0"
}Reader feature flags: 0; writer feature flags: 0. The fixture verifier rejects other flags instead of pretending to understand them. No deletions, overlays or indices are present.
Follow one file's byte ranges.
data/011100000111000101111010480cca4ad6982721bc32f9b3b3.lance · 947 bytes · 6 rows. These ranges were inspected using the official SDK and cross-checked against the original little-endian footer and offset tables.
column metadata: bytes 502 through 610, inclusive (109 bytes). Column id.
Column pages need not share Parquet-style row-group boundaries. A page's encoding describes its buffers. In this fixture, each column has one page; the vector encoding records a fixed-size list of two Float32 values.
Original SDK encoding descriptions
PageLayout {
layout: Some(
MiniBlockLayout(
MiniBlockLayout {
rep_compression: None,
def_compression: None,
value_compression: Some(
CompressiveEncoding {
compression: Some(
Flat(
Flat {
bits_per_value: 64,
data: None,
},
),
),
},
),
dictionary: None,
num_dictionary_items: 0,
layers: [
RepdefAllValidItem,
],
num_buffers: 1,
repetition_index_depth: 0,
num_items: 6,
has_large_chunk: false,
},
),
),
}PageLayout {
layout: Some(
MiniBlockLayout(
MiniBlockLayout {
rep_compression: None,
def_compression: None,
value_compression: Some(
CompressiveEncoding {
compression: Some(
Variable(
Variable {
offsets: Some(
CompressiveEncoding {
compression: Some(
Flat(
Flat {
bits_per_value: 32,
data: None,
},
),
),
},
),
values: None,
},
),
),
},
),
dictionary: None,
num_dictionary_items: 0,
layers: [
RepdefAllValidItem,
],
num_buffers: 1,
repetition_index_depth: 0,
num_items: 6,
has_large_chunk: false,
},
),
),
}PageLayout {
layout: Some(
MiniBlockLayout(
MiniBlockLayout {
rep_compression: None,
def_compression: None,
value_compression: Some(
CompressiveEncoding {
compression: Some(
FixedSizeList(
FixedSizeList {
items_per_value: 2,
has_validity: false,
values: Some(
CompressiveEncoding {
compression: Some(
Flat(
Flat {
bits_per_value: 32,
data: None,
},
),
),
},
),
},
),
),
},
),
dictionary: None,
num_dictionary_items: 0,
layers: [
RepdefAllValidItem,
],
num_buffers: 1,
repetition_index_depth: 0,
num_items: 6,
has_large_chunk: false,
},
),
),
}The final 40 bytes identify metadata-offset tables, counts, format version and the LANC signature. Padding and offset-table bytes are not painted as data buffers in the overview above.
Follow ID 4 back to eight value bytes.
An application ID identifies a record; a row position locates it in this particular snapshot. Here the source rows are ordered by ID, with six rows per fragment. Use the manifest’s physical row counts to map this fixture’s positions to fragments.
- Manifest version 1 selects fragment 0.
- ID 4 is at snapshot position 3, local position 3 in that fragment.
- Its manifest reference resolves to
data/011100000111000101111010480cca4ad6982721bc32f9b3b3.lance. - Field
vector maps to column 2, page 0, value buffer 1: offset 320, length 56 bytes. - After this fixture's eight-byte prefix, two Float32 values occupy bytes 352–359.
The SDK describes an all-valid fixed-size list with two flat 32-bit values. For these two specific files, we verify the eight-byte prefix 00 00 30 00 fe fe fe fe and read the following twelve little-endian floats. This decoder follows the verified layout of those two all-valid vector files.
Reaching eight useful bytes can require a larger storage read. A reader also needs metadata and may fetch or decode a larger block. This demonstration fetches the entire tiny file and measures no device I/O. Selecting a vector column is a projection; locating a row and decoding the page are additional steps.
An index chooses which distances will never be evaluated.
Exact search examined every eligible vector. Here is a second dataset built from the same twelve coordinates, now with a real Lance IVF_FLAT index. Its three supplied centroids are fixed so the partition assignments and results can be independently reproduced. This index is separate from the original two-fragment, unindexed byte fixture.
Index search coordinates
The query coordinates are shared with exact search above. This comparison searches all twelve rows; its exact reference is also unfiltered. Changing the point here updates the candidate arithmetic above.
Color = assigned partition. Square = centroid. Cross = query. Faded points are outside probed partitions. Rings mark exact top-three neighbors; a red ring means the index search missed that neighbor.
1. Rank centroids, then open their partitions
1. C0 at [1, 1]Squared query distance 1.578125 · probe
2. C1 at [5, 2]Squared query distance 8.328125 · leave closed
3. C2 at [9, 4]Squared query distance 53.828125 · leave closed
2. Gather candidate IDs
12345
5 of twelve vectors enter distance ranking. The centroid routes the query. A closed partition can still contain a closer neighbor.
3. Rank candidate distances
Exact top three: 4, 2, 5. Recovered 3/3 · recall@3 = 100.0%.
Why can the nearest centroid lead away from the nearest row?
A partition groups points by their nearest centroid. Its boundary can lie between the query and a very close point. The partition-boundary query (4.125, 1.25) probes C1 first, but its true nearest observation is in another partition. Probing a second centroid admits that missing candidate. Candidate generation made the approximation; the distances for admitted vectors remain exact Float32 squared L2 values.
IVF_FLAT stores unquantized vectors within partitions. Product quantization would introduce another approximation in vector representation; HNSW would add a graph traversal with its own search controls. This lesson implements neither by analogy. Every displayed setting is a real indexed search from Lance 11.0.0, checked against explicit partition membership and independent NumPy ranking.
Probing all three partitions recovers the exact answer here because every fixture vector is indexed and all distances are evaluated. Every candidate distance is evaluated in that exhaustive pass. Compare its selected IDs with the smaller candidate sets produced by fewer probes.
Select an indexed result above to follow its ID into the original byte fixture below. The vector values agree across the two datasets; fragment and row addresses belong to each dataset separately and are never copied from the index into the original manifest.
Download all indexed results and assignments · Rebuild and verify this index. Save both files together and run uv run reproduce-index.py --verify.
The search hands storage a list of IDs
Carry this answer into the file.
Exact and indexed search now use the same query coordinates. Select which answer to retrieve; the selected IDs drive the fragment and page accounting below.
Exact IDs: 4, 2, 5
From selected rows to selected storage
Three nearby vectors can share the work of one.
Search has selected IDs. Now suppose a reader needs their vector column. Map those IDs to the snapshot’s physical rows, group them by fragment, and identify the page buffers. This is the connection between a search answer and a columnar file layout.
IDs 4 · 8 useful value bytes · 58 bytes in referenced vector page buffers · 947 bytes in complete referenced files.
Fragment 0 · 1 selected rows
58 vector page-buffer bytes referenced
Zoom into vector value buffer: bytes 320–375
8-byte prefixID 18 bytesID 28 bytesID 38 bytesID 48 bytesID 58 bytesID 68 bytes
The separate two-byte page buffer is at offset 256. The 56-byte value buffer holds the prefix and all six coordinate pairs.
Fragment 1 · 0 selected rows
No selected rows: no vector page referenced
Zoom into vector value buffer: bytes 320–375
8-byte prefixID 78 bytesID 88 bytesID 98 bytesID 108 bytesID 118 bytesID 128 bytes
The separate two-byte page buffer is at offset 256. The 56-byte value buffer holds the prefix and all six coordinate pairs.
Projection, addressing, and decoding solve different problems.
Projection chooses vector rather than id or group data. Row addressing selects the fragments and positions. The page encoding determines how those positions become values. In this fixture the vector field maps directly to column 2; the manifest and column metadata supply that mapping.
One row and three rows in the same fragment reference the same 58 bytes of vector page buffers. IDs 6 and 7 cross a fragment boundary and reference 116 bytes. Grouping requests can avoid repeating page work, but these totals exclude metadata, padding and transport overhead. Follow the shared page references to see where grouping selected rows can save repeated decoding.
This tiny fixture has one page per column and no compressed vectors. Larger datasets introduce additional page, cache and request costs. What it does show is why column projection alone is insufficient: the reader still needs row locations and an encoding that lets it reach the requested values. Next, inspect the original bytes of the selected row.
Search selects IDs; storage supplies values.
A search result still needs its row values. The verifier separately reads selected row positions 0, 7 and 11 and confirms their complete source records. The SDK returns the same records as the source arrays.
An approximate vector index can narrow the candidate search and trade search effort against recall. The original byte fixture has no ANN index. The separate IVF_FLAT example above adds an index to the same coordinates and checks candidate sets and recall explicitly. Compare its candidate sets with the exact search to see which neighbors each probe reaches.
An application ID survives a rewritten address.
A separate native table starts with 1,024 vectors, then appends eight more. Follow an ID through deletion, compaction and index maintenance. Every displayed version is retained in the downloadable original dataset and reopened by the verifier.
Version 2 · 1,024 logical rows. 1,024 indexed · 0 unindexed · 2 active fragments.
Two fragments and 1,024 rows are covered by a real IVF_FLAT index with two supplied centroids.
Physical row counts include deleted slots before compaction. The indexed and unindexed counts above describe logical rows reported by the pinned SDK. Coverage says which logical rows the index includes; each query chooses the candidates it visits.
Follow an application ID
ID 6 in version 2: physical address 6 = fragment 0, row offset 6.
The address uses the high 32 bits for the fragment and low 32 bits for the row offset. Experimental stable row IDs are disabled here: native _rowid equals _rowaddr. Both are kept as exact decimal strings. The application’s id column is a different identity, and an old physical address is never reused to locate it in a newer version.
Native query point
Independent NumPy top three: 1023, 1022, 1021. All twenty recorded queries agree in every ID and distance. Each query scans all available partitions here, checking that lifecycle changes preserve the exact answer.
Executed operation and original native query plan
write_dataset(first_1024, max_rows_per_file=512, data_storage_version='2.1', enable_stable_row_ids=False); create_index('vector', 'IVF_FLAT', num_partitions=2, ivf_centroids=[[256, 0], [768, 0]])ProjectionExec: expr=[id@2 as id, vector@3 as vector, _distance@0 as _distance, _rowid@1 as _rowid, _rowaddr@4 as _rowaddr]
LanceRead: uri=<dataset>/data, projection=[id, vector, _rowaddr], source=stream(_rowid)
SortExec: TopK(fetch=3), expr=[_distance@0 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
ANNSubIndex: name=vector_idx, k=3, deltas=1, metric=L2
ANNIvfPartition: uuid=68085e33-f445-4e77-974b-9efe98d1b993, minimum_nprobes=2, maximum_nprobes=Some(2), deltas=1
The append and delete plans combine ANN and scan branches. After full index maintenance, the unindexed scan branch is absent. Plan names describe this pinned SDK.
Native metadata, coverage and complete-row verification
Original version 2 manifest
{
"fragments": [
{
"id": 0,
"files": [
{
"fields": [
0,
1
],
"column_indices": [
0,
1
],
"file_major_version": 2,
"file_minor_version": 1,
"file_size_bytes": 5909,
"base_id": null,
"path": "100111111011111100001000eac3224a238ee79de55c10402f.lance"
}
],
"overlays": [],
"physical_rows": 512,
"deletion_file": null,
"row_id_meta": null,
"created_at_version_meta": null,
"last_updated_at_version_meta": null
},
{
"id": 1,
"files": [
{
"fields": [
0,
1
],
"column_indices": [
0,
1
],
"file_major_version": 2,
"file_minor_version": 1,
"file_size_bytes": 6037,
"base_id": null,
"path": "1010000100000111011111002db1974bc8844d05003a576e7e.lance"
}
],
"overlays": [],
"physical_rows": 512,
"deletion_file": null,
"row_id_meta": null,
"created_at_version_meta": null,
"last_updated_at_version_meta": null
}
],
"indices": [
{
"name": "vector_idx",
"type": "IVF_FLAT",
"uuid": "68085e33-f445-4e77-974b-9efe98d1b993",
"fields": [
"vector"
],
"version": 1,
"fragment_ids": [
0,
1
],
"base_id": null
}
],
"statistics": {
"index_type": "IVF_FLAT",
"name": "vector_idx",
"num_indices": 1,
"num_segments": 1,
"indices": [
{
"index_type": "IVF_FLAT",
"uuid": "68085e33-f445-4e77-974b-9efe98d1b993",
"uri": "<dataset>/_indices/68085e33-f445-4e77-974b-9efe98d1b993/index.idx",
"metric_type": "l2",
"num_partitions": 2,
"sub_index": {
"dim": 2,
"index_type": "FLAT",
"metric_type": "l2"
},
"partitions": [
{
"size": 513
},
{
"size": 511
}
],
"centroids": [
[
256,
0
],
[
768,
0
]
],
"loss": 22369792,
"index_file_version": "V3"
}
],
"segments": [
{
"index_type": "IVF_FLAT",
"uuid": "68085e33-f445-4e77-974b-9efe98d1b993",
"uri": "<dataset>/_indices/68085e33-f445-4e77-974b-9efe98d1b993/index.idx",
"metric_type": "l2",
"num_partitions": 2,
"sub_index": {
"dim": 2,
"index_type": "FLAT",
"metric_type": "l2"
},
"partitions": [
{
"size": 513
},
{
"size": 511
}
],
"centroids": [
[
256,
0
],
[
768,
0
]
],
"loss": 22369792,
"index_file_version": "V3"
}
],
"num_indexed_fragments": 2,
"num_indexed_rows": 1024,
"num_unindexed_fragments": 0,
"num_unindexed_rows": 0,
"num_indexed_rows_per_delta": [
1024
],
"updated_at_timestamp_ms": 1789256707526
}
} Every logical row matches the independently constructed IDs and vectors. Every native physical address is checked against the corresponding original fragment row. Canonical full-row SHA-256 for this version: a0fcd3667366e690f41581c01bcd637a4fb9365a0480a85c99efa636a8972d07.
All five captures and twenty queries · Original versioned dataset · Reproduce and verify the lifecycle · Reproduction instructions
pylance 11.0.0. These vectors are synthetic two-dimensional coordinates. The index, manifest and deletion metadata come from the pinned native captures. No retained files were vacuumed.
The chosen row crosses two different page boundaries.
These two native tables contain the same 2,048 IDs and variable-length strings, including 293 nulls. Each ID column has 15 pages; each string column has 16. Small writer batches produce real page boundaries, and a second layout compresses the string values with Zstd.
String compression
Selected native result ID
Native predicate id = 1000 returns ID 1000 in this table’s version 2: fragment 0, row offset 1000. These addresses belong to this physical example, separately from the vector datasets above.
id · field ID 0
01234567891011121314
Page 6 covers rows 896–1023. Its 1 buffer occupies 1,024 bytes.
label · field ID 1
0123456789101112131415
Page 7 covers rows 896–1023. Its 2 buffers total 15,096 bytes.
For row 255, the ID is still on page 0 while the string is on page 1. The original protobuf metadata records each page’s first row and logical length; matching page numbers across columns would select the wrong range.
Decode the ID from its original bytes
The browser reads the 40-byte footer, follows the column metadata offset, decodes the embedded protobuf page layout, then locates the selected row within an all-valid FullZip flat INT64 buffer. It accepts only this bounded encoding and returns the value read from those eight little-endian bytes.
The nullable string has another decoding path
Native SDK result, independently checked against the source: row-1000-abcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdef.
String page 7 uses MiniBlock layout with nullable-item definition information and variable-length values. Its values are stored without a general compression layer. The native verifier reads every string through the SDK and compares it with the independently constructed source.
Original page metadata and encoding layers
{
"id": {
"page": 6,
"firstRow": 896,
"rows": 128,
"buffers": [
{
"position": 113920,
"size": 1024
}
],
"layout": {
"full_zip_layout": {
"bits_per_value": 64,
"num_items": 128,
"num_visible_items": 128,
"value_compression": {
"flat": {
"bits_per_value": "64"
}
},
"layers": [
"REPDEF_ALL_VALID_ITEM"
]
}
}
},
"label": {
"page": 7,
"firstRow": 896,
"rows": 128,
"buffers": [
{
"position": 114944,
"size": 8
},
{
"position": 115008,
"size": 15088
}
],
"layout": {
"mini_block_layout": {
"def_compression": {
"inline_bitpacking": {
"uncompressed_bits_per_value": "16"
}
},
"value_compression": {
"variable": {
"offsets": {
"flat": {
"bits_per_value": "32"
}
}
}
},
"layers": [
"REPDEF_NULLABLE_ITEM"
],
"num_buffers": "1",
"num_items": "128"
}
}
}
}Parsed using schemas from Lance commit ab6b5bbe46009ed78746b444df8db59a8bc5d842, checked against SDK buffer locations. The native verifier also decodes all 4,096 ID values directly from the two original files.
The complete files are 263,743 bytes without string compression and 43,502 bytes with Zstd. They have identical logical values. The repeated strings let you compare how the two encodings occupy these actual files.
Both layouts and sixteen native selections · Original physical tables · Physical reproduction script · Instructions and required protobuf files · All files and hashes
Keep the layers distinct.
Lance's file container organizes columns and pages; its table manifest manages fragments and versions; optional search indices add another structure. A hosted LanceDB service is a separate product. Neither this walkthrough nor its downloadable fixture needs one.
The Python SDK uses the Lance Rust core. Comparing its two interfaces would not establish independent decoding. Here, version-matched protobuf decoding and bounded-range checks substantiate layout; original source values check row decoding; NumPy independently checks exact neighbors.
Your turn.
Download the Lance artefacts and follow another row address. Compare their page layout with a Parquet file in the local explorer.