One slot. Several buffers.

A record batch gathers rows from columns.

This batch has 6 rows and 6 columns. Every array has the same logical length, so slot 0 identifies the same record across all columns. Its city, ID and readings live in different buffers. A row emerges when the reader gathers those matching positions.

Selected original slot: 0. Follow it through the buffer diagrams below.

The schema supplies the interpretation: an integer width, a string layout, a list child type. A batch combines that schema with equally long arrays. Engines can pass a batch between operators while each operator chooses which columns it needs.

A null keeps its slot.

The city bitmap contains 3d. Bit 0 is one, so this slot is valid. Start at the least significant bit of the first byte and move right through logical slots. Packed bits let the reader check missingness separately from the value bytes.

Compare the missing city with the empty city from the opening. Both occupy a logical position. Only the missing city has a cleared validity bit. Counting the array's positions therefore includes nulls; counting valid values checks the bitmap.

An all-valid array can omit this buffer. The ID array does exactly that: every slot has an integer. Nested arrays carry validity at each level. A valid readings list can contain a null child, because the parent and child answer different questions.

Fixed width gives every value an address.

The ID type is int64. Each physical slot reserves eight bytes, so its address is the values-buffer start plus the slot number times eight. The selected logical slot maps to physical slot 0. Read its bytes in little-endian order.

The decoded integer is 9007199254740993. The first ID, 9007199254740993, exceeds JavaScript's exact Number integer range. This guide reads the signed buffer with bigint, preserving every bit before producing display text.

Direct addressing is the payoff of fixed width. A reader can locate a later slot by multiplication, without walking earlier values. Validity still comes first for a nullable fixed-width array: the reserved bytes belong to a slot even when its value is missing.

A string occupies the space between two offsets.

Variable-width strings need a second address calculation. Each slot has a start offset; the next slot's start also serves as its end. The final offset closes the final string, so the offsets buffer contains one more entry than the physical array has slots.

Slot 0: "İzmir"

Subtract neighboring offsets to get the byte length. The first city, "İzmir", occupies 6 UTF-8 bytes. The offsets advance over encoded bytes, while the decoded string has its own character count.

Try the null and empty slots. Both use equal neighboring offsets in this fixture. The bitmap resolves the ambiguity: one value is absent, the other is present with no characters. A reader that used only the difference between offsets would collapse those distinct values.

These int32 offsets can be read directly from the original little-endian buffer. The values buffer simply holds adjoining string bytes. String boundaries come from offsets, so delimiters and terminators are unnecessary.

A list points into another array.

The readings column uses the same start-and-end idea, but its offsets count child slots. A list's contents therefore inherit the child's type and validity. The parent first decides whether a list exists; its offsets then choose the child positions to gather.

Selected list: [4, null, 7]. Its offsets span child slots 0 through 3, excluding the end.

The first list is [4, null, 7]. Its parent is valid and its middle child is null. The next parent is null; the following parent is valid but empty. Each case needs both the parent bitmap and the child layout to preserve its meaning.

This separation lets a child itself be another nested array. A list of structs, for example, uses parent offsets to select struct slots, then follows the struct's children. Each level contributes its own addressing and validity rules.

A small index reaches a shared value.

The category array separates an index from its string dictionary. An index chooses a dictionary slot, then that string uses its own offsets and bytes. Repeated categories can share a dictionary value instead of repeating its string bytes in every row.

The selected category is "coast". Its own validity bitmap decides whether the index is meaningful. A null index stays null, even when the corresponding byte happens to be zero and dictionary entry zero contains a string.

A Parquet dictionary page participates in a storage encoding: pages store encoded dictionary indices for a column chunk. An Arrow dictionary array is already an in-memory typed array. Decoding Parquet can produce dictionary arrays or expand values, depending on the reader and requested representation.

A slice changes the start, keeping the buffers.

Switch the array view to the slice. Its first slot addresses original slot 1; its logical length is 3. The selected view currently has offset 0. Every bitmap and offset lookup adds that array offset before reading a position.

The native verifier checks that the slice shares its original values buffer. Creating that view changes offset and length metadata, so it can avoid copying string contents. Those offset values still address the same original byte buffer.

Keeping a tiny slice can therefore retain a large backing allocation. Copying a selected range may be useful when releasing that larger buffer matters. The choice depends on ownership and lifetime as well as the cost of copying.

The type supplies the decimal point and the clock.

The amount array has type decimal128(24, 2). Its bytes hold signed integers; scale supplies the decimal point. To display a value, the reader preserves the integer's full width and then applies the scale. Floating-point conversion would lose precision before formatting began.

Selected amount: 12345678901234567890.12. Selected moment: 1700000000123456 microseconds since epoch.

The moment type, timestamp[us, tz=UTC], records a time unit and timezone. Its integer counts microseconds from the epoch. Displaying a calendar time requires that type information; the integer bytes alone cannot tell a reader which time unit to use.

Arrow carries both the bytes and their schema. Preserve both when handing a batch to another process. The SQL workbench also keeps exact database text beside typed inspection, so a displayed number can be checked against its original representation.

IPC puts the arrays into messages.

The IPC downloads serialize the same logical records. A schema message describes types; a dictionary batch supplies shared dictionary values; record-batch messages describe array buffers and carry their bodies. Readers rebuild typed arrays from that sequence.

The file footer lets a reader locate record batches without replaying the stream from its beginning. The stream works for sequential transfer, including results arriving from a running producer. Both formats preserve the schema and the dictionary information their arrays need.

Open either download in the IPC inspector to move between its batches and inspect their buffers. Then compare the matching Parquet records: the logical values agree while the storage representation changes.

Parquet stores. Arrow supplies arrays.

Use the distinction to trace an actual query. A scan locates Parquet pages, decompresses them and decodes their columns. The resulting arrays can feed filters and joins. Returning an Arrow batch preserves those typed columns at the next boundary.

These roles explain why a compact Parquet file can expand in memory and why an Arrow slice can be cheap. Storage size and the cost of reaching a value answer different questions. Start from the operation you need, then follow the representation it touches.

Inspect the original buffers.

Native array metadata and bytes · Pinned reproduction script · Apache Arrow format specification.

Try the empty readings list and then the null readings list. Both span zero child slots. Which buffer changes their meaning?