File Format

A Chassis file is one file in format version 3 (ADR-0008), or version 4 if it keeps its vectors in half precision (ADR-0018). It holds two header copies, then append-only regions: segments of slots, heap chunks of upper-layer neighbor lists, and table pages that list where the others start. Each region is mapped once, and committed bytes never move.

Opening a version 1 or 2 file migrates it to version 3 (see Migration).

Top-Level Layout

The header copies sit 64 KiB apart so they never share a page. Every region starts at a multiple of 64 KiB, the Windows mapping granularity and a multiple of every supported page size, so each one can be mapped on its own. The file only grows by appending a region, and never shrinks.

Header

Each copy is a header of 144 + 8 × (segment table pages + heap table pages) bytes, little-endian. A later write version may add fields after these; the length covers them. The valid copy with the higher sequence number is current.

A release checks the read version before the checksum, so a newer layout is reported as too new, not as corrupt. A newer write version only stops writers: IndexReader reads the file and ignores the bytes after the next id. Releases up to 0.6.3 read bytes 8–11 as their version and refuse anything above 2.

A file in half precision is version 4 to readers and writers alike, so that a release from before half precision refuses it and doesn’t read its vectors as f32s. A file in full precision is written as version 3 by every release, with byte 52 zero as it always was.

Live Page

What the writer has added since its last commit, so readers in other processes can route through it. Every field is a little-endian u64. The counts and the routing count are written with release ordering after the data, offsets and table entries they cover. It is not durable: a writer resets it to its committed state when it opens the file, and a reader that opens the file before any writer, after a crash, uses it as found, treating records it can’t read past the committed count as having no neighbors.

Segments

Segment k holds 2^(base + min(k, K)) slots, where base is the segment base log2 and K the number of doubling segments. A new file uses base = 10 and the largest K whose segment fits in 256 MiB, so every segment after the first K has the same C = 2^(base + K) slots. With B = C − 2^base, the slots in all doubling segments together:

• slot s < B is in segment ⌊log2(s + 2^base)⌋ − base, at index s + 2^base − 2^⌊log2(s + 2^base)⌋;
• slot s ≥ B is in segment K + ((s − B) >> log2 C), at index (s − B) & (C − 1).

A segment of n slots holds three arrays, each starting at a multiple of 64 bytes:

A record’s head holds the node’s layer count in bits 0–7 and, for a node above layer 0, the heap position of its upper-layer lists in bits 8–63. Empty neighbor entries are 0xFFFFFFFF. A slot is deleted once its deleted epoch is nonzero; only a flush writes it.

Upper Heap

Heap chunk c is 2^(heap base + min(c, doubling chunks)) bytes; a new file uses 64 KiB chunks that double up to 256 MiB. A node on L layers has one heap entry of L − 1 lists of M neighbor slots (u32), for layers 1 to L − 1. An entry never straddles two chunks.

Table Pages

A table page holds 2^(table page log2) file offsets (u64), 8,192 in a new file, so a page is 64 KiB. Entry i of the segment table is in page i >> log2, at index i & (2^log2 − 1); the heap table works the same way. The header lists up to 512 pages per table, enough for 2^32 slots.

Commits and Recovery

A flush writes data in place, fsyncs, then writes the header copy that does not hold the newest header and fsyncs again. Slots, regions and table entries past the committed counts are leftovers of a flush that never committed: the next writer reuses them, and until then readers may route through them but never return them.

A flush with deletes commits through an intent header first: the previous commit with pending epoch = epoch + 1, then the delete marks, then the commit header in the other copy, with an fsync after each step. Opening a file whose newest header has a pending epoch clears every deleted epoch above the header’s epoch in committed slots, then writes a clean header. Process kills are tested in chassis-core/tests/crash_tests.rs; power loss is simulated in chassis-core/src/power_loss.rs, including crashes inside recovery.

Size Example

10,000 vectors with 768 dimensions and default parameters take 3,232 bytes per slot (24-byte slot header, 3,072-byte vector, 136-byte level-0 record) plus about 4 bytes of upper heap. The first four segments hold 1,024, 2,048, 4,096 and 8,192 slots, so the file is about 50 MB, of which 32 MB is written. The unused end of the last segment is never written, but whether the file system keeps it sparse varies: on APFS, a real build of this example allocated 33.6 MB, while unused ranges of 16 MiB or less next to written data were sometimes allocated in full. ext4 is untested.

The undo file

Adds change the neighbor lists of committed nodes in place before they are committed themselves. Before a committed node’s lists first change after a flush, they are appended to <name>.undo, and a writer that opens the index after a crash writes them back (ADR-0012). The file exists only between a flush and the next one that follows adds, or after a crash; a flush removes it. An index moved or copied while the file exists has to take it along.

After the 8 bytes CHSUNDO1, each entry is the slot (u32), its layer count (u8), then per layer a count (u16) and that many neighbor ids (u32), then an xxh3-64 of the entry, seeded with a hash of the committed vector count, entry point, top layer, delete epoch, deleted count and id mark. Entries saved from another commit fail the checksum and are ignored.

Migration

Opening a version 1 or 2 file converts it. Chassis takes the original’s lock, writes the committed state into <name>.migrating: slots below the graph’s node count, and only the delete marks the graph header committed. It converts neighbor ids from u64 to u32 and moves upper layers into the heap, then fsyncs the new file and renames it over the original, and fsyncs the directory. The graph is converted, not rebuilt, so search results are unchanged. A failed or interrupted migration leaves the original in place.

Validation

On open, Chassis checks that:

• one header copy has the magic, versions it understands and a valid checksum; two valid copies with the same sequence number are corruption;
• the dimensions match, and the graph parameters and geometry are possible;
• the counts fit the tables, and every committed region lies inside the file;
• VectorIndex::open only: the graph parameters match the requested max_connections, and a file with vectors has a graph.

A failed check returns an error and never modifies the file.

Stability

Format version 3 is the format ADR-0008 proposes to freeze. Until that ADR is accepted, it may still change.