A small node's chunks from ONE request: the luxar optimize --pack reader.
A packed store keeps every plain chunk object and adds, per small geometry
node, one pack file holding copies of that node's chunks (luxar.io.chunk_pack):
a 4-byte little-endian header length, a JSON header {"members": {chunk key relative to the node: [offset, length]}}, then the bytes. The root
chunk_packs sidecar's attrs list each pack's key, SHA-256 and node prefix.
Thousands of ~1 KB rungs cost round trips, not bytes, so a laddered
timelapse part read as one pack instead of ~16 chunks plays measurably faster.
This source sits UNDER the caching store, in the zip store's seam: it answers
each chunk key with that chunk's bytes, so L0/L1/L2 cache exactly the keys
and bytes a plain read would have produced, and no loader or decoder knows
packs exist. Every doubt lands on the plain read, never on wrong bytes: an
index built for another content_hash is not adopted, a pack that fails its
SHA-256 or cannot be fetched is dropped, and a key the pack does not hold
(an all-fill chunk zarr never wrote, a metadata document) is read plainly.
A fetched pack's members are handed out once each (the caches hold them
from then on) and its unserved ones are held, least recently used first,
within PackedChunkSource.maxHeldBytes: a rung the view never loads
must not pin bytes outside every cache budget. A member read again after
the caches evicted it, or a member of a pack let go, fetches the pack again —
one request, as a plain read would cost.
A small node's chunks from ONE request: the
luxar optimize --packreader.A packed store keeps every plain chunk object and adds, per small geometry node, one pack file holding copies of that node's chunks (
luxar.io.chunk_pack): a 4-byte little-endian header length, a JSON header{"members": {chunk key relative to the node: [offset, length]}}, then the bytes. The rootchunk_packssidecar's attrs list each pack's key, SHA-256 and node prefix. Thousands of ~1 KB rungs cost round trips, not bytes, so a laddered timelapse part read as one pack instead of ~16 chunks plays measurably faster.This source sits UNDER the caching store, in the zip store's seam: it answers each chunk key with that chunk's bytes, so L0/L1/L2 cache exactly the keys and bytes a plain read would have produced, and no loader or decoder knows packs exist. Every doubt lands on the plain read, never on wrong bytes: an index built for another
content_hashis not adopted, a pack that fails its SHA-256 or cannot be fetched is dropped, and a key the pack does not hold (an all-fill chunk zarr never wrote, a metadata document) is read plainly.A fetched pack's members are handed out once each (the caches hold them from then on) and its unserved ones are held, least recently used first, within PackedChunkSource.maxHeldBytes: a rung the view never loads must not pin bytes outside every cache budget. A member read again after the caches evicted it, or a member of a pack let go, fetches the pack again — one request, as a plain read would cost.