Encoding Package

The encoding package provides semantic type-aware array encoding and quantization.

Encoding package for semantic types and array encoding/decoding.

This package handles: - Semantic type definitions (COORDINATE, COLOR, POSITIVE_SCALAR, etc.) - Encoding modes (AUTO, PRECISION, MEMORY, CUSTOM) - Array encoding/decoding with multiple strategies - Array reference registry for deduplication - Encoding metadata specification

ArrayEncoder

class luxar.encoding.ArrayEncoder(broadcast_rtol: float = 0.0, broadcast_atol: float = 0.0, float16_allowed: bool = False, lut_json_max_bytes: int = 524288)

Bases: StructuralEncoderMixin, PerChannelEncoderMixin, CholeskyEncoderMixin

Unified encoder with internal registry for deduplication.

The encoder writes directly to zarr groups and maintains an internal registry for detecting duplicate arrays. It follows the priority order specified in the encoding specification.

__init__(broadcast_rtol: float = 0.0, broadcast_atol: float = 0.0, float16_allowed: bool = False, lut_json_max_bytes: int = 524288) → None

Initialize encoder with optional broadcasting tolerance and float16 control.

Parameters:

• broadcast_rtol – Relative tolerance for broadcasting check (default: 0.0 = exact equality)

• broadcast_atol – Absolute tolerance for broadcasting check (default: 0.0 = exact equality)

• float16_allowed – Allow float16 encoding in MEMORY mode (default: False for TypeScript compatibility)

• lut_json_max_bytes – Metadata-health cap for the uint16 LUT tier: the LUT values live as JSON in .zattrs AND are duplicated by consolidated .zmetadata, which the viewer parses at scene-open for ALL nodes — one greedy LUT would tax every load. The estimated doubled JSON size must stay under this cap (default 512 KiB ≈ 4-4.5K unique float RGB rows, since the estimate doubles the raw JSON for .zmetadata). The uint8 tier is unaffected (≤256 values is at most ~22 KiB).

Note: Using non-zero tolerance is experimental and should be used with caution. Exact equality (default) is safe for all semantic types including indices and colors.

encode(data: ndarray | float | int | number | tuple | list, zarr_group: Group, name: str, semantic_type: SemanticType, mode: EncodingMode = EncodingMode.AUTO, n_elements: int | None = None, bounds: tuple[float, float] | None = None, positive_scalar_encoding: Literal['linear', 'log'] = 'linear', custom_encoder: str | None = None, color_mode: Literal['sdr', 'hdr'] | None = None, chunks: tuple | None = None, compressor: Any | None = None, deduplicate: bool = True, allow_lut: bool = True, positive_scalar_bits: Literal[8, 16] | None = None, _perchannel_bits: int | None = None) → None

Encode array or scalar and write to zarr group.

Follows priority order: 1. Broadcasting (if scalar input OR all values identical within tolerance) 2. Array reference (if duplicate exists) 3. LUT encoding (≤65536 unique values, tiered uint8/uint16 indices, mode != PRECISION) 4. Dtype encoding (based on semantic type and mode)

Parameters:

• data – Input data - can be: - NumPy array: standard path - Python scalar (float, int): requires n_elements - Tuple/list (for colors): e.g., (1.0, 0.0, 0.0)

• zarr_group – Zarr group to write to

• name – Array name within the group

• semantic_type – Semantic type (REQUIRED - must be explicit)

• mode – Encoding mode (AUTO, PRECISION, MEMORY, CUSTOM)

• n_elements –

  Broadcast target element count.

  • Required if data is scalar/tuple/list because the scalar value is stored once and represents this many elements.

  • Optional for array input, but when provided it opts into broadcast/uniform validation: arrays must have shape (1, …) or be a full-length uniform array. Omit n_elements for non-uniform full arrays.

• bounds – Min/max bounds for BOUNDED_SCALAR (None = auto-detect)

• positive_scalar_encoding – “linear” or “log” for POSITIVE_SCALAR

• custom_encoder – Explicit encoder name for CUSTOM mode

• color_mode – “sdr” or “hdr” for COLOR semantic type (required for float)

• chunks – Optional chunk shape for zarr dataset

• compressor – Optional compressor for zarr dataset

• deduplicate – When True (default), a byte-identical array already written elsewhere is stored as a lightweight array_ref. Pass False for arrays whose reader cannot resolve refs (e.g. line vertices/segments, read as raw chunked zarr) so they are always materialised.

• allow_lut – When True (default), arrays with few unique values may store as an exact lut_uint8/16. Pass False for arrays whose reader has no encoding dispatch (line vertices — same raw-read rationale as deduplicate); grid-snapped coordinates would otherwise LUT-encode and decode as indices.

• positive_scalar_bits – Optional AUTO quantization tier for POSITIVE_SCALAR encoding. None preserves AUTO’s existing selection; 8 selects geometric-log uint8 regardless of the positive_scalar_encoding linear/log choice. MEMORY remains 8-bit and PRECISION remains float32.

• _perchannel_bits – Internal-only. Forces the quantization tier (8 or 16) of the per-channel log encoders for the CHOLESKY_DIAG / CHOLESKY_OFFDIAG semantic types. Set only by encode_cholesky_split(), which owns the certified u8→u16 escalation for the diag/offdiag pair; None (all other callers) defaults to 8.

Raises:

• ValueError – If scalar input lacks n_elements

• ValueError – If n_elements provided but data has different length

• ValueError – If semantic type constraints are violated

• ValueError – If NaN or Inf values are present

• ValueError – If CUSTOM mode lacks custom_encoder

• ValueError – If COLOR with float dtype lacks color_mode

encodes_as_lut(data: ndarray, semantic_type: SemanticType) → bool

Would encode() store data as an (exact) LUT encoding?

LUT is priority 3 and is tried BEFORE the dtype/semantic encoder, so for an eligible array the lossy per-axis quantization the dtype encoder would apply never happens: a LUT stores the original values verbatim and its index array is usually smaller than the quantized alternative.

This is public because a caller outside the encoder needs to know every EXACT path the encoder has before deciding to override it. The gsplat writer’s sigma rail (_resolve_centers_encoding_mode()) escalates centers to PRECISION when uint16 quantization would displace splats past their own σ — but PRECISION also disables LUT, so escalating a LUT-eligible array would replace an exact 1 B/value encoding with an exact 4 B/value one and warn about damage that was never going to happen. The grid snap (gridded_axis_step()) is the rail’s other such check; this is the second, and they must both be asked.

This answers the LUT question ALONE, and encode() gates LUT on more than that, so a True here is necessary but not sufficient for the array actually being stored as a LUT. encode skips LUT under PRECISION and under an explicit allow_lut=False, and two higher-priority paths pre-empt it even when it is eligible: priority 1 (a uniform array, stored as a broadcast constant — skipped for COORDINATE, which is never broadcast) and priority 2 (a content-identical array already written, stored as an array_ref, skipped under deduplicate=False). Both of those are exact too, so for the gsplat sigma rail — which asks only in order NOT to escalate an already-exact array, passes COORDINATE, and writes escalated centers with deduplicate=False — neither can change the answer’s usefulness. A caller wanting “will this be a LUT on disk” must account for them itself.

The second in-repo caller, coordinate_round_trip_slack(), does exactly that for the one gate that DOES change its answer: it takes an allow_lut mirroring encode()’s and does not ask this at all when the write forbids a LUT. It has to — the lines writer encodes vertices with allow_lut=False, so a LUT-eligible lines vertices array is quantized on disk, and treating “eligible” as “exact” there would write chunk bounds the decoded vertices fall outside of (issue #1655).

Costs one np.unique pass over data, and the plan is NOT memoized: encode recomputes it from scratch, so asking and then encoding is two passes (measured 0.77 s each on a 1.65M×4 centers array).

reset() → None

Clear registry (call between independent scenes).

snapshot() → tuple[dict[tuple, tuple[str, str]], dict[str, str]]

Capture deduplication state for a transactional writer operation.

restore(state: tuple[dict[tuple, tuple[str, str]], dict[str, str]]) → None

Restore deduplication state after a transactional writer rollback.

Semantic Types

class luxar.encoding.SemanticType(*values)

Semantic types for array data.

Each semantic type has specific constraints and valid encodings:

• COORDINATE: Spatial positions/centers/vertices. Can be negative. PRECISION → float32; AUTO / MEMORY → uint16 per-axis fixed-point via the generic per-channel linear quantization (linear_perchannel_u16), decoded back to float32 — visually lossless (sub-unit) and ~2× smaller. Coordinates never use uint8 (too coarse) and never float16 (relative precision degrades with magnitude). A per-axis extent ≥ 2¹⁶ auto-falls back to float32.

• COLOR: RGB/RGBA color values. Non-negative. SDR (0-1): uint8, uint16, float16, float32 HDR (>1): geolog_perchannel_u8/u16 (per-channel true-log; AUTO u16, MEMORY u8), float32 (PRECISION)

• BOUNDED_SCALAR: Scalars with known [min, max] bounds. Examples: sharpness [0, 1], opacity [0, 1] Valid: uint8, uint16, float16, float32

• POSITIVE_SCALAR: Non-negative scalars, potentially wide dynamic range. Examples: radii, amplitudes, distances Valid: uint8/uint16 linear (bounded_scalar_*) for narrow ranges; geolog_scalar_uint8/uint16 (min/max-anchored geometric-log, reserved zero level) for wide ranges — no nonzero value can quantize to zero and relative precision is uniform; float32 under PRECISION.

• CHOLESKY: Packed lower-triangular Cholesky factors (whole, unsplit). Shape: (N, d*(d+1)/2) for d-dimensional covariance Valid: float32, float16

• CHOLESKY_DIAG: Diagonal of a packed Cholesky factor (positive, scale-like), shape (N, d). Encoded float32, or with the generic per-channel log quantization (log_perchannel_u8; log_perchannel_u16 when AUTO’s covariance certificate escalates — see encode_cholesky_split).

• CHOLESKY_OFFDIAG: Strictly-lower off-diagonal of a Cholesky factor (signed, ~zero-centred), shape (N, d*(d-1)/2). Encoded float32, or with the generic per-channel signed-log quantization (signed_log_perchannel_u8, escalating to signed_log_perchannel_u16 alongside the diagonal — both halves always share one tier).

• INDEX: Non-negative integer indices or counts. Valid: uint8, uint16, uint32, uint64

COORDINATE = 'coordinate'

COLOR = 'color'

BOUNDED_SCALAR = 'bounded_scalar'

POSITIVE_SCALAR = 'positive_scalar'

CHOLESKY = 'cholesky'

CHOLESKY_DIAG = 'cholesky_diag'

CHOLESKY_OFFDIAG = 'cholesky_offdiag'

INDEX = 'index'

Encoding Modes

class luxar.encoding.EncodingMode(*values)

Encoding preference modes.

Modes control the trade-off between precision and storage efficiency:

• AUTO: Automatically analyze data and select appropriate encoding. Balanced approach that considers data range and semantic type.

• PRECISION: Preserve maximum precision by using float32 for all non-index data. Recommended for scientific accuracy.

• MEMORY: Minimize storage size aggressively using quantization (uint8, uint16, float16) where safe. Recommended for large datasets and streaming. May introduce quantization error.

• CUSTOM: User explicitly specifies which encoder to use for each array. Provides full control over encoding choices. Requires custom_encoder parameter in encode() calls.

Note: Broadcasting and Array Reference optimizations apply in ALL modes as they are lossless. Mode only affects dtype/quantization encoding.

AUTO = 'auto'

PRECISION = 'precision'

MEMORY = 'memory'

CUSTOM = 'custom'

Coordinate Grid Snap

The shared predicate behind the COORDINATE grid snap, and the level count it is asked about. Both are public because the gsplat writer’s sigma rail must ask the encoder the same question it asks itself.

luxar.encoding.gridded_axis_step(col: ndarray, lo: float, extent: float, levels: float) → tuple[float, int] | None

The spacing of the regular grid col lies on, or None.

Returns (step, n_distinct). The test IS the guarantee: a candidate spacing is accepted only after replaying the encoder’s own quantization and the decoder’s own dequantization over the distinct values and getting all of them back bit-exactly at the COORDINATE decode dtype (float32). That is both stricter and more permissive than testing the gaps for equality, and both directions matter:

• A grid with MISSING rungs still qualifies. Frames 0,1,2,7,8,9 are what a spatial tile of a stacked dataset sees, or what a filter that empties one timepoint in one region leaves behind; an “all gaps equal” test rejects that axis and leaves it broken by exactly the defect the snap exists to prevent.

• A grid the float32 input only approximates still qualifies, because the spacing is least-squares refit against every rung rather than taken from the smallest gap. A 0.1 s frame interval jitters by more than a relative 1e-6 once rounded to float32, so a gap-equality test with any usable tolerance rejects it.

• Conversely, continuous values whose smallest gap merely happens to be coarse do NOT qualify, because they do not survive the replay.

The only cap on distinct values is levels itself: an axis with more distinct values than the encoding has levels cannot be represented on any grid, so there is nothing to snap to. Capping lower would reject stacks that fit perfectly – a 10 000-frame timelapse quantizes exactly at u16 – and would save no work, since np.unique has already run by then.

Module-level rather than a private encoder method because it is a SHARED predicate: the gsplat writer’s sigma rail (_center_quantization_offender()) must know whether this encoder will store an axis exactly before deciding to escalate it to float32, and the two must never answer differently. Call it with exactly the lo/extent/levels the encoder will use.

A caller that needs the distinct values for something ELSE as well — PerChannelEncoderMixin.coordinate_round_trip_slack() needs the COUNT to short-circuit its LUT probe — goes through _gridded_step_from_uniques() with the np.unique it already ran, rather than paying for a second pass. The split is deliberately a PRIVATE sibling sharing one body: this public entry point keeps its exact signature, so the shared contract cannot acquire a “and pass the right uniques” footgun that an out-of-module caller could get wrong.

luxar.encoding.COORDINATE_LEVELS = 65535.0

Convert a string or number to a floating-point number, if possible.