Gaussian Splats Dimension Mapping

Version: 2.0 Last Updated: 2026-08-26

Overview

Gaussian splats declare their intrinsic dimensionality with explicit mapping to scene dimensions. This allows lower-dimensional splats (e.g., 3D) to be added to higher-dimensional scenes (e.g., 5D) without padding — splats store only the dimensions they actually span, and declare which remaining dimensions they should be visible across.

Design Principles

  1. Explicit over implicit: Users must specify dimension mapping when splat_ndim != scene_ndim

  2. Automatic only when safe: Auto-map when splat_ndim == scene_ndim (unambiguous)

  3. Mathematically consistent: Covariance must match splat dimensions exactly

  4. Mutually exclusive: Dimensions are either spanned (position + covariance) or extended (always visible)

API

Parameters

The add_gsplats() method on Scene and Group accepts these dimension mapping parameters:

def add_gsplats(
    self,
    name: str,
    centers: np.ndarray,           # Shape (N, D_splat)
    amplitudes: Union[float, np.ndarray],
    cholesky_factors: np.ndarray,  # Shape (N, k) where k = D_splat*(D_splat+1)/2
    colors: Optional[np.ndarray] = None,
    labels: Optional[Sequence[str]] = None,        # Per-splat hover labels (CSR text)
    image_labels: Optional[Any] = None,            # Per-splat hover thumbnails (CSR images)
    parent: Optional[Node] = None,
    extend_to_all: Optional[Union[List[str], str]] = None,
    dim_order: Optional[List[str]] = None,
    fill: Optional[Dict[str, float]] = None,
    fill_sigma: Optional[Dict[str, float]] = None,
    partition: Any = None,                         # True / dict(max_elements=..., rule=...) →
                                                   #   compile-time kind=partition wrapper
    **attrs: Any,
) -> Union[GSplats, Group]:

cholesky_factors packs the lower-triangular factor L of the covariance (Σ = L·Lᵀ) in row-major order. Its diagonal is scale-like: isotropic std σ in 3D uses [σ, 0, σ, 0, 0, σ], not 1/σ.

dim_order — Maps data columns to scene dimensions by name. Also reorders and embeds Cholesky factors automatically. For example, dim_order=["z", "y", "x"] declares that the first column of centers corresponds to the scene’s “z” dimension, etc.

When dim_order is specified and the data has fewer dimensions than the scene, the fill and fill_sigma parameters provide fixed coordinates and covariance widths for the unmapped dimensions. The method automatically expands the centers and Cholesky factors to scene dimensionality.

Auto-mapping behavior when None:

  • If splat_ndim == scene_ndim: auto-map to all scene dimensions in order

  • If splat_ndim < scene_ndim and extend_to_all covers the remaining dims: auto-map to spatial dimensions

  • Otherwise: raises an error

extend_to_all — Which scene dimensions the splats should be visible across without attenuation. Can be a list of dimension names or a single name string. Must not overlap with mapped dimensions. Extended dimensions are never attenuated by slice position.

fill — Fixed coordinate values for dimensions not covered by dim_order. For example, fill={"time": 5.0} places all splats at time=5.0.

fill_sigma — Standard deviations for unmapped dimensions in the Cholesky embedding (default 1.0). Controls splat extent in filled dimensions.

For a hand-authored stacked time/channel axis from lower-dimensional splats, use dim_order=["x", "y", "z"], pair fill={"time": t} with fill_sigma={"time": 0.0}, and pass extend_to_all=[] (omitting it auto-broadcasts every unmapped dimension). The embedding treats zero as semantic no-extent and regularizes that semantic zero to 1e-7 before Cholesky decomposition. If the input already has full scene-dimensional centers and factors without dim_order embedding, author a strictly positive diagonal smaller than the coordinate step, because the public writer rejects a literal zero. Use extend_to_all instead when the geometry should remain visible at every coordinate.

Usage Examples

3D Splats in 4D Scene (Extension)

result = fit_gaussian_splats(volume_3d, num_splats=10000)
# result.centers.shape = (10000, 3)
# result.cholesky_factors.shape = (10000, 6)

from luxar import LuxarZarrCompiler, Dimensions, Dimension

dims = Dimensions([
    Dimension("x", display=True), Dimension("y", display=True),
    Dimension("z", display=True), Dimension("time", display=False),
])
with LuxarZarrCompiler("scene.luxar.zarr") as compiler:
    scene = compiler.create_scene(dimensions=dims)

    scene.add_gsplats(
        "cell_structure",
        centers=result.centers,              # (N, 3) - native dimensionality
        amplitudes=result.amplitudes,
        cholesky_factors=result.cholesky_factors,  # (N, 6) - 3D covariance
        colors=result.colors,
        dim_order=["x", "y", "z"],           # Splats span spatial dims
        extend_to_all=["time"]               # Visible at all timepoints
    )

Full-Dimensional Splats (Automatic)

result = fit_gaussian_splats(data_4d, num_splats=50000)
# result.centers.shape = (50000, 4)

from luxar import LuxarZarrCompiler, Dimensions, Dimension

dims = Dimensions([
    Dimension("x", display=True), Dimension("y", display=True),
    Dimension("z", display=True), Dimension("time", display=False),
])
with LuxarZarrCompiler("scene.luxar.zarr") as compiler:
    scene = compiler.create_scene(dimensions=dims)

    scene.add_gsplats(
        "dynamic_process",
        centers=result.centers,              # (N, 4)
        amplitudes=result.amplitudes,
        cholesky_factors=result.cholesky_factors,  # (N, 10)
        # dim_order=None → auto-maps to all scene dimensions
    )

Multiple Extended Dimensions

from luxar import LuxarZarrCompiler, Dimensions, Dimension

dims = Dimensions([
    Dimension("x", display=True), Dimension("y", display=True),
    Dimension("z", display=True), Dimension("time", display=False),
    Dimension("channel", display=False),
])
with LuxarZarrCompiler("scene.luxar.zarr") as compiler:
    scene = compiler.create_scene(dimensions=dims)

    result = fit_gaussian_splats(volume_3d, num_splats=5000)

    scene.add_gsplats(
        "nuclei",
        centers=result.centers,              # (N, 3)
        amplitudes=result.amplitudes,
        cholesky_factors=result.cholesky_factors,  # (N, 6)
        dim_order=["x", "y", "z"],
        extend_to_all=["time", "channel"]    # Visible at all times and channels
    )

Dimension Reordering with Fill

from luxar import LuxarZarrCompiler, Dimensions, Dimension

dims = Dimensions([
    Dimension("x", display=True), Dimension("y", display=True),
    Dimension("z", display=True), Dimension("time", display=False),
])
with LuxarZarrCompiler("scene.luxar.zarr") as compiler:
    scene = compiler.create_scene(dimensions=dims)

    # Centers are in [z, y, x] order from fitting
    result = fit_gaussian_splats(data_zyx, num_splats=1000)

    scene.add_gsplats(
        "reordered",
        centers=result.centers,              # (N, 3) in [z,y,x] order
        amplitudes=result.amplitudes,
        cholesky_factors=result.cholesky_factors,
        dim_order=["z", "y", "x"],          # Explicit ordering by name
        extend_to_all=["time"]
    )

Validation Rules

Rule 1: Shape Consistency

len(dim_order) == centers.shape[1]

The number of specified dimensions must match the splat data dimensionality.

Rule 2: Cholesky Consistency

cholesky_factors.shape[1] == splat_ndim * (splat_ndim + 1) // 2

Cholesky factors must match splat dimensionality, not scene dimensionality. Their values follow the covariance-factor packing described above.

Rule 3: Mutual Exclusivity

set(dim_order) & set(extend_to_all) == {}

A dimension cannot be both spanned and extended:

  • Spanned (dim_order): Splat has position and covariance in this dimension

  • Extended (extend_to_all): Splat is visible everywhere in this dimension

Rule 4: Complete Coverage

set(dim_order) | set(extend_to_all) | set(fill.keys()) == set(scene_dimension_names)

Every scene dimension must be covered by dim_order, extend_to_all, or fill.

Rule 5: Dimension Name Validity

All dimension names must exist in the scene.

Storage Format (Zarr)

Metadata

After dimension mapping is applied, the compiler writes scene-dimensionality data. The zarr metadata reflects the final (expanded) dimensionality:

{
  "type": "gsplats",
  "n_splats": 10000,
  "ndim": 4,
  "has_colors": true,
  "extend_to_all": ["time"]
}

Key fields:

  • ndim: Final splat dimensionality after expansion (matches scene dimensionality). The viewer derives cholesky_k from this as ndim*(ndim+1)/2.

  • extend_to_all: Scene dimension names the splats extend across (visible everywhere). Stored only when non-empty.

  • has_colors: Whether the node has per-splat colors

Zarr Arrays

gsplat_node/
  ├── zarr.json                 (attributes above; default format 3)
  ├── centers                   (N, ndim) uint16 (AUTO; float32 if an axis extent ≥ 2¹⁶) / float32 (PRECISION)
  ├── amplitudes                (N,) or (1,) uint8/uint16 (AUTO) / float32 (PRECISION)
  ├── cholesky_factors_diag     (N, d) or (1, d) uint8 (AUTO, certified — escalates to uint16 if the covariance certificate fails) / float32 (PRECISION), d=ndim (diagonal, scale-like terms)
  ├── cholesky_factors_offdiag  (N, k-d) or (1, k-d) uint8 (AUTO, certified as above) / float32 (PRECISION) (signed off-diagonal; omitted when ndim==1; k=ndim*(ndim+1)/2)
  └── colors                    (N, 3) or (1, 3) uint8/uint16 (AUTO) / float32 (PRECISION)

On-disk dtypes follow the encoding mode (default AUTO quantizes; all arrays decode to float32 on read — see GSplats Zarr Format for the per-array encodings).

Arrays use scene dimensionality (after dim_order expansion), not the original data dimensionality. Since format v3.1 the Cholesky factors are stored on disk split into cholesky_factors_diag (N, ndim) + cholesky_factors_offdiag (N, k-ndim) — each encoded/quantized independently — and recombined into the packed (N, k) cholesky_factors form immediately on read (Python reader and viewer loader), so nothing downstream of the storage boundary sees the split. (1D splats have no off-diagonal terms, so cholesky_factors_offdiag is omitted; legacy v3.0 files store a single packed cholesky_factors, read via presence-detect fallback.)

Viewer Processing

Loading Phase

interface GSplatsMetadata {
  type: 'gsplats';
  n_splats: number;
  ndim: number;                      // Splat dims (matches scene ndim after expansion)
  has_colors: boolean;
  chunk_size: number;
  ordering: 'morton' | 'hilbert' | 'none';
  extend_to_all?: string[];          // Extended dim names (e.g., ["time"])
  // ... plus ordering bounds, transform, rendering params
}

nD to 3D Projection

When rendering with displayDims = [d0, d1, d2] and a slicePosition:

  1. Map display dimensions to splat indices — For each display dimension, find the corresponding splat dimension index (or null if the display dimension is extended/not in splat)

  2. Extract 3D centers — For spanned display dimensions, read from splat centers. For extended display dimensions, place at current slice position. Components with no display dimension behind them (a 2D or 1D view) are zero-filled. The visible centre is the stored coordinate on each displayed dimension, read directly — it is not shifted by any correlation between the displayed and hidden dimensions (see step 3).

  3. Extract 3D Cholesky submatrix — Compute the marginal Cholesky of the sub-covariance over the displayed splat dimensions: the displayed 3D covariance is the marginal Σ_dd (the displayed rows/columns of Σ), not the conditional covariance Σ_dd − Σ_dh·Σ_hh⁻¹·Σ_hd (Schur complement) of the slice through the hidden coordinate. This is a deliberate visualisation choice: every splat is drawn with its full displayed footprint at its own centre, and the slice position only attenuates it (step 4). For categorical/discrete hidden axes such as time the distinction is moot — those axes are gated exactly, and carry no covariance worth conditioning on. For a continuous hidden axis correlated with a displayed one (a non-zero Σ_dh), the exact conditional slice would shift the visible centre by Σ_dh·Σ_hh⁻¹·(s − μ_h) and narrow the footprint to the Schur complement; that conditional projection is not implemented. Note the renderer’s output buffer is always the 6-element packed-3D layout [L00, L10, L11, L20, L21, L22], regardless of how many dimensions are displayed, so with n = min(displayDims.length, 3) < 3 the marginal is only n×n and the remaining rows are synthesized, not extracted:

    • Off-diagonals are 0 — the phantom axis is uncorrelated with the real ones, leaving the in-plane profile exactly as authored.

    • The diagonal is the geometric mean of the real Cholesky pivots, which equals (det Σ_S)^(1/2n) and is therefore rotation-invariant. A 2D splat thus renders as a round blob at its own in-plane scale.

    The phantom diagonal is deliberately not a small epsilon. In sum projection (additive, luminous, volumetric) the shader scales amplitude by the Gaussian’s extent along the view ray, sigmaRay = 1/√(rᵀΣ⁻¹r), so an ε-thin splat viewed face-on is scaled by ~1e-5 and discarded — the scene renders black. luxar.gsplats.lift depends on this directly: it calibrates amplitude as opacity / (rayIntegralFactor · σ), which holds for a 2D lift only because √(σ·σ) == σ.

    See wasm/rust/src/gsplats_processing.rs::compute_display_cholesky_3d and its TypeScript twin.

  4. Gate and attenuate in hidden dimensions — Spanned dimensions that are neither displayed nor extended are handled in two passes. Extended dimensions never attenuate.

    • Discrete hidden dims (categorical/discrete axes, e.g. time) are a hard gate, precomputed on the TypeScript side before the kernel runs: a splat is visible only if its centre lies within half a step (discreteSteps[dim] * 0.5, default step 1) of the slice position on every discrete hidden dim. No Gaussian falloff is applied on these axes.

    • Continuous hidden dims attenuate the amplitude by one joint Gaussian, not a per-dimension product. The kernel forms the offset vector diff = slicePosition − center over all continuous hidden dims, extracts the marginal Cholesky of Σ_hh over exactly those dims (compute_marginal_cholesky, so hidden–hidden correlations are honoured), and evaluates a single Mahalanobis distance d² = diffᵀ·Σ_hh⁻¹·diff. The falloff is a shifted, truncated Gaussian that reaches exactly zero at the truncation radius (C⁰-continuous, so a splat crossing the cutoff fades out rather than popping):

      attenuation = max(0, (exp(−d²/2) − c) / (1 − c)), with c = exp(−truncate²/2)

    Only the amplitude is multiplied by attenuation; the 3D centre and Cholesky from steps 2–3 are written unchanged. A splat whose attenuated amplitude falls below min_amplitude (or is NaN) is dropped from the compacted output.

// Fused kernel — gsplats_processing.rs::project_gsplats_nd_to_3d, per splat i.
if discrete_visibility[i] == 0 { continue; }            // hard gate (precomputed in TS)

let attenuation = if continuous_hidden_dims.is_empty() { 1.0 } else {
    for (h, &dim) in continuous_hidden_dims.iter().enumerate() {
        diff[h] = slice_position[dim] - positions[i * ndim + dim];
    }
    // Marginal Cholesky of Σ_hh over the continuous hidden dims (correlations kept).
    compute_marginal_cholesky(cholesky, i * packed_size, continuous_hidden_dims,
                              num_continuous, &mut hidden_cholesky);
    let d = mahalanobis_distance_internal(&diff[..num_continuous], &hidden_cholesky, num_continuous);
    let c = (-0.5 * truncate * truncate).exp();          // shift so the tail hits 0 at `truncate`
    (((-0.5 * d * d).exp() - c) / (1.0 - c)).max(0.0)
};

let amplitude3d = amplitudes[i] * attenuation;
if amplitude3d < min_amplitude || amplitude3d.is_nan() { continue; }   // culled
// centers3d / cholesky3d (steps 2–3) are emitted unattenuated; only the amplitude changes.

The TypeScript reference kernel in wasm/typescript/gsplats-processing.ts (the >16D path) applies the same shifted-Gaussian truncation.

References