feat: Demo sever, Video, Voxels and elements of VSLAM#2
Open
richiejp wants to merge 22 commits into
Open
Conversation
…o, Vulkan)
Turn time-lapse video into ONE large coherent point cloud via upstream
DA3's sliding-window streaming recipe: run the fused multi-view pass on
overlapping windows, then stitch each window into a single global frame
with a weighted-Umeyama Sim3 on the overlap. No ggml/model changes — pure
host orchestration on top of the existing inference + back_project.
Streaming (mirrors da3_streaming, minus loop closure):
- src/sim3.{hpp,cpp}: weighted-Umeyama Sim3 — Horn quaternion rotation via
the existing linalg jacobi_eigen_sym (reflection-free), optimal Umeyama
scale (outlier-robust, unlike the RMS-ratio form), Huber IRLS; compose/apply.
- src/stream.{hpp,cpp}: overlapping window loop over depth_pose_multi, dense
conf-weighted overlap correspondence (matched by pixel, both-valid),
cumulative Sim3 by composition, global accumulation with per-window conf
filter + budget subsample. counts are per input frame for build-up.
- C API da_capi_points_stream (ABI 5→6); removed the unused
da_capi_depth_pose_multi. Bound in the Go server as PointsStream.
- server: video bake routes through streaming, lifts the 24-frame cap,
exposes chunk/overlap/fps; build-up steps strided to ~30 for long clips.
- tests (host-only): test_sim3 (exact recovery, reflection trap, Huber,
compose) and test_stream_seam (overlap realignment under disjoint masks).
Also lands this session's demo + GPU work it builds on:
- DA3 studio server (server/): lazy model registry, points/gaussian bake,
.splat export + embedded WebGL EWA viewer, model picker, scene gallery.
- Vulkan GPU inference: graph-input upload fix for host norm tensors and
skip-unreferenced-input fix (src/dino_backbone.cpp, src/backend.cpp);
ggml bumped to v0.15.3 (conv_transpose_2d coopmat2 fix, PR #24924).
- gs_head 224×224 guard; multi-view unified on preprocess_real (engine).
Verified: host unit tests pass; end-to-end flower-bed clip → 32 frames /
4 stitched windows (overlap M=423360, RMS ~0.01) → 996k-point coherent
cloud at 60fps (vs ~508k single-window).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
A full multi-view streaming window on GIANT (40 layers) overflowed the fixed ggml metadata context: 12 views needed just over kGraphSize=16384 nodes and aborted with "not enough space in the context's memory pool" mid-backbone. Base multi-view and single-image giant fit; giant x 12-view did not. Raise kGraphSize to 49152 — sized for the heaviest window the server allows (24-view giant, ~33k nodes) with margin. The pool is only tensor metadata (~408 B/node, ~20 MB), allocated per compute() call, so the cost is trivial. Fixes the server SIGABRT when baking a giant-model video via streaming. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The drop box held a <span id="droptype"> whose text setSrc() updated, but
reflectChosen() overwrites the whole box via drop.textContent when a file is
chosen — destroying that span. The next source switch then hit
$("droptype").textContent on a null element and threw, aborting setSrc()
before it updated the file input's accept filter (and multiple / vidparams).
So after picking one file, toggling Video/Photos only moved the highlight and
the picker kept filtering to the first type.
Drop the span; setSrc() no longer touches it. Set accept/multiple fresh in
drop.onclick right before opening the picker (and clear value so re-picking
the same file re-fires onchange). A shared dropText() restores the prompt.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
DA3's single-image gaussians are view-synthesis primitives: their per-pixel
depth is intentionally spread (verified bit-exact against the DA3 reference via
the regenerated giant parity dumps — test_gs_head/test_gs_adapter pass at 2e-3,
and the reference's own gaussians span the same 100x+ depth range at full res).
They only composite into a coherent image from the INPUT camera. The viewer was
free-orbiting them and auto-framing to their huge bounding box, which collapsed
the scene to a floater dot ("no discernible objects").
For gaussian scenes the viewer now frames from the input camera (the world
origin, where the pose head places the single view) looking at the content's
median, and clamps orbit (+-0.3 rad) and zoom to stay near that view. WASD
panning is disabled there so the fixed input view can't be nudged off. Points/
streaming scenes are untouched: free orbit + bbox framing. Result: gaussian mode
now renders the recognizable input view instead of floaters.
… colour
DA3's single-image gaussians only resolve into a sharp picture from the exact
input viewpoint, and their learned SH-DC colour is a deliberately flat, near-grey
base (higher SH bands masked x0.025). The demo baked SH-DC colour and viewed the
depth-spread gaussians through a wrong, too-wide, non-anamorphic projection -- a
scattered grey blob cloud ("extremely washed out").
- Colour each gaussian by its own source pixel (de-normalized p.chw, the exact
model input), same convention as the point cloud -> faithful, saturated colour.
- Expose the input camera intrinsics + processed size from da_capi_gaussians
(ABI 6 -> 7); thread through the Go server into the scene manifest and the
/api/reconstruct response as `cam`.
- Viewer renders gaussian scenes from the origin (pose is canonical identity)
through the input K (anamorphic fx/fy, pillarboxed to height), so every gaussian
lands on its source pixel and the input view resolves. Small clamped orbit gives
parallax. Points scenes unchanged (free orbit + bbox framing).
Result: gaussian scenes composite into the recognizable, correctly-coloured input
image (soft, as inherent to DA3's 224^2 single-image gaussians) instead of a
washed-out blob cloud. Verified by headless GPU render on two scenes; points path
unregressed. Host tests pass (sim3/stream_seam/linalg/reconstruct/gs_adapter, and
capi on CPU with ABI 7; the ctest Vulkan aborts here are the known env issue).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…p-closure pose-graph Collapse the doubled sheets and drift that appear when stitching many sliding-window reconstructions into one coherent cloud: - A: voxel surface fusion (fuse.*) — de-densify + merge coincident sheets. - B: per-seam point-to-plane ICP (icp.*, spatial_hash.hpp) — tighten each window-to-window seam before it accumulates. - C: loop-closure detection + Sim3 pose-graph optimisation (posegraph.*, sim3.*) — remove global drift when the trajectory revisits a place. All params are scene-relative (monocular DA3 lives at arbitrary scale). CPU geometry loops are OpenMP-parallelised. Includes analytic self-tests, synthetic scene/trajectory generators, open3d/gtsam reference-oracle scripts, and standalone dump harnesses. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add Engine::metric_branch — run the metric ViT-L + DPT/sky branch at the streaming resolution (preprocess_real / 504px, so it stays pixel-aligned with the multi-view anyview depth and fits VRAM). The streaming path uses it to fit a per-window robust-median scale that rescales depth + camera translation, turning the arbitrary-scale relative cloud into absolute metres. Includes probes for scale consistency, relative-vs-metric cloud scale, and the max metric depth predicted across clips. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…-ordered additive reveal Point-cloud rendering controls, all scene-relative (radius ∝ depth is used as a proxy throughout): - Surfels: estimate per-point normals (spatial-hash Jacobi PCA) and orient each splat into a flat surface disk instead of a round blob. - Voxel (Minecraft) mode: snap the cloud onto a grid, one cube per cell. - Near bias: depth-weighted thinning that keeps more near points, fewer far. - Confidence-ordered reveal: order points frame-major then smallest-radius (≈ highest-confidence) first, so the build-up reveals solid surface first. - First-frame dedup: overlapping frames re-observe the same surface; keep the first frame to fill each cell so revealing a new frame is purely additive. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…GPU boxes Auto-pick now favours a discrete GPU and only falls back to an iGPU when no discrete device is present. On multi-GPU machines the iGPU often enumerates first (device 0) but is far weaker and can wedge on big models. DA_DEVICE still forces an explicit device by name (e.g. "Vulkan1", "CUDA0"). Co-Authored-By: Claude Opus 4.8 <[email protected]>
…pipeline & C API Integration of the streaming features into stream_points + the flat C API (these edits are superimposed on the same StreamParams / da_capi_points_stream surface, so they land together): - StreamParams gains icp_refine / loop_close / fuse / metric; da_capi_points_stream exposes them (ABI 8) and applies A/B/C + metric rescale per window. - Flythrough: stream emits a global camera pose (centre + forward) per input frame; retrieved via new da_capi_stream_last_poses (ABI 9 — kept as a getter because da_capi_points_stream is already at the FFI stack-arg ceiling). - Per-phase GPU/CPU timing log; purego bindings + test-harness call sites updated. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…, flythrough, reveal Server + WebGL viewer for the streaming demo. These UI edits are superimposed on the shared bake handler / render loop, so the features land together: - Bake controls: ICP (B) / loop-closure (C) / fusion (A) toggles, Metric (m), Near bias, Voxel res, Surfels; window size uncapped (was clamped to 24). - Rendering: oriented-surfel splats, an opaque depth-tested cube program for voxel (Minecraft) mode, scene-relative clip planes. - Progressive reveal: one cloud file per scene; the viewer reveals a growing, confidence-ordered prefix instead of reloading a bigger file each step. - Flythrough (🎥): glide the camera along the per-frame capture path as the build-up advances, using the poses exported by the stream. - no-cache headers so a re-baked scene / rebuilt UI shows up on reload. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…ow size Removing the window cap let a large giant/nested window overflow the fixed 49152-node metadata context — ggml_new_object hard-asserts on overflow and aborts the process (crashing the demo server). Make the graph-node budget a per-call parameter that the multi-view backbone scales with its view count, and recreate the backend scheduler when a later, bigger graph needs more nodes than the first call sized it for. Validated: a 32-view giant window now bakes in one pass (was ~25 views to the crash). Very large windows now fail on VRAM instead of the node pool. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…g flythrough While the flythrough camera follows the capture path, display the input frame it's currently at (round(flyU) -> the per-frame view_<i>.jpg thumbnail the bake already saves) in a top-right PiP, updated as the camera advances and hidden in free-orbit mode. Lets you compare the reconstruction against the real footage. Co-Authored-By: Claude Opus 4.8 <[email protected]>
The "Surface fusion (A)" path now runs a truncated signed-distance fusion
instead of a voxel downsample. Each point splats a confidence-weighted
signed distance along its camera-oriented normal into a voxel field; a
voxel straddling the surface holds the weighted-mean SDF of every sheet
whose truncation band reaches it, so two offset copies of a surface
collapse to their midplane. Sheets farther apart than 2*trunc stay
distinct. This actually removes ghosting, where the old voxel downsample
only thinned the cloud (kept both sheets when their gap exceeded a cell).
- src/tsdf.{hpp,cpp}: fuse_tsdf() — PCA normals, nearest-camera orientation
for globally-consistent SDF sign, normal-march splat, zero-crossing
extraction onto the level set. Scene-relative voxel/truncation.
- src/da_capi.cpp: route the fuse flag through fuse_tsdf (frame_pos orients
the normals; weights default to 1/radius so near/high-confidence wins).
- tests/test_tsdf.cpp: ground-truth de-ghosting metric — a known doubled
sheet collapses to 17% thickness (single sheet on the true midplane),
while fuse_voxel at the same cell leaves it at 99.9% (both ghosts survive).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adds two scene-relative knobs for Surface fusion (A), revealed in the
video panel when fusion is on:
- Fuse voxel — voxel edge as a fraction of the scene diagonal (detail
vs. density); default 0.40%.
- Merge range — truncation as a multiple of the voxel (max merged gap =
2×, in voxels); higher removes more ghosting/drift but can
fuse genuinely-separate surfaces; default 4.
Plumbing: da_capi_set_fuse_params(ctx, voxel_frac, trunc_mult) stashes the
knobs on the ctx for the next streamed fuse (da_capi_points_stream is at the
purego arg ceiling, so a getter/setter split is the established pattern here);
consumed by fuse_tsdf, which resolves fractions/multiples against the cloud's
own bbox. ABI 9 -> 10. TsdfParams gains voxel_frac/trunc_mult; absolute
voxel/trunc still win when set (harness path). Go binding + bakeVideo thread
the two form values through.
Verified end to end: 0.004/4 -> 39,395 pts, 0.010/6 -> 4,363 pts on the same clip.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
TSDF (like the old voxel fusion) re-sorts points spatially and the fuse block only rescaled the per-frame counts proportionally, so the progressive/ flythrough reveal — which reveals a frame-major prefix — showed a spatial voxel slab unrelated to the camera position. During a flythrough large chunks of already-passed geometry read as missing until the slab swept to them. fuse_tsdf now tags each output voxel with the FIRST input frame that observed it (via a per-input-point frame array built from the stream counts), emits frame-major by that frame, and returns the per-point frames; da_capi rebuilds real per-frame counts from them. The reveal now reveals each surface as soon as any camera has seen it and never hides it again, tracking the capture path (clustered by sliding window but monotonic, cumulating to the full cloud). (Nearest-camera assignment was tried and rejected: it back-loads the reveal by hiding already-seen surfaces until the camera's closest approach.) Co-Authored-By: Claude Opus 4.8 <[email protected]>
…ear colour The TSDF emits one point per surface voxel but was keeping the averaged input radius (~0.0014 on a scene where voxels sit ~0.011 apart), so splat scale/spacing collapsed from 0.39 (dense cloud) to 0.07 — the splats covered 7% of the gap and the dark background showed through, reading as a dark, sparse surface. Size each fused splat to 0.6*voxel so the one-point-per-cell surface tiles solidly again (scale/spacing back to ~0.34). Also average colour in LINEAR light (sRGB<->linear) rather than gamma-encoded sRGB. Gamma-space averaging biases merged colour toward black at any local contrast; the global mean barely moved here (so it was not the cause of the perceived darkening) but it is the correct way to blend and helps texture edges. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…r mode Voxel mode: when Surface fusion (A) is on, render one cube per FUSION voxel (cube cell = the C-side fuse_voxel_frac of the bbox diagonal) instead of re-blocking the fused cloud on the separate Voxel-res grid. The single "Fuse voxel" slider now drives both the fusion detail and the block size, so the two never disagree; Voxel-res is used only when fusion is off. UI: a single updateControls() shows only the inputs a given mode/model/source uses — streaming controls are video-only; window/overlap/conf/near-bias and the de-ghost toggles are hidden in single-image gaussians mode; surfels hide in voxel + gaussians modes; metric shows only for a metric-capable model; Voxel res hides when fusion drives the grid. Wired to mode/src/model/fuse changes. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…hading) Voxel cubes are face-lit (0.40 ambient .. 1.0 lit), so away from the light they read much darker than the unshaded points. Add a ☀ brightness slider to the viewer that multiplies the fragment output (u_bright uniform in both the cube and splat programs, default 1.0, range 0.5..3). Cranking it overbrightens the shaded faces to match the points; it doubles as a general exposure control for point/gaussian scenes. Runtime only — the continuous render loop picks up the value each frame, no rebake. Co-Authored-By: Claude Opus 4.8 <[email protected]>
The brightness slider pushed the control row past its box: #steprange is flex:1 but range inputs default to min-width:auto, so it wouldn't shrink to make room. Give it min-width:60px and let .play wrap, so the brightness control drops to a second line on narrow panels instead of overflowing. Co-Authored-By: Claude Opus 4.8 <[email protected]>
The step readout printed the full label ("60 · full 60-view cloud · 81147
pts") on the last step, which crowded the sliders and reflowed the brightness
control onto the frame row as the label appeared/disappeared. Show just the
frame number, and give the brightness control flex-basis:100% so it always sits
on its own line with a flex:1 slider that fills the width.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Two one-time stderr diagnostics so a VRAM/latency complaint can be attributed without guesswork: - backend.cpp: reports whether the fused flash-attention nodes actually run on the GPU or get silently offloaded to CPU (Vulkan gates flash on coopmat2 / subgroup shuffle+vote + head_dim%8; when unmet, supports_op returns false and the scheduler runs attention on CPU, materializing the O(N^2) scores there). Uses the same supports_op the scheduler uses, so it reports the real placement. Guarded to log once per backend. - dino_backbone.cpp: reports resolved embed/heads/head_dim/depth (=model size) and the img/patch -> Ntok/view, S views -> Ntok*S global-attn token count, which is what per-frame activation memory scales with. Re-fires only when a larger window arrives (the VRAM high-water mark), so streaming's many windows don't spam. Validated on Vulkan (RTX 5070 Ti, NV_coopmat2): flash runs on-device, da3-base resolves to embed=768/heads=12/depth=12, 504x280 -> Ntok=721, S=6 -> 4326 tokens. No behavior change; pure DA_LOG output. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Attempting to create coherant worlds from phone video. There are changes to the C api which may need some more review.