Research: SfM / multi-view depth as an improvement or replacement for single-image Depth Anything

[MJohnson459]

Why

The L1 pipeline (#18) gets depth from a single image per frame (relative Depth Anything V2-Small), which has two structural costs we currently pay workarounds for:

• No metric scale. Monocular depth is ambiguous up to a per-frame affine in disparity, and the ambiguity is not stable frame to frame — so we refit scale against lidar every frame (Theil-Sen), with the floor plane as fallback. The whole lidar_rescale/depth_rescale machinery exists only to patch this.
• No temporal consistency. Each frame is estimated from scratch, so depth flickers on a static scene and every downstream stabiliser (plane-fit hold, EMA, inlier gates) is compensating.

But the robot moves, and we know how it moves: wheel + kinematic-ICP odometry gives a metrically-scaled baseline between any two camera poses. Multi-view geometry could give metric depth directly — no lidar refit, no floor fallback — and hold consistency across frames for free.

Candidate approaches (roughly increasing ambition)

1. Odometry-scaled two-view stereo: pick frame pairs with enough baseline from odometry, rectify, and run classical or learned stereo matching. The baseline is known metrically, so depth is metric by construction.
2. Learned multi-view / pointmap models (DUSt3R / MASt3R / VGGT family): feed 2+ frames, get consistent geometry; still needs a scale anchor, but odometry provides it once per pair rather than per pixel-fit.
3. Hybrid: keep Depth Anything for dense per-frame structure, use multi-view triangulation of tracked features (metric, sparse) as the scale/shape anchor instead of the lidar — decoupling the camera pipeline from the lidar entirely.
4. Full incremental SfM / SLAM-style mapping (COLMAP-like, or a visual-odometry frontend): probably too heavy and solves more than we need, but worth a literature pass for embedded-friendly variants.

Constraints to respect

• Inference stays off-board (same depth-server split as feat: Depth perception from mono camera #18); the Pi-side client must stay torch-free and light.
• Latency budget: the current single-image path is ~0.6 s capture→publish and already inference-bound; multi-view must not regress this materially (or must degrade gracefully to the single-image path when stationary — note SfM fundamentally has no signal when the robot isn't moving, so the single-image path likely remains as the stationary fallback).
• Rolling-shutter webcam, motion blur, and textureless floor are the expected failure modes to test explicitly.

Evaluation plan

The offline harness from #18 transfers directly: record a driving bag (pixi run record), replay through the candidate (tools/depth_bag_replay.py pattern), and score depth-vs-lidar range error at matching bearings (tools/depth_obstacles.py, tools/depth_bag_eval.py). Success criteria vs the current pipeline:

• metric accuracy ≥ current lidar-rescaled Depth Anything on the same bag (mean/RMSE/p90 range error)
• temporal stability: less frame-to-frame depth flicker on a static-scene bag
• works (or cleanly falls back) when stationary
• latency within ~2x of the current path on the same hardware