Catalog dedup, curation + Forge Studio (Phase 3)

[arpitg1304]

What

Find near-duplicate episodes from the Phase 2 embeddings and curate a clean, labeled training set — plus Forge Studio, a themed HTML app to explore it all.

forge catalog dedup -c ./forge-catalog --threshold 0.97
forge curate -c ./forge-catalog --where "overall_score > 6" \
    --dedup 0.97 --dedup-policy keep-higher-quality --label approved
forge studio -c ./forge-catalog -o studio.html

Phase 3 of the Forge Data Engine, building on the catalog (#30) and embeddings (#31).

Backend

• Two new tables (bumps catalog SCHEMA_VERSION 2 → 3, additive):
  • dedup_edges — near-dup pairs as facts (similarity, not verdicts), canonical order a < b.
  • curation_labels — an append-log of approved/rejected/held; latest-row-wins per episode.
• forge/catalog/dedup.py: compute_dedup_edges (cosine over episode embeddings, max over shared cameras, idempotent, O(N²) with a logged cap) and curate (WHERE filter + dedup policy → labels).
• DuckDB v_curation view + v_dup_losers(threshold, policy) macro (keep-higher-quality / keep-longer / keep-first).
• CLI: forge catalog dedup, forge curate, forge studio. The per-dataset forge dedup (perceptual-hash) is unchanged — catalog dedup is namespaced to avoid collision.

Forge Studio

forge studio renders a self-contained, themed single-page app — Overview · Corpus · Dedup review · Snapshot — from real catalog data and embedded video thumbnails (extracted via the readers, as data URIs). The design system matches the Forge Studio mockup and the forge visualize dark theme (monochrome chrome; cyan = telemetry, violet = AI-derived, amber = human decision). One shareable file, no server. The Dedup tab does keep/reject with keyboard shortcuts and exports a forge curate command. (Template JS validated with node --check.)

Design notes

• Edges are facts, not verdicts — which episode wins is decided at curation by policy, never baked into the edge.
• Reuses the Phase 2 vectors, the readers for thumbnails, forge.io for cloud sources, and the catalog writer/commit machinery. No existing behavior changed; base CLI still doesn't import heavy deps.

Verified on real data

Ran end-to-end on the droid_100 catalog in MinIO: forge catalog dedup (24 pairs @0.9), forge curate (84 approved / 16 dedup-rejected), forge studio → a 1.7 MB file with 100 episodes and 100 real thumbnails. The in-repo example catalog now ships dedup_edges + curation_labels (catalog.json → v3), so curation + studio work from a fresh checkout (excluded from the built wheel like the rest of that folder).

Testing

• tests/test_catalog_dedup.py: edge canonical order + threshold + idempotency, v_dup_losers policies, curate approve/reject + latest-wins, studio HTML smoke. 13 tests.
• Full existing suite: no regressions.

Not in this phase

Phase 4 — forge snapshot create/materialize (freeze a curated set → export to LeRobot/RLDS for training). A hosted Studio service is a future productization; the catalog is zero-server by design so it stays optional.

🤖 Generated with Claude Code