One Video, One World: Turning Monocular Video into Physical 4D Scenes

Junhao Chen  Boran Zhang  Mingjin Chen  Henghaofan Zhang  Saining Zhang
Congcong Zhu  Hao Zhao  Ruqi Huang  Zhihao Li  Yufei Wang

Tsinghua University · SparcAI Inc. · University of Science and Technology of China
The Hong Kong Polytechnic University · University of Electronic Science and Technology of China · Nanyang Technological University

OVOW reconstructs instance-level, simulation-ready 4D mesh scenes from monocular video. Given a single video, our method decomposes the scene into physically independent mesh instances and recovers rigid-body motions and non-rigid mesh deformations, yielding instance-level meshes ready for downstream physics simulation and editing. We demonstrate results of multi-object collisions, rigid-body motions, and deforming object motions across tabletop, indoor, and in-the-wild scenarios.

---

📖 Abstract

We introduce OVOW, the first training-free system that reconstructs instance-level, simulation-ready 4D mesh scenes from a single monocular video. Recent 4D reconstruction achieves impressive rendering quality, but its outputs (e.g., implicit fields, Gaussian primitives, or point clouds) lack the watertight topology, instance separation, and standardized physical interfaces required by physics simulators and embodied AI. OVOW closes this gap with a four-stage pipeline: a vision-language model discovers, labels, and motion-classifies all instances; category-aware reconstruction yields per-instance meshes for rigid objects and topology-consistent mesh sequences for deformable ones; an iterative render-match-optimize procedure recovers metric scale and 6-DoF pose trajectories; and physics-grounded assembly enforces ground contact and inter-object support. Crucially, we model all motion, rigid and non-rigid, through direct vertex deformation without category-specific priors or skeleton rigging, producing watertight mesh scenes ready for downstream physics simulation and editing. We further establish the first benchmark for structured Video-to-4D evaluation, with metrics for geometric correctness, instance separation, and physical plausibility beyond visual fidelity; the same pipeline doubles as a scalable engine for synthesizing paired video-to-4D simulation data for future 4D world models and embodied AI. Across two synthetic benchmarks (static and 4D), OVOW attains the best overall layout and geometry accuracy and the lowest photometric and semantic error among all baselines, and on monocular video runs one to two orders of magnitude faster than the baselines, while downstream physics simulation confirms its physical stability.

✨ Highlights

• First structured Video-to-4D system. Turns a monocular video into instance-level, simulation-ready 4D mesh scenes — watertight, instance-separated, and physics-engine ready (URDF).
• Fully training-free. Composes pretrained foundation models into a tightly integrated pipeline; no task-specific training.
• Unified motion model. All motion — static, rigid, and deformable — is modeled via direct vertex deformation, with no category priors or skeleton rigging.
• New benchmark & data engine. The first benchmark scoring geometric correctness, instance separation, and physical plausibility beyond visual fidelity; the same pipeline synthesizes paired video ↔ 4D scene data at scale.
• Fast & stable. On video, amortizes to 3.35 s/frame — one to two orders of magnitude faster than single-image baselines — while reconstructed scenes stay physically stable under gravity in simulation.

🔧 Method

OVOW is a fully training-free pipeline with four stages:

#	Stage	What it does	Foundation models
1	VLM-Guided Scene Decomposition	Discovers, uniquely names, and motion-classifies every instance (static / rigid / deformable); produces dense per-frame masks.	Qwen3-VL, SAM3
2	Instance-Level Mesh Reconstruction	Amodal inpainting + feed-forward image-to-3D for static/rigid objects; topology-consistent mesh sequences for deformable ones; metric scale recovery.	FLUX.2, Hi3DGen, Motion324, VGGT
3	Spatiotemporal Pose & Deformation Recovery	Iterative render-match-optimize for metric scale, orientation, and per-frame 6-DoF pose trajectories; decouples global motion from local vertex deformation.	RoMa v2, FoundationPose
4	Physics-Grounded Scene Assembly	Ground-plane estimation and contact projection enforce ground contact and inter-object support; exports a coherent, simulation-ready scene with recovered HDR lighting.	—

🌐 Interactive Demos

Explore browser-native, interactive 3D/4D reconstructions on the project page — instance-level meshes with recovered rigid and non-rigid motion, spanning video-to-4D and image-to-3D examples.

🚀 Code

The code is currently under internal company review and will be released here soon.

Please ⭐ star and watch the repo to be notified as soon as it is available.

📌 Citation

If you find our work useful, please consider citing:

@misc{chen2026videoworldturningmonocular,
      title={One Video, One World: Turning Monocular Video into Physical 4D Scenes},
      author={Junhao Chen and Boran Zhang and Mingjin Chen and Henghaofan Zhang and Saining Zhang and Congcong Zhu and Hao Zhao and Ruqi Huang and Zhihao Li and Yufei Wang},
      year={2026},
      eprint={2606.31388},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2606.31388},
}

🔗 Links

• arXiv: https://arxiv.org/abs/2606.31388
• Project Page: https://OneVideoOneWorld.github.io
• Code: https://github.com/yisuanwang/OVOW