+
---
-## โจ Table of Contents of EmbodiedGen
-[](https://horizonrobotics.github.io/EmbodiedGen/) Follow the documentation to get started!
-
-- [๐ผ๏ธ Image-to-3D](#image-to-3d)
-- [๐ Text-to-3D](#text-to-3d)
-- [๐จ Texture Generation](#texture-generation)
-- [๐ 3D Scene Generation](#3d-scene-generation)
+## โจ What's New in V2
+
+- ๐ฌ **3D Vibe Coding** โ build and edit sim-ready scenes through natural-language dialogue via Claude Code slash commands (`/embodiedgen:*`), each edit a bounded, physics-validated skill call.
+- ๐ **Large-scale scene generation** โ multi-room, navigable, instance-editable houses at controllable complexity tiers (`minimalist` โ `detail`).
+- ๐ฆ **One world, every simulator** โ a standardized layout loads with consistent geometry, collision, and physics across **SAPIEN, Isaac Sim, Isaac Gym, MuJoCo, Genesis, and PyBullet**.
+- ๐งฉ **Pluggable 3D backends** โ switch between **SAM3D**, **TRELLIS**, and the **Hunyuan3D Pro** cloud API with a single flag.
+- ๐งฅ **Beyond rigid bodies** โ text-conditioned garments deploy as deformable meshes in Genesis.
+- ๐ฆพ **Part-level affordance** โ functional part segmentation, per-part semantics, and simulation-validated 6-DoF grasp poses for any generated asset.
+- ๐ค **Closed-loop robot learning** โ policies trained purely in EmbodiedGen-generated worlds transfer to real robots (task success **9.7 โ 79.8%** in sim, **21.7 โ 75.0%** on real robots, from a companion [sim-to-real RL study](https://arxiv.org/abs/2603.18532)).
+
+## ๐ Table of Contents
+
+- [๐ Quick Start](#quick-start)
+- [๐งฑ Generate โ Sim-Ready 3D Assets](#generate-assets)
+- [๐ Scale โ Large-Scale Scenes](#scale-scenes)
+- [๐ Compose โ Task-Driven Worlds](#compose-worlds)
+- [๐ฌ Edit โ 3D Vibe Coding](#vibe-coding)
+- [๐ฆ Export โ Any Simulators](#any-simulators)
+- [๐ค Train โ Robot Learning](#robot-learning)
- [โ๏ธ Articulated Object Generation](#articulated-object-generation)
-- [๐๏ธ Layout (Interactive 3D Worlds) Generation](#layout-generation)
-- [๐ฎ Any Simulators](#any-simulators)
-
-[๐ฌ Feedback Wanted: How Do You Use EmbodiedGen & Whatโs Missing?](https://github.com/HorizonRobotics/EmbodiedGen/issues/66)
+- [๐งฉ 3D Scene Completion](#3d-scene-completion)
-## ๐ Quick Start
+---
-[](https://horizonrobotics.github.io/EmbodiedGen/)
+
----
+```sh
+# Image โ 3D (backends: SAM3D | TRELLIS | HUNYUAN3D, via --image3d_model)
+img3d-cli --image_path apps/assets/example_image/sample_01.jpg --output_root outputs/imageto3d
-
+The same generate-and-export path reaches **soft bodies**: text-conditioned garments deploy as deformable meshes in Genesis.
-### โ๏ธ Service
-Run the image-to-3D generation service locally.
-Models downloaded automatically on first run, please be patient.
-```sh
-# Run in foreground
-python apps/image_to_3d.py
-# Or run in the background
-CUDA_VISIBLE_DEVICES=0 nohup python apps/image_to_3d.py > /dev/null 2>&1 &
-```
+Any generated URDF can be further annotated with **part-level affordances** โ functional part segmentation, per-part semantics, and simulation-validated 6-DoF grasps (requires `bash install.sh affordance`):
-### โก API
-Generate physically plausible 3D assets from image input via the command-line API.
```sh
-img3d-cli --image_path apps/assets/example_image/sample_00.jpg apps/assets/example_image/sample_01.jpg \
---n_retry 2 --output_root outputs/imageto3d
-
-# See result(.urdf/mesh.obj/mesh.glb/gs.ply) in ${output_root}/sample_xx/result
+affordance-cli --urdf-paths apps/assets/example_affordance/ear_hear/sample.urdf \
+ --output-dirs outputs/affordance_annotation/ear_hear
```
-Support the use of [SAM3D](https://github.com/facebookresearch/sam-3d-objects) or [TRELLIS](https://github.com/microsoft/TRELLIS) as 3D generation model, modify `IMAGE3D_MODEL` in `embodied_gen/scripts/imageto3d.py` to switch model.
+โก๏ธ Full guides: [Image-to-3D](https://horizonrobotics.github.io/EmbodiedGen/docs/tutorials/image_to_3d.html) ยท [Text-to-3D](https://horizonrobotics.github.io/EmbodiedGen/docs/tutorials/text_to_3d.html) ยท [Texture Generation](https://horizonrobotics.github.io/EmbodiedGen/docs/tutorials/texture_edit.html) ยท [Soft-Body Simulation](https://horizonrobotics.github.io/EmbodiedGen/docs/tutorials/soft_body.html) ยท [Affordance](https://horizonrobotics.github.io/EmbodiedGen/docs/tutorials/affordance.html)
---
+
+
-### โ๏ธ Service
-Deploy the text-to-3D generation service locally.
-
-Text-to-image model based on the Kolors model, supporting Chinese and English prompts.
-Models downloaded automatically on first run, please be patient.
```sh
-python apps/text_to_3d.py
-```
-
-### โก API
-Text-to-image model based on SD3.5 Medium, English prompts only.
-Usage requires agreement to the [model license(click accept)](https://huggingface.co/stabilityai/stable-diffusion-3.5-medium), models downloaded automatically.
-
-For large-scale 3D asset generation, set `--n_image_retry=4` `--n_asset_retry=3` `--n_pipe_retry=2`, slower but better, via automatic checking and retries. For more diverse results, omit `--seed_img`.
+# Room / house from a natural-language description (requires `bash install.sh room`)
+room-cli -m embodied_gen.scripts.room_gen.gen_room \
+ --output-root outputs/rooms --prompt "Wipe the table in a simple dining room"
+# or specify the profile explicitly: --room-type Kitchen --seed 42 --complexity simple
-```sh
-text3d-cli --prompts "small bronze figurine of a lion" "A globe with wooden base" "wooden table with embroidery" \
- --n_image_retry 1 --n_asset_retry 1 --n_pipe_retry 1 --seed_img 0 \
- --output_root outputs/textto3d
+# 3DGS background scene from text (requires `bash install.sh scene3d`)
+scene3d-cli --prompts "Art studio with easel and canvas" --output_dir outputs/bg_scenes/ --seed 0
```
-Text-to-image model based on the Kolors model.
-```sh
-bash embodied_gen/scripts/textto3d.sh \
- --prompts "A globe with wooden base and latitude and longitude lines" "ๆฉ่ฒ็ตๅจๆ้ป๏ผๆ็ฃจๆ็ป่" \
- --output_root outputs/textto3d_k
-```
-ps: models with more permissive licenses found in `embodied_gen/models/image_comm_model.py`
+โก๏ธ Full guides: [Room Generation](https://horizonrobotics.github.io/EmbodiedGen/docs/tutorials/room_gen.html) ยท [3D Scene Generation](https://horizonrobotics.github.io/EmbodiedGen/docs/tutorials/scene_gen.html)
---
+
+From a natural-language task description, EmbodiedGen parses a **scene graph** and composes a physically stable, directly loadable **interactive 3D world** โ background, context objects, manipulated targets, distractors, and robot.
+
+
```sh
-CUDA_VISIBLE_DEVICES=0 scene3d-cli \
---prompts "Art studio with easel and canvas" \
---output_dir outputs/bg_scenes/ \
---seed 0 \
---gs3d.max_steps 4000 \
---disable_pano_check
+bash install/install_agent_plugin.sh # register the plugin in Claude Code
```
----
-
-
-
+| Command | What it does |
+|---------|--------------|
+| `/embodiedgen:gen_assets` | Generate 3D assets from images or text |
+| `/embodiedgen:gen_indoor` | Generate rooms or multi-room houses |
+| `/embodiedgen:gen_bg` | Generate 3DGS background scenes |
+| `/embodiedgen:gen_layout` | Compose task-driven interactive worlds |
+| `/embodiedgen:vibe3d` | Insert / remove / place objects in a scene via natural language |
+| `/embodiedgen:sim` | Render layouts in SAPIEN simulation |
+| `/embodiedgen:convert` | Export assets to USD / MJCF / URDF |
+| `/embodiedgen:process` | Scale or rotate existing assets |
+โก๏ธ Full guide: [3D Vibe Coding](https://horizonrobotics.github.io/EmbodiedGen/docs/tutorials/vibe_coding.html)
---
+
-Text-to-image model based on SD3.5 Medium, usage requires agreement to the [model license](https://huggingface.co/stabilityai/stable-diffusion-3.5-medium). All models auto-downloaded at the first run.
+โก๏ธ Full guide: [Any Simulators](https://horizonrobotics.github.io/EmbodiedGen/docs/tutorials/any_simulators.html) (conversion API & examples)
-You can generate any desired room as background using `scene3d-cli`. As each scene takes approximately 30 minutes to generate, we recommend pre-generating them for efficiency and adding them to `outputs/example_gen_scenes/scene_part_list.txt`.
+---
-We provided some sample background assets created with `scene3d-cli`. Download them(~2G) using:
-```sh
-hf download HorizonRobotics/EmbodiedGenData \
- --repo-type dataset --local-dir outputs \
- --include "example_gen_scenes/scene_00[01][0-9]/**" \
- "example_gen_scenes/scene_part_list.txt"
-```
+
+
---
-
+
---
## For Developer
+
```sh
pip install -e .[dev] && pre-commit install
python -m pytest # Pass all unit-test are required.
-# mkdocs serve --dev-addr 0.0.0.0:8000
-# mkdocs gh-deploy --force
```
## ๐ Citation
If you use EmbodiedGen in your research or projects, please cite:
+
+```bibtex
+@article{wang2026embodiedgenv2,
+ title = {EmbodiedGen V2: An Agentic, Simulation-Ready 3D World Engine for Embodied AI},
+ author = {Xinjie Wang and Liu Liu and Taojun Ding and Andrew Choi and Chaodong Huang and
+ Mengao Zhao and Ziang Li and Jackson Jiang and Chunlei Yu and Shengxiang Liu and
+ Wei Xu and Zhizhong Su},
+ journal = {arXiv preprint arXiv:2506.10600},
+ year = {2026}
+}
+```
```bibtex
@misc{wang2025embodiedgengenerative3dworld,
- title={EmbodiedGen: Towards a Generative 3D World Engine for Embodied Intelligence},
- author={Xinjie Wang and Liu Liu and Yu Cao and Ruiqi Wu and Wenkang Qin and Dehui Wang and Wei Sui and Zhizhong Su},
- year={2025},
- eprint={2506.10600},
- archivePrefix={arXiv},
- primaryClass={cs.RO},
- url={https://arxiv.org/abs/2506.10600},
+ title = {EmbodiedGen: Towards a Generative 3D World Engine for Embodied Intelligence},
+ author = {Xinjie Wang and Liu Liu and Yu Cao and Ruiqi Wu and Wenkang Qin and
+ Dehui Wang and Wei Sui and Zhizhong Su},
+ year = {2025},
+ eprint = {2506.10600},
+ archivePrefix = {arXiv},
+ primaryClass = {cs.RO},
+ url = {https://arxiv.org/abs/2506.10600}
}
```
@@ -341,7 +273,8 @@ If you use EmbodiedGen in your research or projects, please cite:
## ๐ Acknowledgement
EmbodiedGen builds upon the following amazing projects and models:
-๐ [Trellis](https://github.com/microsoft/TRELLIS) | ๐ [Hunyuan-Delight](https://huggingface.co/tencent/Hunyuan3D-2/tree/main/hunyuan3d-delight-v2-0) | ๐ [Segment Anything](https://github.com/facebookresearch/segment-anything) | ๐ [Rembg](https://github.com/danielgatis/rembg) | ๐ [RMBG-1.4](https://huggingface.co/briaai/RMBG-1.4) | ๐ [Stable Diffusion x4](https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler) | ๐ [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN) | ๐ [Kolors](https://github.com/Kwai-Kolors/Kolors) | ๐ [ChatGLM3](https://github.com/THUDM/ChatGLM3) | ๐ [Aesthetic Score](http://captions.christoph-schuhmann.de/aesthetic_viz_laion_sac+logos+ava1-l14-linearMSE-en-2.37B.html) | ๐ [Pano2Room](https://github.com/TrickyGo/Pano2Room) | ๐ [Diffusion360](https://github.com/ArcherFMY/SD-T2I-360PanoImage) | ๐ [Kaolin](https://github.com/NVIDIAGameWorks/kaolin) | ๐ [diffusers](https://github.com/huggingface/diffusers) | ๐ [gsplat](https://github.com/nerfstudio-project/gsplat) | ๐ [QWEN-2.5VL](https://github.com/QwenLM/Qwen2.5-VL) | ๐ [GPT4o](https://platform.openai.com/docs/models/gpt-4o) | ๐ [SD3.5](https://huggingface.co/stabilityai/stable-diffusion-3.5-medium) | ๐ [ManiSkill](https://github.com/haosulab/ManiSkill) | ๐ [SAM3D](https://github.com/facebookresearch/sam-3d-objects)
+๐ [Trellis](https://github.com/microsoft/TRELLIS) | ๐ [Hunyuan-Delight](https://huggingface.co/tencent/Hunyuan3D-2/tree/main/hunyuan3d-delight-v2-0) | ๐ [Segment Anything](https://github.com/facebookresearch/segment-anything) | ๐ [Rembg](https://github.com/danielgatis/rembg) | ๐ [RMBG-1.4](https://huggingface.co/briaai/RMBG-1.4) | ๐ [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN) | ๐ [Kolors](https://github.com/Kwai-Kolors/Kolors) | ๐ [Aesthetic Score](http://captions.christoph-schuhmann.de/aesthetic_viz_laion_sac+logos+ava1-l14-linearMSE-en-2.37B.html) | ๐ [Pano2Room](https://github.com/TrickyGo/Pano2Room) | ๐ [Diffusion360](https://github.com/ArcherFMY/SD-T2I-360PanoImage) | ๐ [Kaolin](https://github.com/NVIDIAGameWorks/kaolin) | ๐ [diffusers](https://github.com/huggingface/diffusers) | ๐ [gsplat](https://github.com/nerfstudio-project/gsplat) | ๐ [ChatGPT](https://platform.openai.com/docs/models/gpt-4o) | ๐ [SD3.5](https://huggingface.co/stabilityai/stable-diffusion-3.5-medium) | ๐ [ManiSkill](https://github.com/haosulab/ManiSkill) | ๐ [SAM3D](https://github.com/facebookresearch/sam-3d-objects) | ๐ [infinigen](https://github.com/princeton-vl/infinigen)
+๐ [Trellis](https://github.com/microsoft/TRELLIS) | ๐ [Hunyuan-Delight](https://huggingface.co/tencent/Hunyuan3D-2/tree/main/hunyuan3d-delight-v2-0) | ๐ [Hunyuan3D-Part](https://github.com/Tencent-Hunyuan/Hunyuan3D-Part) | ๐ [GraspGen](https://github.com/NVlabs/GraspGen) | ๐ [Segment Anything](https://github.com/facebookresearch/segment-anything) | ๐ [Rembg](https://github.com/danielgatis/rembg) | ๐ [RMBG-1.4](https://huggingface.co/briaai/RMBG-1.4) | ๐ [Stable Diffusion x4](https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler) | ๐ [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN) | ๐ [Kolors](https://github.com/Kwai-Kolors/Kolors) | ๐ [ChatGLM3](https://github.com/THUDM/ChatGLM3) | ๐ [Aesthetic Score](http://captions.christoph-schuhmann.de/aesthetic_viz_laion_sac+logos+ava1-l14-linearMSE-en-2.37B.html) | ๐ [Pano2Room](https://github.com/TrickyGo/Pano2Room) | ๐ [Diffusion360](https://github.com/ArcherFMY/SD-T2I-360PanoImage) | ๐ [Kaolin](https://github.com/NVIDIAGameWorks/kaolin) | ๐ [diffusers](https://github.com/huggingface/diffusers) | ๐ [gsplat](https://github.com/nerfstudio-project/gsplat) | ๐ [QWEN-2.5VL](https://github.com/QwenLM/Qwen2.5-VL) | ๐ [GPT4o](https://platform.openai.com/docs/models/gpt-4o) | ๐ [SD3.5](https://huggingface.co/stabilityai/stable-diffusion-3.5-medium) | ๐ [ManiSkill](https://github.com/haosulab/ManiSkill) | ๐ [SAM3D](https://github.com/facebookresearch/sam-3d-objects) | ๐ [infinigen](https://github.com/princeton-vl/infinigen)
---
diff --git a/apps/assets/example_scene/text_prompts.txt b/apps/assets/example_scene/text_prompts.txt
index 727df54..eae0049 100644
--- a/apps/assets/example_scene/text_prompts.txt
+++ b/apps/assets/example_scene/text_prompts.txt
@@ -1,75 +1,57 @@
# Living Room
-Cozy living room with soft sofa and warm lighting
-Spacious living room with wooden floor and large windows
-Modern living room with neutral colors and minimalist decor
-Bright living room with plants and natural sunlight
-Classic living room with comfortable armchairs and carpet
+Rustic spacious living room with wooden beams and stone fireplace
+Minimalist spacious living room with neutral tones and plush throw blankets
# Kitchen
Clean kitchen with white cabinets and stainless steel appliances
-Kitchen with wooden countertops and hanging pots
Rustic kitchen with brick walls and vintage stove
-Bright kitchen with large island and bar stools
-Modern kitchen with sleek design and open shelves
+Farmhouse kitchen with open shelves and ceramic sink
# Bedroom
-Minimalist bedroom with white walls and cozy bed
-Warm bedroom with wooden furniture and soft rugs
-Bedroom with desk and natural light
-Classic bedroom with bedside lamps and curtains
-Bright bedroom with colorful pillows and plants
-
-# Dining Room
-Elegant dining room with wooden table and chandelier
-Casual dining room with bench seating and simple decor
-Bright dining room with large windows and natural light
-Rustic dining room with exposed beams and farmhouse table
-Modern dining room with glass table and sleek chairs
+Classic spacious bedroom with bedside lamps and curtains
+Minimalist bedroom with white linens and wooden bedframe
+Modern bedroom with sleek furniture and abstract wall art
# Office Room
Organized office with wooden desk and ergonomic chair
-Office with computer setup and bookshelves
-Bright office with large windows and plants
-Minimalist office with clean desk and laptop
-Cozy office with soft lighting and storage cabinets
+Industrial office with metal desk and exposed brick wall
+Bright office with large windows and potted plants
# Study Room
-Quiet study room with large desk and bookshelf
-Study room with comfortable chair and lamp
-Bright study room with natural light and workspace
Classic study room with wooden furniture and papers
-Modern study room with clean design and minimal clutter
+Rustic study room with wooden bookshelves and woven rug
# Gym Room
-Home gym with treadmill and free weights
-Gym with yoga mats and dumbbells
-Bright gym with large mirrors and exercise bike
Minimalist gym with clean floor and resistance bands
-Spacious gym with bench press and fitness equipment
+Home gym with yoga mats and natural light
# Workshop
-Woodworking workshop with tools and workbench
Workshop with organized tool storage
Bright workshop with large windows and materials
-Metalworking workshop with machines and safety gear
Creative workshop with paint and crafting supplies
-# Bathroom
-Bright modern bathroom with wooden accents
-Cozy tiled bathroom with warm lighting
-
-# Toilet Room
-Minimalist toilet room with white ceramic fixtures
-Compact toilet room with wall-mounted sink
-
# Storage Room
Organized storage room with labeled boxes
-Cluttered storage room with metal shelves
-
-# Meeting Room
-Bright meeting room with glass walls
-Classic meeting room with wooden table
+Minimalist storage room with stackable bins and clean floor
+Compact storage room with foldable shelves and bright lighting
# Studio Room
Art studio with easel and canvas
-Music studio with acoustic foam panels
\ No newline at end of file
+Dance studio with mirrored walls and wooden floor
+Creative studio with mixed media supplies and colorful decor
+
+# Bathroom
+Modern bathroom with white tiles and glass shower
+Cozy bathroom with clawfoot tub and soft towels
+Minimalist bathroom with clean lines and monochrome palette
+
+# Lab or Office Room
+Sleek, sterile modern laboratory with bright white surfaces
+Spotless laboratory with labeled shelves and tools
+Well-lit clean lab with uncluttered work surfaces
+Plain, clean laboratory with basic lab furniture
+Bright, organized laboratory with advanced digital displays
+Modern workspace mixing sterile lab benches and casual seating
+Immaculate lab filled with advanced scientific devices
+Simple sterile lab with white walls and clear counters
+Minimalist office laboratory with seamless white surfaces
\ No newline at end of file
diff --git a/apps/common.py b/apps/common.py
index 55e30da..449d026 100644
--- a/apps/common.py
+++ b/apps/common.py
@@ -14,11 +14,20 @@
# implied. See the License for the specific language governing
# permissions and limitations under the License.
-import spaces
-from embodied_gen.utils.monkey_patches import monkey_path_trellis
+import spaces # noqa: E402
+from embodied_gen.utils.monkey_patch.gradio import (
+ _disable_xformers_flash3,
+ _neutralize_warp_in_parent,
+ _patch_open3d_cuda_device_count_bug,
+)
+from embodied_gen.utils.monkey_patch.trellis import monkey_path_trellis
+_neutralize_warp_in_parent()
+_patch_open3d_cuda_device_count_bug()
+_disable_xformers_flash3()
monkey_path_trellis()
+
import gc
import logging
import os
@@ -45,7 +54,7 @@
RembgRemover,
SAMPredictor,
)
-from embodied_gen.models.sr_model import ImageRealESRGAN, ImageStableSR
+from embodied_gen.models.sr_model import ImageRealESRGAN
from embodied_gen.scripts.render_gs import entrypoint as render_gs_api
from embodied_gen.scripts.render_mv import build_texture_gen_pipe, infer_pipe
from embodied_gen.scripts.text2image import (
@@ -73,7 +82,6 @@
current_dir = os.path.dirname(current_file_path)
sys.path.append(os.path.join(current_dir, ".."))
from thirdparty.TRELLIS.trellis.pipelines import TrellisImageTo3DPipeline
-from thirdparty.TRELLIS.trellis.utils import postprocessing_utils
logging.basicConfig(
format="%(asctime)s - %(levelname)s - %(message)s", level=logging.INFO
@@ -81,18 +89,20 @@
logger = logging.getLogger(__name__)
os.environ["GRADIO_ANALYTICS_ENABLED"] = "false"
-
+os.environ.setdefault("OPENAI_API_KEY", "sk-placeholder")
MAX_SEED = 100000
+# Global variables for lazy initialization
+_RBG_REMOVER = None
+_RBG14_REMOVER = None
+
# DELIGHT = DelightingModel()
# IMAGESR_MODEL = ImageRealESRGAN(outscale=4)
# IMAGESR_MODEL = ImageStableSR()
if os.getenv("GRADIO_APP").startswith("imageto3d"):
- RBG_REMOVER = RembgRemover()
- RBG14_REMOVER = BMGG14Remover()
SAM_PREDICTOR = SAMPredictor(model_type="vit_h", device="cpu")
if "sam3d" in os.getenv("GRADIO_APP"):
- PIPELINE = Sam3dInference()
+ PIPELINE = Sam3dInference(device="cuda")
else:
PIPELINE = TrellisImageTo3DPipeline.from_pretrained(
"microsoft/TRELLIS-image-large"
@@ -107,10 +117,8 @@
)
os.makedirs(TMP_DIR, exist_ok=True)
elif os.getenv("GRADIO_APP").startswith("textto3d"):
- RBG_REMOVER = RembgRemover()
- RBG14_REMOVER = BMGG14Remover()
if "sam3d" in os.getenv("GRADIO_APP"):
- PIPELINE = Sam3dInference()
+ PIPELINE = Sam3dInference(device="cuda")
else:
PIPELINE = TrellisImageTo3DPipeline.from_pretrained(
"microsoft/TRELLIS-image-large"
@@ -130,11 +138,7 @@
elif os.getenv("GRADIO_APP") == "texture_edit":
DELIGHT = DelightingModel()
IMAGESR_MODEL = ImageRealESRGAN(outscale=4)
- PIPELINE_IP = build_texture_gen_pipe(
- base_ckpt_dir="./weights",
- ip_adapt_scale=0.7,
- device="cuda",
- )
+ PIPELINE_HAS_IP_ADAPTER = False
PIPELINE = build_texture_gen_pipe(
base_ckpt_dir="./weights",
ip_adapt_scale=0,
@@ -163,6 +167,9 @@ def preprocess_image_fn(
rmbg_tag: str = "rembg",
preprocess: bool = True,
) -> tuple[Image.Image, Image.Image]:
+ """Preprocess image with lazy model initialization to avoid CUDA init at import time."""
+ global _RBG_REMOVER, _RBG14_REMOVER
+
if isinstance(image, str):
image = Image.open(image)
elif isinstance(image, np.ndarray):
@@ -170,7 +177,16 @@ def preprocess_image_fn(
image_cache = image.copy() # resize_pil(image.copy(), 1024)
- bg_remover = RBG_REMOVER if rmbg_tag == "rembg" else RBG14_REMOVER
+ # Lazy initialization - models are created on first call within @spaces.GPU context
+ if rmbg_tag == "rembg":
+ if _RBG_REMOVER is None:
+ _RBG_REMOVER = RembgRemover()
+ bg_remover = _RBG_REMOVER
+ else:
+ if _RBG14_REMOVER is None:
+ _RBG14_REMOVER = BMGG14Remover()
+ bg_remover = _RBG14_REMOVER
+
image = bg_remover(image)
image = keep_largest_connected_component(image)
@@ -276,7 +292,7 @@ def image_to_3d(
sam_image: Image.Image = None,
is_sam_image: bool = False,
req: gr.Request = None,
-) -> tuple[dict, str]:
+) -> tuple[object, str]:
if is_sam_image:
seg_image = filter_image_small_connected_components(sam_image)
seg_image = Image.fromarray(seg_image, mode="RGBA")
@@ -286,6 +302,7 @@ def image_to_3d(
if isinstance(seg_image, np.ndarray):
seg_image = Image.fromarray(seg_image)
+ logger.info("Start generating 3D representation from image...")
if isinstance(PIPELINE, Sam3dInference):
outputs = PIPELINE.run(
seg_image,
@@ -334,7 +351,7 @@ def image_to_3d(
def extract_3d_representations_v2(
- state: dict,
+ state: object,
enable_delight: bool,
texture_size: int,
req: gr.Request,
@@ -401,7 +418,7 @@ def extract_3d_representations_v2(
def extract_3d_representations_v3(
- state: dict,
+ state: object,
enable_delight: bool,
texture_size: int,
req: gr.Request,
@@ -650,27 +667,62 @@ def generate_texture_mvimages(
sub_idxs: tuple[tuple[int]] = ((0, 1, 2), (3, 4, 5)),
req: gr.Request = None,
) -> list[str]:
+ global PIPELINE, PIPELINE_HAS_IP_ADAPTER
+
output_root = os.path.join(TMP_DIR, str(req.session_hash))
use_ip_adapter = True if ip_img_path and ip_adapt_scale > 0 else False
- PIPELINE_IP.set_ip_adapter_scale([ip_adapt_scale])
- img_save_paths = infer_pipe(
- index_file=f"{output_root}/condition/index.json",
- controlnet_cond_scale=controlnet_cond_scale,
- guidance_scale=guidance_scale,
- strength=strength,
- num_inference_steps=num_inference_steps,
- ip_adapt_scale=ip_adapt_scale,
- ip_img_path=ip_img_path,
- uid=uid,
- prompt=prompt,
- save_dir=f"{output_root}/multi_view",
- sub_idxs=sub_idxs,
- pipeline=PIPELINE_IP if use_ip_adapter else PIPELINE,
- seed=seed,
- )
- gc.collect()
- torch.cuda.empty_cache()
+ if PIPELINE is None:
+ PIPELINE = build_texture_gen_pipe(
+ base_ckpt_dir="./weights",
+ ip_adapt_scale=0,
+ device="cuda",
+ )
+
+ if use_ip_adapter and not PIPELINE_HAS_IP_ADAPTER:
+ logger.info("Load IP adapter into default texture pipeline")
+ if hasattr(PIPELINE.unet, "encoder_hid_proj"):
+ PIPELINE.unet.text_encoder_hid_proj = (
+ PIPELINE.unet.encoder_hid_proj
+ )
+ PIPELINE.load_ip_adapter(
+ "./weights/Kolors-IP-Adapter-Plus",
+ subfolder="",
+ weight_name=["ip_adapter_plus_general.bin"],
+ )
+ PIPELINE_HAS_IP_ADAPTER = True
+
+ if PIPELINE_HAS_IP_ADAPTER:
+ PIPELINE.set_ip_adapter_scale(
+ [ip_adapt_scale if use_ip_adapter else 0.0]
+ )
+
+ try:
+ img_save_paths = infer_pipe(
+ index_file=f"{output_root}/condition/index.json",
+ controlnet_cond_scale=controlnet_cond_scale,
+ guidance_scale=guidance_scale,
+ strength=strength,
+ num_inference_steps=num_inference_steps,
+ ip_adapt_scale=ip_adapt_scale if use_ip_adapter else 0.0,
+ ip_img_path=ip_img_path if use_ip_adapter else None,
+ uid=uid,
+ prompt=prompt,
+ save_dir=f"{output_root}/multi_view",
+ sub_idxs=sub_idxs,
+ pipeline=PIPELINE,
+ seed=seed,
+ )
+ finally:
+ if use_ip_adapter and PIPELINE_HAS_IP_ADAPTER:
+ logger.info("Unload IP adapter from default texture pipeline")
+ if hasattr(PIPELINE, "unload_ip_adapter"):
+ PIPELINE.unload_ip_adapter()
+ else:
+ PIPELINE = None
+ PIPELINE_HAS_IP_ADAPTER = False
+ gc.collect()
+ torch.cuda.empty_cache()
return img_save_paths + img_save_paths
diff --git a/apps/image_to_3d.py b/apps/image_to_3d.py
index 5bf5fc9..09158cb 100644
--- a/apps/image_to_3d.py
+++ b/apps/image_to_3d.py
@@ -41,14 +41,31 @@
app_name = os.getenv("GRADIO_APP")
if app_name == "imageto3d_sam3d":
- enable_pre_resize = False
+ _enable_pre_resize_default = False
sample_step = 25
bg_rm_model_name = "rembg" # "rembg", "rmbg14"
elif app_name == "imageto3d":
- enable_pre_resize = True
+ _enable_pre_resize_default = True
sample_step = 12
bg_rm_model_name = "rembg" # "rembg", "rmbg14"
+current_rmbg_tag = bg_rm_model_name
+
+
+def set_current_rmbg_tag(rmbg: str) -> None:
+ global current_rmbg_tag
+ current_rmbg_tag = rmbg
+
+
+def preprocess_example_image(
+ img: str,
+) -> tuple[object, object, gr.Button]:
+ image, image_cache = preprocess_image_fn(
+ img, current_rmbg_tag, _enable_pre_resize_default
+ )
+ return image, image_cache, gr.Button(interactive=True)
+
+
with gr.Blocks(delete_cache=(43200, 43200), theme=custom_theme) as demo:
gr.HTML(image_css, visible=False)
gr.HTML(lighting_css, visible=False)
@@ -72,12 +89,10 @@
๐ผ๏ธ Generate physically plausible 3D asset from single input image.
- """.format(
- VERSION=VERSION
- ),
+ """.format(VERSION=VERSION),
elem_classes=["header"],
)
-
+ enable_pre_resize = gr.State(_enable_pre_resize_default)
with gr.Row():
with gr.Column(scale=3):
with gr.Tabs() as input_tabs:
@@ -98,11 +113,9 @@
height=400,
elem_classes=["image_fit"],
)
- gr.Markdown(
- """
+ gr.Markdown("""
If you are not satisfied with the auto segmentation
- result, please switch to the `Image(SAM seg)` tab."""
- )
+ result, please switch to the `Image(SAM seg)` tab.""")
with gr.Tab(
label="Image(SAM seg)", id=1
) as samimage_input_tab:
@@ -159,11 +172,11 @@
)
with gr.Row():
randomize_seed = gr.Checkbox(
- label="Randomize Seed", value=False
+ label="Randomize Seed", value=True
)
project_delight = gr.Checkbox(
label="Back-project Delight",
- value=True,
+ value=False,
)
gr.Markdown("Geo Structure Generation")
with gr.Row():
@@ -262,7 +275,6 @@
has quality inspection, open with an editor to view details.
"""
)
- enable_pre_resize = gr.State(enable_pre_resize)
with gr.Row() as single_image_example:
examples = gr.Examples(
label="Image Gallery",
@@ -272,11 +284,12 @@
glob("apps/assets/example_image/*")
)
],
- inputs=[image_prompt, rmbg_tag, enable_pre_resize],
- fn=preprocess_image_fn,
- outputs=[image_prompt, raw_image_cache],
+ inputs=[image_prompt],
+ fn=preprocess_example_image,
+ outputs=[image_prompt, raw_image_cache, generate_btn],
run_on_click=True,
examples_per_page=10,
+ cache_examples=False,
)
with gr.Row(visible=False) as single_sam_image_example:
@@ -339,10 +352,24 @@
)
image_prompt.upload(
- preprocess_image_fn,
- inputs=[image_prompt, rmbg_tag, enable_pre_resize],
+ lambda img, rmbg: preprocess_image_fn(
+ img, rmbg, _enable_pre_resize_default
+ ),
+ inputs=[image_prompt, rmbg_tag],
outputs=[image_prompt, raw_image_cache],
+ queue=False,
+ ).success(
+ active_btn_by_content,
+ inputs=image_prompt,
+ outputs=generate_btn,
+ )
+
+ rmbg_tag.change(
+ set_current_rmbg_tag,
+ inputs=[rmbg_tag],
+ outputs=[],
)
+
image_prompt.change(
lambda: tuple(
[
@@ -381,10 +408,9 @@
est_mu_text,
],
)
- image_prompt.change(
- active_btn_by_content,
- inputs=image_prompt,
- outputs=generate_btn,
+ image_prompt.clear(
+ lambda: gr.Button(interactive=False),
+ outputs=[generate_btn],
)
image_prompt_sam.upload(
diff --git a/apps/text_to_3d.py b/apps/text_to_3d.py
index e5a176d..3f63456 100644
--- a/apps/text_to_3d.py
+++ b/apps/text_to_3d.py
@@ -69,9 +69,7 @@
๐ Create 3D assets from text descriptions for a wide range of geometry and styles.
- """.format(
- VERSION=VERSION
- ),
+ """.format(VERSION=VERSION),
elem_classes=["header"],
)
@@ -106,7 +104,7 @@
0, MAX_SEED, label="Seed", value=0, step=1
)
randomize_seed = gr.Checkbox(
- label="Randomize Seed", value=False
+ label="Randomize Seed", value=True
)
rmbg_tag = gr.Radio(
choices=["rembg", "rmbg14"],
@@ -159,7 +157,7 @@
label="Randomize Seed", value=False
)
project_delight = gr.Checkbox(
- label="Back-project Delight", value=True
+ label="Back-project Delight", value=False
)
gr.Markdown("Geo Structure Generation")
with gr.Row():
diff --git a/apps/visualize_asset.py b/apps/visualize_asset.py
index 233df4a..4161967 100644
--- a/apps/visualize_asset.py
+++ b/apps/visualize_asset.py
@@ -23,14 +23,18 @@
os.makedirs(gradio_tmp_dir, exist_ok=True)
os.environ["GRADIO_TEMP_DIR"] = gradio_tmp_dir
+import colorsys
import shutil
import uuid
import xml.etree.ElementTree as ET
+import zipfile
from pathlib import Path
import gradio as gr
+import numpy as np
import pandas as pd
-from app_style import custom_theme, lighting_css
+import trimesh
+from app_style import custom_theme
from embodied_gen.utils.tags import VERSION
try:
@@ -49,25 +53,65 @@
# --- Configuration & Data Loading ---
RUNNING_MODE = "local" # local or hf_remote
CSV_FILE = "dataset_index.csv"
+HF_REPO_ID = "HorizonRobotics/EmbodiedGenData"
+HF_LOCAL_DIR = "EmbodiedGenData"
+CAMERA_ZOOM = 3.2
+
+# Compatible with huggingface space zero GPU
+import spaces
+from huggingface_hub import snapshot_download
+
+
+@spaces.GPU
+def fake_gpu_init():
+ pass
+
+
+fake_gpu_init()
if RUNNING_MODE == "local":
- DATA_ROOT = "/horizon-bucket/robot_lab/datasets/embodiedgen/assets"
+ DATA_ROOT = "/horizon-bucket/robot_lab/datasets/embodiedgen/assets_v2"
elif RUNNING_MODE == "hf_remote":
- from huggingface_hub import snapshot_download
-
+ # Only fetch the index and preview videos up front; per-asset mesh/urdf/
+ # usd/mjcf files are pulled lazily on demand (see `ensure_hf_files`).
snapshot_download(
- repo_id="HorizonRobotics/EmbodiedGenData",
+ repo_id=HF_REPO_ID,
repo_type="dataset",
- allow_patterns=f"dataset/**",
- local_dir="EmbodiedGenData",
+ allow_patterns=[
+ f"dataset/{CSV_FILE}",
+ "dataset/**/*.mp4",
+ ],
+ local_dir=HF_LOCAL_DIR,
local_dir_use_symlinks=False,
)
- DATA_ROOT = "EmbodiedGenData/dataset"
+ DATA_ROOT = os.path.join(HF_LOCAL_DIR, "dataset")
else:
raise ValueError(
f"Unknown RUNNING_MODE: {RUNNING_MODE}, must be 'local' or 'hf_remote'."
)
+
+def ensure_hf_files(rel_patterns: list[str]) -> None:
+ """Lazily download files for one asset in hf_remote mode.
+
+ Args:
+ rel_patterns: Glob patterns relative to ``DATA_ROOT`` (the repo's
+ ``dataset/`` folder), e.g. ``"cat/sub/uid/mesh/**"``.
+
+ No-op when running locally, where all files already exist on disk.
+ Progress toasts are emitted by the calling action handlers.
+ """
+ if RUNNING_MODE != "hf_remote":
+ return
+ snapshot_download(
+ repo_id=HF_REPO_ID,
+ repo_type="dataset",
+ allow_patterns=[f"dataset/{p}" for p in rel_patterns],
+ local_dir=HF_LOCAL_DIR,
+ local_dir_use_symlinks=False,
+ )
+
+
csv_path = os.path.join(DATA_ROOT, CSV_FILE)
df = pd.read_csv(csv_path)
TMP_DIR = os.path.join(
@@ -120,6 +164,82 @@ def _unique_path(
return str(dst)
+def _bounding_radius(mesh: trimesh.Trimesh | trimesh.Scene) -> float:
+ """Radius of the mesh/scene bounding sphere (half the diagonal)."""
+ lo, hi = mesh.bounds
+ return float(np.linalg.norm(np.asarray(hi) - np.asarray(lo)) / 2)
+
+
+def _camera_position(radius: float | None) -> tuple:
+ """Initial camera `(alpha, beta, radius)`; only push the distance."""
+ if not radius or radius <= 0:
+ return (None, None, None)
+ return (None, None, CAMERA_ZOOM * radius)
+
+
+def _visual_radius(visual_path: str | None) -> float | None:
+ if not visual_path:
+ return None
+ return _bounding_radius(trimesh.load(visual_path, process=False))
+
+
+def _pastel_color(i: int) -> list[int]:
+ """Distinct pastel RGBA for the i-th convex piece (golden-ratio hues)."""
+ h = (i * 0.6180339887498949) % 1.0
+ r, g, b = colorsys.hsv_to_rgb(h, 0.45, 0.98)
+ return [int(r * 255), int(g * 255), int(b * 255), 255]
+
+
+def _colored_collision_path(
+ collision_path: str | None, session_hash: str
+) -> tuple[str | None, float | None]:
+ """Export the collision mesh as a GLB with one color per convex piece.
+
+ The convex-decomposition pieces are stored as separate objects inside the
+ single collision ``.obj``; they are recovered via connected-component split
+ and each is tinted a distinct pastel color for visualization.
+
+ Returns ``(glb_path, bounding_radius)`` for camera framing.
+ """
+ if not collision_path:
+ return None, None
+
+ tmp_root = (
+ Path(os.environ.get("GRADIO_TEMP_DIR", "/tmp"))
+ / "model3d-cache"
+ / session_hash
+ )
+ tmp_root.mkdir(parents=True, exist_ok=True)
+
+ # rolling cleanup for same kind
+ prev = _prev_temp.get(session_hash, {})
+ old = prev.get("collision")
+ if old and old.exists():
+ old.unlink()
+
+ loaded = trimesh.load(collision_path, process=False)
+ if isinstance(loaded, trimesh.Scene):
+ parts = list(loaded.geometry.values())
+ else:
+ parts = loaded.split(only_watertight=False)
+ if len(parts) == 0:
+ parts = [loaded]
+
+ scene = trimesh.Scene()
+ for i, part in enumerate(parts):
+ part.visual = trimesh.visual.ColorVisuals(
+ mesh=part, vertex_colors=_pastel_color(i)
+ )
+ scene.add_geometry(part)
+
+ dst = tmp_root / f"collision-{uuid.uuid4().hex}.glb"
+ scene.export(str(dst))
+
+ prev["collision"] = dst
+ _prev_temp[session_hash] = prev
+ return str(dst), _bounding_radius(scene)
+
+
# --- Helper Functions (data filtering) ---
def get_primary_categories():
return sorted(df["primary_category"].dropna().unique())
@@ -345,6 +465,20 @@ def show_asset_from_gallery(
)
row = subset.iloc[index]
+
+ # In hf_remote mode, pull only what the two mesh viewers need: the visual
+ # GLB + collision OBJ (mesh/), plus the tiny URDF used to locate them and
+ # show metadata. USD/MJCF/USD-textures are fetched only on button click.
+ gr.Info("โณ Loading 3D model, please wait...")
+ rel_dir = row["asset_dir"]
+ urdf_name = os.path.basename(row["urdf_path"])
+ ensure_hf_files(
+ [
+ f"{rel_dir}/{urdf_name}",
+ f"{rel_dir}/mesh/**",
+ ]
+ )
+
visual_path, collision_path, desc = _extract_mesh_paths(row)
urdf_path = os.path.join(DATA_ROOT, row["urdf_path"])
@@ -416,41 +550,217 @@ def render_meshes(
if visual_path
else None
)
- collision_unique = (
- _unique_path(collision_path, session_hash, "collision")
- if collision_path
- else None
- )
+ visual_cam = _camera_position(_visual_radius(visual_path))
+
+ if collision_path:
+ collision_unique, collision_r = _colored_collision_path(
+ collision_path, session_hash
+ )
+ else:
+ collision_unique, collision_r = None, None
+ collision_cam = _camera_position(collision_r)
+
+ if visual_unique or collision_unique:
+ gr.Info("โ
3D model loaded.")
if switch_viewer:
yield (
- gr.update(value=visual_unique),
+ gr.update(value=visual_unique, camera_position=visual_cam),
gr.update(value=None, visible=False),
- gr.update(value=collision_unique, visible=True),
+ gr.update(
+ value=collision_unique,
+ visible=True,
+ camera_position=collision_cam,
+ ),
False,
)
else:
yield (
- gr.update(value=visual_unique),
- gr.update(value=collision_unique, visible=True),
+ gr.update(value=visual_unique, camera_position=visual_cam),
+ gr.update(
+ value=collision_unique,
+ visible=True,
+ camera_position=collision_cam,
+ ),
gr.update(value=None, visible=False),
True,
)
-def create_asset_zip(asset_dir: str, req: gr.Request):
+def _rel_dir(asset_dir: str) -> str:
+ """Repo-relative asset dir (path under DATA_ROOT), for HF glob patterns."""
+ return os.path.relpath(asset_dir, DATA_ROOT)
+
+
+def _find_urdf_stem(asset_dir: str) -> str | None:
+ for f in os.listdir(asset_dir):
+ if f.lower().endswith(".urdf"):
+ return os.path.splitext(f)[0]
+ return None
+
+
+def _zip_items(
+ zip_path: str,
+ items: list[tuple[str, str]],
+ exclude_suffixes: tuple[str, ...] = (),
+) -> str:
+ """Write files/dirs into a zip.
+
+ Args:
+ zip_path: Output archive path.
+ items: ``(src_abspath, arcname)`` pairs. Directories are walked and
+ stored under ``arcname``.
+ exclude_suffixes: Lowercase filename suffixes to skip when walking
+ directories, e.g. ``(".glb",)``.
+ """
+ with zipfile.ZipFile(zip_path, "w", zipfile.ZIP_DEFLATED) as zf:
+ for src, arcname in items:
+ if os.path.isdir(src):
+ for root_d, _, files in os.walk(src):
+ for fn in files:
+ if exclude_suffixes and fn.lower().endswith(
+ exclude_suffixes
+ ):
+ continue
+ fp = os.path.join(root_d, fn)
+ rel = os.path.relpath(fp, src)
+ zf.write(fp, os.path.join(arcname, rel))
+ elif os.path.isfile(src):
+ zf.write(src, arcname)
+ return zip_path
+
+
+def _fmt_zip_path(
+ asset_dir: str, fmt: str, req: gr.Request
+) -> tuple[str, str, str]:
+ """Build the per-format output zip path named ``
+
+## What can it do?
+
+
+
+!!! note
+ DIPO is maintained in its own repository. Please refer to the [DIPO codebase](https://github.com/RQ-Wu/DIPO) for installation and usage.
diff --git a/docs/documentation/tutorials/digital_twin.md b/docs/documentation/tutorials/digital_twin.md
new file mode 100644
index 0000000..bf2eafd
--- /dev/null
+++ b/docs/documentation/tutorials/digital_twin.md
@@ -0,0 +1,11 @@
+# ๐ง Real-to-Sim Digital Twin Creation
+
+Recreate real-world scenes in simulation: capture a photo of each real object, generate its **physically plausible 3D twin** with [Image-to-3D](image_to_3d.md), and compose the twins into a simulator scene ([Any Simulators](any_simulators.md)).
+
+```sh
+img3d-cli --image_path path/to/real_capture.jpg --output_root outputs/digital_twin
+```
+
+Each asset keeps metric scale, collision geometry, and physical properties inferred from the image, so the resulting scene behaves consistently with its real counterpart โ below, an EmbodiedGen digital twin replayed in MuJoCo.
+
+
diff --git a/docs/tutorials/image_to_3d.md b/docs/documentation/tutorials/image_to_3d.md
similarity index 72%
rename from docs/tutorials/image_to_3d.md
rename to docs/documentation/tutorials/image_to_3d.md
index 0081d6b..cd4329b 100644
--- a/docs/tutorials/image_to_3d.md
+++ b/docs/documentation/tutorials/image_to_3d.md
@@ -1,11 +1,17 @@
-# ๐ผ๏ธ Image-to-3D: Physically Plausible 3D Asset Generation
+# ๐ผ๏ธ Image-to-3D โ Physically Plausible 3D Asset Generation
Generate **physically plausible 3D assets** from a single input image, supporting **digital twin** and **simulation environments**.
---
## โก Command-Line Usage
-Support the use of [SAM3D](https://github.com/facebookresearch/sam-3d-objects) or [TRELLIS](https://github.com/microsoft/TRELLIS) as 3D generation model, modify `IMAGE3D_MODEL` in `embodied_gen/scripts/imageto3d.py` to switch model.
+Three 3D generation backends are supported:
+
+- [`SAM3D`](https://github.com/facebookresearch/sam-3d-objects) โ local model (default)
+- [`TRELLIS`](https://github.com/microsoft/TRELLIS) โ local model
+- `HUNYUAN3D` โ Tencent Hunyuan3D Pro cloud API (no local GPU model needed)
+
+Select the backend via `--image3d_model` (case-insensitive). Omit to use the default `SAM3D`.
```bash
img3d-cli --image_path apps/assets/example_image/sample_00.jpg \
@@ -13,6 +19,18 @@ apps/assets/example_image/sample_01.jpg \
--n_retry 2 --output_root outputs/imageto3d
```
+### Using the Hunyuan3D Cloud Backend
+
+Hunyuan3D Pro runs entirely on Tencent Cloud โ useful when you don't have a local GPU. It requires Tencent Cloud Hunyuan3D `SecretId` / `SecretKey` and network access to `ai3d.tencentcloudapi.com` and the COS download host.
+
+```bash
+export TENCENT_SECRET_ID='your-secret-id'
+export TENCENT_SECRET_KEY='your-secret-key'
+img3d-cli --image3d_model HUNYUAN3D \
+ --image_path apps/assets/example_image/sample_00.jpg \
+ --output_root outputs/imageto3d_hunyuan
+```
+
You will get the following results:
- 
-
+
-## [๐ง Real-to-Sim Digital Twin Creation](digital_twin.md)
+---
-
-
-> ***EmbodiedGen*** is a generative engine to create diverse and interactive 3D worlds composed of high-quality 3D assets(mesh & 3DGS) with plausible physics, leveraging generative AI to address the challenges of generalization in embodied intelligence related research.
-
----
diff --git a/docs/landing/README.md b/docs/landing/README.md
new file mode 100644
index 0000000..99b544d
--- /dev/null
+++ b/docs/landing/README.md
@@ -0,0 +1,55 @@
+# EmbodiedGen V2 โ Project Landing Page
+
+Standalone static site (`docs/landing/`). No build step, no framework, no dependencies โ
+open `index.html` over any static server.
+
+On GitHub Pages this is published at the site **root** (`https://horizonrobotics.github.io/EmbodiedGen/`);
+the MkDocs documentation is published under **`/docs/`**. See `.github/workflows/deploy_docs.yml`.
+
+## Local preview
+```bash
+cd docs/landing
+python3 -m http.server 8123
+# open http://localhost:8123
+```
+Fonts load from Google Fonts (needs internet); a system fallback stack is configured.
+The repo `.gitignore` ignores `*.html` / `*.mp4` / `*.json` globally, but re-includes them under
+`docs/landing/**` (see the negation block at the end of `.gitignore`), so these site files are tracked normally.
+
+## Files
+| File | Responsibility |
+|------|----------------|
+| `index.html` | Structure + all section containers |
+| `style.css` | Design system (CSS vars, type, grid, components, responsive) โ single source of truth |
+| `script.js` | Scroll-spy, reveal-on-scroll, lazy video play/pause, asset 3D viewer, vibe sessions, count-up, BibTeX copy |
+| `tools/cut_videos_from_master.sh` | Section videos cut from the master demo video |
+| `tools/build_assets.sh` | Figures (JPG), brand logo, sim-ready models + `assets.json`, vibe clips, affordance |
+| `tools/build_affordance.py` | Part-seg GLB + baked grippers + `affordance.json` |
+| `favicon.svg` + `favicon-{32,180}.png` | Browser tab icon โ brand-blue 3D cube (SVG primary, PNG fallback / apple-touch). Static brand asset. |
+
+## Assets โ how they're produced (idempotent)
+Two scripts own disjoint sets of outputs under `assets/`:
+
+- **`tools/cut_videos_from_master.sh`** โ the five section videos + posters
+ (`hero_bg`, `scenes_gen`, `worlds`, `multi_sim`, `closed_loop`), cut as timestamped
+ segments from the master demo video. The master `.mov` is **not** kept in the repo.
+- **`tools/build_assets.sh`** โ everything else: figure JPGs (โค1 MB), `logo.png`,
+ the GLB models + `assets.json`, the six `scene_*` vibe clips, and affordance data.
+ Reads READ-ONLY sources from `outputs/embodiedgenv2/{paper/figure,video}` and the asset buckets.
+
+## Performance notes (production)
+- Section videos use `preload="none"` + `data-autoplay`: fetched and played only when
+ scrolled into view, paused when offscreen (see `script.js`).
+- The `model-viewer` library (~1 MB) and the first GLB load lazily โ warmed right after
+ page load (idle) or when the Assets section nears, whichever comes first.
+- Hero streams progressively (`+faststart` + poster). Large figures are JPG inside
+ collapsed `
+
+ EmbodiedGenV2
+An Agentic, Simulation-Ready 3D World Engine
for Embodied AI
+ + From intent to executable 3D worlds + +
+ + +1Horizon Robotics ยท 2WuwenAI
+ + +
Paper
Code
Video
HuggingFaceGenerative simulation infrastructure for embodied AI
++ From text, images, and natural-language dialogue to simulation-ready 3D worlds โ + assets, interactive scenes, and large-scale environments, deployable across every major simulator. +
+Generate Sim-Ready Assets
Metric geometry, collision proxies, and physical properties in one package.
โ + +Scale Large-Scale Scenes
Multi-room, navigable environments with editable furniture instances.
โ + +Compose Task-Driven Worlds
Natural-language tasks become physically stable, simulator-ready layouts.
โ + +Edit Vibe Coding
Refine a deployable world state through natural-language dialogue.
โ + +Export Cross-Simulator
One standardized layout, six engines, zero manual adaptation.
โ + +Train Closed-Loop Learning
Generated environments as online RL worlds for robot policies.
โ +View the system overview figure
+
+ Sim-Ready 3D Assets
++ Real generated assets โ drag to rotate. Each carries metric geometry, + a collision proxy, physical properties, and + cross-simulator interfaces. +
+Pipeline text or a single image โ a sim-ready asset
+-
+
- 01
Input Preparation
Text โ image, or segment the foreground โ even under occlusion.
+ - 02
3D Generation & QC
Pluggable image-to-3D, gated by semantic, geometric & aesthetic checks.
โป auto-retry
+ - 03
Geometry & Texture
Mesh repair, simplification, convex decomposition, baked texture.
+ - 04
Physical Recovery
A VLM infers real-world scale, mass and friction for metric rescaling.
+ - 05
Cross-Format Export
One URDF intermediate โ URDF ยท USD ยท MJCF for every major engine.
+
View the full pipeline figure
+
+ Beyond rigid bodies โ soft-body simulation
++ The same generate-and-export path reaches soft bodies: 12 text-conditioned garments deploy + as deformable meshes in Genesis, with no manual preparation. +
+ +12 generated garments ยท per-vertex displacement ยท Genesis
+Large-Scale Scenes
++ Beyond tabletops โ multi-room, navigable, instance-editable houses, generated as + sim-ready backgrounds at a controllable complexity tier. +
+View the large-scale generation pipeline
+
+ Task-Driven Worlds
++ From a natural-language task, EmbodiedGen parses a Scene Graph and composes a + physically stable, directly loadable layout โ then settles it under gravity in simulation. +
+View the scene-generation pipeline
+
+ Vibe Coding for Sim-Ready 3D Worlds
++ Build and edit worlds through natural-language dialogue. Each instruction is a + bounded, physics-validated skill call that preserves a deployable, + sim-ready world state โ add, remove, replace, refine. +
+One World, Every Simulator
++ One standardized layout โ no manual adaptation. The same generated scene loads + with consistent geometry, collision, textures, and physical metadata across six mainstream + physics simulators. +
+Closed-Loop Robot Learning
++ Generated worlds are not just viewable โ they are online training environments. + Policies trained purely in EmbodiedGen-generated worlds transfer to real robots. +
++ Results from a companion sim-to-real RL study; policies trained in EmbodiedGen-generated worlds. +
+View the policy-learning & deployment figure
+
+ BibTeX
+@article{wang2026embodiedgenv2,
+ title = {EmbodiedGen V2: An Agentic, Simulation-Ready 3D World
+ Engine for Embodied AI},
+ author = {Xinjie Wang and Liu Liu and Taojun Ding and Andrew Choi and
+ Chaodong Huang and Mengao Zhao and Ziang Li and Jackson Jiang and
+ Chunlei Yu and Shengxiang Liu and Wei Xu and Zhizhong Su},
+ journal = {arXiv preprint arXiv:2506.10600},
+ year = {2026}
+}
+ @misc{wang2025embodiedgengenerative3dworld,
+ title = {EmbodiedGen: Towards a Generative 3D World Engine for
+ Embodied Intelligence},
+ author = {Xinjie Wang and Liu Liu and Yu Cao and Ruiqi Wu and
+ Wenkang Qin and Dehui Wang and Wei Sui and Zhizhong Su},
+ year = {2025},
+ eprint = {2506.10600},
+ archivePrefix = {arXiv},
+ primaryClass = {cs.RO},
+ url = {https://arxiv.org/abs/2506.10600}
+}
+