ROSA: Reconstructing Object Shape and Appearance Textures by Adaptive Detail Transfer

Julian Kaltheuner¹  ·  Patrick Stotko¹  ·  Reinhard Klein¹

¹University of Bonn  

WACV 2025

Paper (arxiv)   |   Project Page

Code for reproducing the results will be released soon.

Abstract

Reconstructing an object's shape and appearance in terms of a mesh textured by a spatially-varying bidirectional reflectance distribution function (SVBRDF) from a limited set of images captured under collocated light is an ill-posed problem. Previous state-of-the-art approaches either aim to reconstruct the appearance directly on the geometry or additionally use texture normals as part of the appearance features. However, this requires detailed but inefficiently large meshes, that would have to be simplified in a post-processing step, or suffers from well-known limitations of normal maps such as missing shadows or incorrect silhouettes. Another limiting factor is the fixed and typically low resolution of the texture estimation resulting in loss of important surface details. To overcome these problems, we present ROSA, an inverse rendering method that directly optimizes mesh geometry with spatially adaptive mesh resolution solely based on the image data. In particular, we refine the mesh and locally condition the surface smoothness based on the estimated normal texture and mesh curvature. In addition, we enable the reconstruction of fine appearance details in high-resolution textures through a pioneering tile-based method that operates on a single pre-trained decoder network but is not limited by the network output resolution.

Citation

If you use the code for your own research, please cite our work as

@misc{kaltheuner2025rosareconstructingobjectshape,
      title={ROSA: Reconstructing Object Shape and Appearance Textures by Adaptive Detail Transfer}, 
      author={Julian Kaltheuner and Patrick Stotko and Reinhard Klein},
      year={2025},
      eprint={2501.18595},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2501.18595}, 
}

Install

git clone https://github.com/vc-bonn/rosa.git
cd rosa

git submodule update --init

conda create -n ROSA python=3.10
conda activate ROSA

pip install torch==2.12.0+cu126 torchvision==0.27.0+cu126 \
  --index-url https://download.pytorch.org/whl/cu126

pip install \
  setuptools==70.2.0 \
  wheel==0.47.0 \
  ninja==1.13.0 \
  torchhull==0.3.0 \
  imageio==2.37.3 \
  open3d==0.19.0 \
  trimesh==4.12.2 \
  tensorboard==2.20.0

pip install git+https://github.com/NVlabs/nvdiffrast.git@253ac4fcea7de5f396371124af597e6cc957bfae --no-build-isolation
pip install --no-build-isolation "git+https://github.com/facebookresearch/pytorch3d.git@7f8a8a142fa1383a7895ed01dd7082105f9a81fb"
pip install bpy==3.6.0 --extra-index-url https://download.blender.org/pypi/
pip install git+https://github.com/JuKalt/cholespy_multiGPU.git@605cd3f39ada37bf78ee4fe2cfd65674b0eb7d43
pip install git+https://github.com/JuKalt/large-steps-pytorch-multiGPU.git@6014f3930acedd173b4d050e874a1777a2c9665a

Example data

The synthetic example code samples meshes from data/objs and SVBRDF textures from data/svbrdf, as configured in configs/io/ioSynthetic.json.

Create the expected folders:

mkdir -p data/objs data/svbrdf

Stanford Armadillo mesh

Download the Stanford Armadillo model from the Stanford 3D Scanning Repository. The direct download is Armadillo.ply.gz.

The synthetic loader expects .obj meshes, so decompress and convert the Armadillo mesh to OBJ, then place it in data/objs, for example as:

data/objs/armadillo.obj

Deschaintre SVBRDF materials

Download SVBRDF materials from the dataset released with Single-Image SVBRDF Capture with a Rendering-Aware Deep Network by Deschaintre et al. The dataset link on the project page is DeepMaterialsData.zip.

Place example SVBRDF .png files in:

data/svbrdf/

The synthetic loader expects each SVBRDF image to have width 5 * height; it uses the diffuse, roughness, and specular map tiles from that packed texture.

Run

After installation and after placing at least one mesh, one SVBRDF texture, and the decoder checkpoint in the paths configured by configs/io/ioSynthetic.json, start a synthetic reconstruction with:

conda activate ROSA
python Main.py --device 0 \
  --ioConfig configs/io/ioSynthetic.json \
  --methodConfig configs/method/config.json

Main.py requires a CUDA-capable GPU. The default device in the script is 0, so set --device explicitly unless you want to run on cuda:0.

Command-line arguments

• --device: CUDA device index to use, for example 0 for cuda:0.
• --seed: Random seed for Python, NumPy, and PyTorch. Defaults to 1.
• --ioConfig: Path to the IO/data configuration. Defaults to configs/io/ioSynthetic.json.
• --methodConfig: Path to the optimization configuration. Defaults to configs/method/config.json.

IO configuration

configs/io/ioSynthetic.json controls the data, checkpoint, render setup, and output location:

• paths.svbrdf: Folder containing packed SVBRDF .png files.
• paths.meshes: Folder containing input .obj meshes.
• paths.decoder: Pre-trained texture decoder checkpoint, expected at data/ckpts/decoderSmallFC.pt by default.
• paths.out_path: Base output folder. Each processed sample is written to a numbered subfolder such as outs/0000.
• images.num_train_cameras: Number of synthetic training views rendered for the optimization.
• images.num_test_cameras: Number of synthetic held-out test views generated by the dataset.
• images.resolution: Render resolution in pixels.
• images.light_strength: Collocated light strength used by the renderer.

Method configuration

configs/method/config.json controls initialization and optimization:

• initialization.number_verts_target: Target vertex count for the initial mesh.
• optimization.S: Geometry optimizer scale parameter.
• optimization.epochs: Number of optimization epochs.
• optimization.loss_weights.image: Weight for the image reconstruction loss.
• optimization.loss_weights.shape: Weight for the mask/shape loss.
• optimization.loss_weights.normal: Weight for the normal-guided geometry loss.

For a quick smoke test, reduce optimization.epochs, images.num_train_cameras, and images.resolution in copied config files and pass those copies through --ioConfig and --methodConfig.

Monitoring and outputs

The optimizer writes TensorBoard logs to the active output folder. To monitor a run:

tensorboard --logdir outs/0000

Each completed sample folder contains:

• mesh.obj: Exported optimized mesh with UV coordinates.
• mesh.pt: Serialized optimized mesh state.
• texture_decoder.pt: Serialized optimized texture decoder state.
• texture.pt: Optimized texture tensor and channel mapping.
• diffuse.png, normal.png, roughness.png, specular.png: Exported texture maps.
• events.out.tfevents.*: TensorBoard scalar and image summaries.

Acknowledgements

This research has been funded by the Federal Ministry of Education and Research under grant no. 01IS22094A WEST-AI, by the Federal Ministry of Education and Research of Germany as well as the state of North-Rhine Westphalia as part of the Lamarr-Institute for Machine Learning and Artificial Intelligence, INVIRTUO under grant no. PB22-063A and by the DFG project KL 1142/11-2.