EraseLoRA: MLLM-Driven Foreground Exclusion and Background Subtype Aggregation for Dataset-Free Object Removal

⭐ ECCV 2026

Sanghyun Jo^1,2*†, Donghwan Lee^2*, Eunji Jung^2*, Seong Je Oh², Kyungsu Kim^2†

¹OGQ    ²Seoul National University
^*Equal contribution    ^†Corresponding authors

---

This is the code release. It contains the test-time adaptation core (BRSA), the SD3.5 inpainting pipeline, single-image and dataset demos, and the paper's OpenImages V7 evaluation subset (190 images). The MLLM foreground-exclusion stage (BFE) is provided as precomputed background tags. Please ⭐ star and 👁️ watch for updates.

---

≥ 23%	≈ ½	0	MLLM
More Faithful Background	Foreground Re-generation	Training Data	Background-aware Reasoning
vs. prior dataset-free	nearly halved	fully dataset-free	not attention surgery

💡 TL;DR

Object removal is not hole-filling. It must prevent the masked target from reappearing and reconstruct the occluded background with structural and contextual fidelity. Prior dataset-free methods perform attention surgery, redirecting the diffusion model's self-attention away from the mask, yet they carry no notion of what the background actually is. EraseLoRA replaces attention surgery with explicit background-aware reasoning: a multimodal large-language model (MLLM) separates target foreground, non-target foregrounds, and clean background, then test-time optimization aggregates background subtypes as complementary pieces, all with no training data.

📢 News

• [2026.07] 💻 Code is released: BRSA test-time adaptation on SD3.5-M (model-agnostic to SD3.5-L / SDXL-Inpainting / FLUX.1-Fill), with single-image and dataset demos and the paper's 190-image OpenImages V7 evaluation subset. The BFE stage ships as precomputed background tags.
• [2026.06] 🚀 EraseLoRA is publicly released: the project page and arXiv preprint are now available.

🎯 Motivation

Recent dataset-free approaches manipulate the diffusion model's internal self-attention so it stops referencing the masked region, yet they fail in two ways rooted in the same cause: the absence of explicit background-aware reasoning.

Failure 1 · Non-Target Re-generation. Treating the masked region as the sole foreground, prior methods misinterpret non-target objects outside the mask as background and regenerate them, producing unwanted content.

Failure 2 · Background Blurring. Applying uniform attention constraints without distinguishing diverse background subtypes blurs textures and misaligns structure at the boundaries between disparate regions.

🔍 Overview

EraseLoRA is a dataset-free framework that replaces attention surgery with background-aware reasoning and test-time adaptation. It runs in two stages.

1. Background-aware Foreground Exclusion (BFE) leverages a multimodal large-language model to separate the target foreground to remove, the non-target foregrounds that must be preserved, and the clean background available for reconstruction, all from a single image-mask pair. By explicitly excluding non-target foregrounds, BFE stops the diffusion model from mistaking other objects for background and regenerating them.

2. Background-aware Reconstruction with Subtype Aggregation (BRSA) performs test-time optimization that treats the inferred background subtypes as complementary pieces, enforcing their consistent integration through reconstruction and alignment objectives. This preserves local detail within each subtype and coherent structure across subtype boundaries, with no explicit attention intervention.

As a model-agnostic plug-in applicable to diverse diffusion backbones, EraseLoRA requires no training data and adds no auxiliary networks to the backbone.

How the MLLM Reasons About the Scene

Unlike prior work that uses MLLMs to reason over the visible scene for generation, EraseLoRA prompts the MLLM to infer which foreground and background tags lie behind the target, then extracts their masks.

📊 Main Results

EraseLoRA reconstructs backgrounds at least 23% more faithful to the original scene than previous dataset-free methods, while nearly halving unwanted foreground re-generation. It surpasses all dataset-driven approaches on both axes, despite using no training data, no auxiliary networks, and no attention surgery.

Quantitative Comparison (OpenImages V7 and RORD)

EraseLoRA leads all dataset-free methods on background similarity (BG Sim.) and foreground similarity (FG Sim.), and surpasses dataset-driven methods with no training data. Higher BG Sim. and lower FG Sim. are better.

Qualitative Comparison

Without any paired data, EraseLoRA avoids unwanted background changes and non-target regeneration, matching or surpassing dataset-driven methods.

Faithful Removal Without Regenerating Non-Target Objects

Prior dataset-free methods treat only the masked region as foreground and regenerate non-target objects. EraseLoRA identifies and excludes non-target foregrounds, then reconstructs the masked region from diverse background subtypes.

Conceptual Comparison

EraseLoRA is the only dataset-free method that identifies non-target foregrounds, leverages multiple background subtypes, and stays model-agnostic.

🔬 Analysis & Ablations

Effect of Foreground Exclusion (BFE)

Adding BFE to prior dataset-free methods suppresses regeneration of non-target objects, and EraseLoRA stays robust in occlusion cases.

Loss Components in BRSA

Reconstruction preserves structure and the puzzle loss completes the mask. Using both objectives together yields a coherent background reconstruction inside the masked region.

Ablation: Foreground Exclusion and Loss Components

(Left) Adding BFE to prior methods improves both background and foreground similarity. (Right) Combining the reconstruction and puzzle losses in BRSA yields the most faithful reconstruction.

Applicability Across MLLMs and Tag2Mask Models

EraseLoRA remains effective across a range of MLLMs and Tag2Mask segmentation models, confirming it is not tied to a specific backbone.

🎨 Applications

Interactive Control

An interactive interface lets users generate customized non-target foreground masks for BFE from manual points, giving fine control over what is preserved.

📦 What's in This Release

EraseLoRA has two stages. Stage 1 (BFE) uses a multimodal LLM to name the clean background and the non-target foregrounds; its outputs are shipped as precomputed background tags and an optional non-target mask. Stage 2 (BRSA) is the test-time procedure this code runs.

• Test-time adaptation core (tta_sd35_pipeline.py). SD35InpaintingPipeline implements BRSA: the background reconstruction and attention puzzle losses, the LoRA test-time optimization, and the mask-anchored inpainting inference, all on a frozen SD3.5 backbone.
• Pipeline builder (core/diffusion/). build_pipeline('SD3.5-M') loads the diffusion backbone. EraseLoRA is model-agnostic, so SD3.5-L, SDXL-Inpainting, and FLUX.1-Fill are drop-in alternatives.
• Demos. demo_tta_sd35_single.py (single image) and demo_tta_sd35_dataset.py (a folder of images) run TTA and inference end to end.
• Datasets. The full 190-image OpenImages V7 subset used in the paper (datasets/OPIV7/), plus a 4-image sample (datasets/samples/), each with masks and precomputed background tags.

🗂️ Repository Structure

EraseLoRA/
├── core/
│   └── diffusion/
│       ├── __init__.py          # build_pipeline: pretrained diffusion backbones
│       └── utils.py             # prompt assembly from background tags
├── tta_sd35_pipeline.py         # BRSA core: SD35InpaintingPipeline + TTAInpaintingDataset
├── tta_helper.py                # I/O utils (padding, cropping, loading, tag parsing)
├── demo_tta_sd35_single.py      # single-image TTA / inference
├── demo_tta_sd35_dataset.py     # batch TTA + inference over a dataset folder
├── datasets/
│   ├── OPIV7/                   # 190-image OpenImages V7 subset (paper eval set) + MLLM CoT
│   └── samples/                 # 4-image sample set (img / mask / mask2 / tags.json)
├── requirements.txt
└── DATASETS.md

Where to look first: tta_sd35_pipeline.py is the whole method; SD35InpaintingPipeline.train_tta and inpaint_inference are Stage 2.

🛠️ Installation

python3 -m venv venv && source ./venv/bin/activate     # Windows: .\venv\Scripts\activate

# 1) PyTorch matched to your CUDA (tested: torch 2.7.1 / CUDA 11.8)
python3 -m pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 \
    --index-url https://download.pytorch.org/whl/cu118

# 2) The rest
python3 -m pip install -r requirements.txt

Requirements: Python ≥ 3.10, PyTorch ≥ 2.5, CUDA ≥ 11.8, and a single GPU (SD3.5-M test-time adaptation fits on a 24 GB card). SD3.5 is gated on the Hugging Face Hub: accept the license on the model page, then export your token so the weights can download.

export HF_TOKEN=hf_xxx     # your own token; never commit it

🚀 Quick Start

1. Dataset TTA and inference (recommended)

Runs Stage 2 on every image in a folder. With no arguments it uses the shipped sample set.

CUDA_VISIBLE_DEVICES=0 python3 demo_tta_sd35_dataset.py \
    --data_dir ./datasets/samples \
    --save_dir ./results/samples \
    --iters 500 --rank 32 --lr 1e-4 \
    --lambda_bg_recon 1.0 --lambda_attention_puzzle 0.2

2. Single image: TTA then inference

CUDA_VISIBLE_DEVICES=0 python3 demo_tta_sd35_single.py \
    --mode tta_infer \
    --image_path ./datasets/samples/img/image_001.png \
    --mask_path ./datasets/samples/mask/mask_001_HR.png \
    --mask2_path ./datasets/samples/mask2/mask2_001.png \
    --prompt "A photo of green grass and white lines." \
    --tags "green grass,white lines" \
    --save_dir ./results/single/001 \
    --iters 500 --rank 32 --lr 1e-4 \
    --lambda_bg_recon 1.0 --lambda_attention_puzzle 0.2

--mode tta runs the test-time optimization only, and --mode infer reuses an existing LoRA checkpoint (--lora_path) for inference (mask2 is not needed at inference).

Outputs

Each image writes to its save_dir:

• result_<id>.jpg: the object-removed image (mask region reconstructed)
• cross_attention_<id>.jpg: per-tag cross-attention visualization
• checkpoint-<step>.safetensors: the test-time LoRA weights
• loss_history.json: the training loss curves

📁 Datasets

EraseLoRA needs no training data; datasets are used only for evaluation. This repository ships the full OpenImages V7 subset (190 images) selected in the paper under datasets/OPIV7/, plus a tiny 4-image sample under datasets/samples/. RORD (MIT-licensed, distributed via AI Hub) is not shipped; download it from its official source. See DATASETS.md for sources, licenses, and layout.

🙏 Acknowledgements

EraseLoRA builds on the Stable Diffusion 3.5 family from Stability AI, and on the Hugging Face Diffusers, PEFT, and Accelerate libraries. The RORD benchmark is by Sagong et al. and evaluation images come from OpenImages V7. We thank the authors of every dataset and foundation model we build on.

📖 Citation

If you find EraseLoRA useful in your research, please consider citing:

@inproceedings{jo2026eraselora,
  title     = {{EraseLoRA}: {MLLM}-Driven Foreground Exclusion and Background Subtype Aggregation for Dataset-Free Object Removal},
  author    = {Jo, Sanghyun and Lee, Donghwan and Jung, Eunji and Oh, Seong Je and Kim, Kyungsu},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

📜 License

Our first-party code is released under CC BY-NC 4.0 (Creative Commons Attribution-NonCommercial 4.0): free for non-commercial academic research and education with attribution. Any commercial use requires prior written permission. For commercial licensing, please contact both:

• Sanghyun Jo: [email protected]
• Kyungsu Kim: [email protected]

Pretrained diffusion backbones and datasets follow their original licenses (see the model pages and DATASETS.md).