MotionLCM — Real-Time Text-to-Motion Generation

Live demo, capability mapping, and motion composition for MotionLCM (ECCV 2024) — built end-to-end as a graduate research project.

---

Overview

This project takes the MotionLCM paper from research to a public, interactive product while extending it with novel capabilities. Working solo over 8 weeks at UNC Charlotte, I:

• Reproduced the paper's pipeline on Google Colab (T4 GPU)
• Validated all four model variants (VAE, MLD, MotionLCM, ControlNet)
• Benchmarked them against a 7-metric quality framework
• Deployed a live demo on Hugging Face Spaces
• Built a 60-prompt capability cookbook mapping what the model can and cannot do
• Implemented 3 motion composition methods that produce hybrid motions impossible in the training data

Try it now: huggingface.co/spaces/soumyanilain/MotionLCM

---

What is MotionLCM?

MotionLCM generates realistic 3D human motion from natural language descriptions in ~30 milliseconds on consumer GPUs — a 1929× speedup over the MLD baseline. It uses Latent Consistency Distillation to compress a 50-step DDIM diffusion process into 1-4 steps without quality loss.

Stage	Model	Parameters	Role
1	Sentence-T5-XL	~1.2B (frozen)	Text encoder
2	Motion VAE	17.6M	Encodes 22-joint motion to latent
3	MLD (teacher)	8.2M	DDIM denoiser, ~50 steps
4	MotionLCM	8.2M	LCD-distilled, 1-4 steps
5	ControlNet	~4M	Spatial joint trajectory control

---

Quick Start

Use the live demo (zero setup)

Visit huggingface.co/spaces/soumyanilain/MotionLCM, type a prompt, and click Generate. First-time visitors wait ~60s for the Space to wake up; subsequent generations are instant.

Two tabs available:

• Generate Motion — single prompt → 3D skeleton animation
• Compose Motions — combine two prompts via Sequential / Spatial Mix / Linear Blend

Run locally

git clone https://github.com/Dai-Wenxun/MotionLCM.git
cd MotionLCM
pip install -r requirements.txt

# Apply patches (see /patches/ in this repo)
python demo.py --cfg configs/motionlcm_t2m.yaml --example assets/example.txt

Notebooks in this repo:

• notebooks/Phase3_Control_Benchmark.ipynb — ControlNet + CLT + speed benchmark + evaluation metrics
• notebooks/PhaseA_PromptCookbook.ipynb — 60-prompt capability test
• notebooks/PhaseB_Composition.ipynb — motion composition (sequential, spatial mix, blend)

---

Project Phases

Phase 1-2: Setup & Pretrained Demos

Reproduced all four pretrained demos on Colab T4: VAE reconstruction (132 frames), MLD text-to-motion (~24s/sample), MotionLCM inference (~30ms/sample, 1929× speedup), and custom prompt generation.

Bugs patched along the way:

• demo.py Float16/Float32 dtype mismatch
• plot_script.py matplotlib ax.lines deprecation
• Skeleton out-of-bounds in axis limits
• Session persistence with Drive remount script

Phase 3: Control & Quantitative Analysis

ControlNet validation with 6 controllable joints (pelvis, feet, head, wrists) at densities 100%, 25%, 5/2/1 control points.

CLT (Consistency Latent Tuning) — inference-time gradient optimization on the initial latent.

Speed benchmark (5 runs each, mean ± std):

Method	Steps	Mean Time	Speedup
MLD (Teacher)	~50 DDIM	~24.0s	1.0×
MotionLCM	1-4	~30ms	1929×
ControlNet	1-4 + control	~50ms	~480×
CLT	1-4 + grad opt	~2-5s	~5-12×

Evaluation metrics (computed per motion): jitter, foot skating, bone length variance, average joint speed, root displacement, minimum joint distance, diversity (pairwise).

Phase 4: Hugging Face Spaces Deployment

Got the project from a 7+ GB Colab repo down to an 846MB / 1GB HF Space.

Deployment challenges solved:

Challenge	Solution
1 GB storage limit	Removed deps/ (7.3GB Sentence-T5), kept only 2 essential checkpoints (367MB)
Python 3.13 audioop crash	Pinned python_version: 3.12 in README YAML
Binary files rejected by git	Tracked *.ckpt, *.npy, *.pkl, *.png in .gitattributes for git-lfs
matplotlib ax.lines crash	Runtime patch: while ax.lines: ax.lines[0].remove()
Missing HumanML3D dataset	Created dummy dataset files + monkey-patched Text2MotionDataset.__init__
Sentence-T5 too large (6.8GB)	Changed config to download from HF Hub at runtime instead of local files
Prompt format mismatch	Discovered demo.py expects '{nframes} {text}' on one line

Phase A: Prompt Cookbook (extension)

Systematically tested 60 prompts across 8 categories to map what MotionLCM can generate.

Headline results:

• 60/60 prompts succeeded (100% success rate)
• Average quality: 4.78/5.0 (auto-rated on jitter, foot skating, skeleton consistency)
• 7 categories scored 4.5+ avg quality
• "Edge cases" (violin, juggling, swimming) scored lowest — expected, not in HumanML3D

This deliverable doubles as a capability documentation for end users — a kind of MotionLCM cookbook explaining which prompts work well.

Phase B: Motion Composition (extension)

Implemented three methods to combine two motions into a hybrid that doesn't exist in the training data:

1. Sequential — generates A and B separately, smoothly stitches them with a cosine-taper bridge. Produces "do A then B" motions like walks forward → bows politely.

2. Spatial Mix — takes upper-body joints (chest, head, arms) from one motion and lower-body joints (pelvis, legs, feet) from another. Produces motions like waves while walking — a combination not in HumanML3D.

3. Linear Blend — frame-wise interpolation at chosen alpha. Produces hybrid styles like 50% confident walk + 50% sad walk.

All three methods are deployed as a "Compose Motions" tab in the live demo.

Note: This is joint-space composition (post-decoding). True latent-space composition would require modifying the inference pipeline to expose the encoder. The joint-space approach is a reasonable simplification that produces compelling demonstrable results.

---

Architecture

Text Prompt
     │
     ▼
┌─────────────────┐
│  Sentence-T5    │  (1.2B, frozen, downloaded from HF Hub)
└────────┬────────┘
         │ text embeddings
         ▼
┌─────────────────┐
│   MotionLCM     │  (8.2M, 1-4 denoising steps)
│  Latent Denoiser│  ← optional: ControlNet branch (~4M)
└────────┬────────┘
         │ motion latent
         ▼
┌─────────────────┐
│   Motion VAE    │  (17.6M decoder)
│     Decoder     │
└────────┬────────┘
         │
         ▼
   3D Skeleton
   (22 joints × N frames @ 20fps)

---

Key Numbers

Metric	Value
Inference speed	~30 ms
Speedup vs baseline	1929×
Model size	8.2M parameters
Output resolution	22 joints × 20 fps
Live demo size	846 MB / 1 GB
Prompts tested in cookbook	60
Composition methods	3
Deployment issues solved	10+

---

Tech Stack

ML & Inference: PyTorch 2.10 · CUDA 12.8 · Sentence-T5-XL · HumanML3D 22-joint skeleton · Latent Consistency Distillation

Frontend: Gradio 6 · custom CSS · Helvetica Neue · responsive design

Backend: Python subprocess wrapper · monkey-patched dataset loader · matplotlib 3D + FuncAnimation · ffmpeg for MP4 export

Infrastructure: Hugging Face Spaces (CPU Basic, free tier) · Python 3.12 · git-lfs for checkpoints

---

Repository Structure

.
├── README.md                          # this file
├── notebooks/
│   ├── Phase3_Control_Benchmark.ipynb   # ControlNet + CLT + benchmark + evaluation
│   ├── PhaseA_PromptCookbook.ipynb      # 60-prompt capability test
│   └── PhaseB_Composition.ipynb         # 3-mode motion composition
├── hf_space/
│   ├── app.py                         # Gradio app deployed to HF Space
│   ├── README.md                      # HF Space metadata
│   └── requirements.txt
├── patches/
│   ├── demo_patch.py                  # dtype + dataset monkey-patch wrapper
│   └── plot_script_patch.txt          # matplotlib compatibility
├── results/
│   ├── speed_benchmark.png            # MLD vs MotionLCM vs ControlNet vs CLT
│   ├── quality_metrics_fair.png       # 7-metric evaluation chart
│   ├── prompt_cookbook_report.md      # capability map across 8 categories
│   └── composition_examples/          # ghost-overlay PNGs
└── presentations/
    ├── Phase34_Detailed.pptx          # 18-slide progress report
    └── Final_Report.md                # comprehensive write-up

---

Lessons Learned

1. Reproducing research is harder than the paper makes it look. The original code expected dependencies and dataset paths that didn't match modern Colab environments. Most of Phase 1 was patching things the paper never mentioned.

2. Deployment surfaces problems that local development hides. The HF Space deployment exposed 10+ issues — Python version mismatches, Git-LFS configuration, missing config flags, prompt format conventions — none of which appeared in local testing.

3. A working demo is worth 10 papers. People understand text → animated 3D skeleton instantly. They don't need to read about LCD distillation to be impressed.

4. Building extensions matters. The capability cookbook and composition tab took the project from "I reproduced a paper" to "I built something on top of it."

---

Author

Soumyanil Ain MS Computer Science · UNC Charlotte · Spring 2026

• 📧 [email protected]
• 🌐 github.com/soumyanilain
• 📊 Tableau Portfolio
• 🚀 Live Demo

---

Acknowledgements

• Original paper: Wenxun Dai, Ling-Hao Chen, Jingbo Wang, Jinpeng Liu, Bo Dai, Yansong Tang. MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Model. ECCV 2024.
• Original code: github.com/Dai-Wenxun/MotionLCM
• HumanML3D dataset: Guo et al., 2022.
• Sentence-T5: sentence-transformers/sentence-t5-large

This project is a graduate-level extension of MotionLCM. All extensions (capability cookbook, motion composition, deployment infrastructure) are original contributions.

---

License

MIT License. See LICENSE for details.

The original MotionLCM repository is licensed under MIT. This project follows the same license.