🏋️ Gym Exercise Classification with TFC-GCN

Skeleton-based action recognition using MediaPipe landmarks and a lightweight Graph Convolutional Network

---

Overview

This project classifies 18 gym exercises from short video clips (~1.5s) using skeleton keypoints — no raw pixels, no background, no clothing. The approach mirrors how a physical therapist assesses movement: by observing how the body moves, not what it looks like.

The pipeline extracts 33 body landmarks per frame with MediaPipe PoseLandmarker, computes three complementary skeleton features (J, B, C), and classifies them with TFC-GCN — a lightweight graph convolutional network with only 193K parameters.

Key finding: The split methodology matters critically. An initial split at clip level yielded 87.87% accuracy — later revealed as inflated by data leakage (91.5% of test clips had their source video in train). After correcting to a source-video-grouped split, the TFC-GCN v4 ensemble achieves 71.72% — a rigorous, leakage-free result.

---

Results Summary (Clean Split — No Data Leakage)

Model	Params	Test Acc	F1 (w.)	Notes
MLP (flat)	4,935,826	73.74%	73.90%	No structural bias; 25× more params
BiLSTM	868,580	68.69%	68.72%	Temporal bias only
TFC-GCN v1 (SGD)	193,664	18.86%	12.66%	SGD collapses on small datasets
TFC-GCN v2 (Adam)	193,664	70.03%	70.49%	Flip aug. destroys deadlift recall
TFC-GCN v3	193,664	69.36%	69.59%	Label smoothing
TFC-GCN v4	193,664	70.03%	69.67%	Class-weighted CE + CosineWarmRestart
TFC-GCN v4 Ensemble	3×193,664	71.72%	71.71%	3 seeds, soft voting
MLP-Compact	190,518	65.66%	66.58%	Parameter-matched baseline

Parameter efficiency: TFC-GCN v4 achieves 36.2% accuracy per 100K params vs. 1.5% for the flat MLP — 24× more efficient. When matched at ~190K params, the GCN beats MLP-Compact by +6.06pp, confirming that graph inductive biases provide genuine value.

---

Pipeline

flowchart LR
    A("📹 Raw Videos\n3,088 MP4\n18 classes") --> B

    subgraph B["⚙️ Stage 1 — Curation"]
        direction TB
        B1["Discover & count"] --> B2["Undersample to 123/class"]
        B2 --> B3["Source-video-grouped split\n70/15/15"]
    end

    B --> C

    subgraph C["🦴 Stage 2 — Pose Extraction"]
        direction TB
        C1["MediaPipe\n33 landmarks / frame"] --> C2["Sample 32 frames uniform"]
        C2 --> C3["Compute J, B, C features"]
        C3 --> C4["Tensor (9, 32, 33)"]
    end

    C --> D

    subgraph D["🧠 Stage 3 — TFC-GCN v4"]
        direction TB
        D1["3-stream input\nJ | B | C"] --> D2["GCNTFCLayer × 4"]
        D2 --> D3["Global Avg Pool\n+ Dropout"]
        D3 --> D4["FC → 18 classes"]
    end

    D --> E("✅ 71.72%\nEnsemble — Clean Split")

    style A fill:#2d3748,color:#fff,stroke:#4a5568
    style E fill:#276749,color:#fff,stroke:#2f855a

Loading

---

Data Leakage — Critical Methodology Note

The dataset contains source videos split into multiple short clips. An initial split at clip level allowed clips from the same source video to appear in both train and test — 91.5% of test samples had their source video in train.

Model	Leaked split	Clean split	Drop
MLP	86.09%	73.74%	−12pp
BiLSTM	89.64%	68.69%	−21pp
TFC-GCN v1	87.87%	18.86%	−69pp

Fix: metadata_clean.json groups all clips of the same source video into the same split. All results in this project use this clean split unless otherwise noted.

---

Skeleton Features — J, B, C

Three complementary views of the same movement:

── J — Joint Position (relative to hip midpoint) ──────────────────
  J[t,i] = x[t,i] − (x[t,23] + x[t,24]) / 2
  → Invariant to absolute position in frame and body size.

── B — Bone Vector (joint-to-parent geometry) ─────────────────────
  B[t,i] = x[t,i] − x[t, parent(i)]
  → Captures pose geometry: joint angles, bone orientations.

── C — Relative Displacement (movement dynamics) ──────────────────
  C[t,i] = x[t−1,i] − center[t+1]
  → Captures speed and direction of each joint's movement.

Concatenated → (9, 32, 33) float32 tensor per video (~37 KB · ~77 MB for the full dataset).

---

Architecture — TFC-GCN

flowchart TB
    J("x_J — 3×32×33") --> IC1["Conv 1×1 · 3→64"]
    B("x_B — 3×32×33") --> IC2["Conv 1×1 · 3→64"]
    C("x_C — 3×32×33") --> IC3["Conv 1×1 · 3→64"]

    IC1 & IC2 & IC3 --> FU["Concat + Fuse · 192→64"]

    FU --> L1["GCNTFCLayer  64→64"]
    L1 --> L2["GCNTFCLayer  64→64"]
    L2 --> L3["GCNTFCLayer  64→48  stride 2"]
    L3 --> L4["GCNTFCLayer  48→48  stride 2"]

    L4 --> GAP["Global Avg Pool"]
    GAP --> DO["Dropout  p=0.5"]
    DO --> OUT(["FC 48 → 18 · class logits"])

    style J fill:#1a365d,color:#fff,stroke:#2b6cb0
    style B fill:#1a365d,color:#fff,stroke:#2b6cb0
    style C fill:#1a365d,color:#fff,stroke:#2b6cb0
    style OUT fill:#276749,color:#fff,stroke:#2f855a

Loading

Each GCNTFCLayer stacks:

Component	Role
GCNLayer — Â ⊙ M	Aggregates anatomically connected joints; M is a learnable attention mask
TFC Block	4 parallel streams: PointConv · PointConv+Pool · CrossExtractionConv · GatedCNN (GLU)
SST-Att	Spatial-temporal attention over joints and frames per class

Total: 193,664 parameters — under 1 MB.

---

Training — TFC-GCN v4

Hyperparameter	Value	Rationale
Optimizer	Adam (β₁=0.9, β₂=0.999)	SGD diverges with ~1,500 training samples
Learning rate	1×10⁻³	Standard for Adam on small datasets
Scheduler	CosineAnnealingWarmRestarts (T₀=50)	Periodic restarts escape local minima
Loss	Class-weighted CE + label smoothing ε=0.1	Compensates imbalanced test distribution
Weight decay	1×10⁻⁴	L2 regularization
Early stopping	patience=30 on val_acc	Best checkpoint at epoch 106, val_acc=78.05%
Augmentation	Temporal flip (50%) + Gaussian noise σ=0.01 + scale ×[0.95,1.05]	No horizontal flip (destroys deadlift recall)
Normalization	Z-score per channel over train set	Critical — without it accuracy drops below 20%

Ensemble: 3 models (seeds 42, 0, 1) with soft-voting. Individual seeds: 70.03% / 68.69% / 65.66%. Ensemble: 71.72% (+1.68pp over best seed).

---

Dataset

• Original: 3,088 videos · 18 classes · ~1.48s @ 25fps · 1280×720
• After undersample: 2,214 videos (123/class)
• After pose extraction: 2,126 tensors (96.0% success rate)
• Clean split: Train 1,501 · Val 328 · Test 297

Highest failure rates: incline bench press (13%) and bench press (12%) — exercises performed lying on a bench, where full-body landmark detection is harder.

---

Repository Structure

.
├── 01_prepare_dataset.py       # Curation, undersample, source-video-grouped splits
├── 02_extract_pose.py          # MediaPipe extraction + J/B/C features (--t-frames flag)
├── 03_tfc_gcn_model.py         # TFC-GCN architecture (193K params)
├── 04_train.py                 # TFC-GCN v1 — SGD + Nesterov + warmup + cosine
├── 04_train_v2.py              # TFC-GCN v2 — Adam + horizontal flip
├── 05_train_v3.py              # TFC-GCN v3 — Adam + label smoothing
├── 06_train_v4.py              # TFC-GCN v4 — Adam + class weights + CosineWarmRestart ✓
├── eval_v4_ensemble.py         # Ensemble evaluation (3 seeds, soft voting)
├── 05_evaluate.py              # Single-model evaluation — metrics, confusion matrix
├── main.py                     # End-to-end pipeline orchestrator
│
├── baselines/
│   ├── baseline_a_mlp.py       # MLP flat 9504→512→128→18 (4.93M params)
│   ├── baseline_b_lstm.py      # BiLSTM on (T, 99) sequences (868K params)
│   ├── baseline_c_mlp_gap.py   # MLP with temporal GAP (47K params)
│   ├── baseline_d_mlp_compact.py  # MLP parameter-matched to TFC-GCN (190K params)
│   └── COMPARACION.md          # Baseline comparison (original split)
│
├── INFORME/
│   ├── INFORME_COMPLETO.md     # Full academic report (template-compliant)
│   ├── generate_figures.py     # Generates all report figures
│   └── figures/                # 16 figures (PNG) for the report
│
├── processed_data/
│   ├── metadata.json           # Original split (has data leakage — do not use for eval)
│   ├── metadata_clean.json     # Clean split grouped by source video ✓
│   ├── features/               # (9,32,33) tensors — T=32 (2,126 files, ~77 MB)
│   ├── features_t48/           # (9,48,33) tensors — T=48 (experimental)
│   └── analysis/               # Dataset & feature plots (PNG)
│
├── results/                    # TFC-GCN v1 — original split
├── results_clean/              # TFC-GCN v1 — clean split
├── results_v2_clean/           # TFC-GCN v2
├── results_v3_clean/           # TFC-GCN v3
├── results_v4_clean/           # TFC-GCN v4 seed=42  ← main model
├── results_v4_s0/              # TFC-GCN v4 seed=0
├── results_v4_s1/              # TFC-GCN v4 seed=1
│
├── INFORME_FINAL.md            # Development notes & iterative analysis
├── COMPARACION_CLEAN.md        # Full model comparison on clean split
├── PRESENTACION_FINAL.md       # Final presentation slide guide
├── PRESENTACION_1.md           # First presentation (leakage narrative)
├── TESIS.md                    # Master's thesis proposal (FormCheck system)
├── articles/                   # Reference PDFs (TFC-GCN, GLU, NTU)
├── requirements.txt
└── .gitignore

Not tracked by git: raw_datasets/ (21 GB) · processed_data/features/ (~77 MB) · venv/ · *.pth model weights · pose_landmarker_lite.task

---

Quick Start

# 1. Clone & install
git clone <repo-url> && cd VCP/TP
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt

# 2. Full pipeline — clean split (recommended)
python 01_prepare_dataset.py
python 02_extract_pose.py --metadata processed_data/metadata_clean.json
python 06_train_v4.py --metadata processed_data/metadata_clean.json --output results_v4_clean
python eval_v4_ensemble.py   # ensemble of 3 seeds → 71.72%

# 3. Run all baselines
python baselines/baseline_a_mlp.py --metadata processed_data/metadata_clean.json --output baselines/results_a_clean
python baselines/baseline_b_lstm.py --metadata processed_data/metadata_clean.json --output baselines/results_b_clean
python baselines/baseline_d_mlp_compact.py --metadata processed_data/metadata_clean.json --output baselines/results_d_clean

---

References

1. Wang K, Deng H. TFC-GCN: Lightweight Temporal Feature Cross-Extraction Graph Convolutional Network for Skeleton-Based Action Recognition. Sensors 2023, 23, 5593. https://doi.org/10.3390/s23125593
2. Dauphin YN et al. Language Modeling with Gated Convolutional Networks. ICML 2017.
3. Kipf TN, Welling M. Semi-Supervised Classification with Graph Convolutional Networks. ICLR 2017.
4. MediaPipe Pose Landmark Model. Google LLC, 2023. https://developers.google.com/mediapipe/solutions/vision/pose_landmarker

---

MIA304 Visión y Percepción Computarizada · Universidad de San Andrés · Mayo 2026