Supreme — Project Overview

What is Supreme?

Supreme is a World Models architecture (Ha & Schmidhuber, 2018) enhanced with HOPE (Hierarchical memory from the Nested Learning paper). The core innovation replaces the standard MDN-RNN memory module with a HOPE-based memory system that combines a Self-Referential Learning Module (Self-Modifying Titans) and a Continuum Memory System (CMS) with multi-frequency memory tiers.

The goal: build an agent that learns a compressed world model of its environment, then trains entirely inside its own hallucinated dreams — transferring learned policies back to reality.

Architecture

Supreme follows the V-M-C (Vision-Memory-Controller) decomposition from World Models:

Environment Frame ──► V (ConvVAE) ──► z_t (latent)
                                         │
                                         ▼
                          ┌──────────────────────────────┐
              a_t ──────► │  M (HOPEMemory)              │
                          │  ┌────────────────────────┐  │
                          │  │ Self-Modifying Titans   │  │
                          │  │ (128-dim, 6 matrices,   │  │
                          │  │  Delta Gradient Descent) │  │
                          │  └──────────┬─────────────┘  │
                          │             ▼                │
                          │  ┌────────────────────────┐  │
                          │  │ CMS (4 frequency levels) │  │
                          │  │ Freq: [1, 16, 1M, 16M]  │  │
                          │  │ Output: 256-dim h_t      │  │
                          │  └──────────┬─────────────┘  │
                          │             ▼                │
                          │  ┌────────────────────────┐  │
                          │  │ MDN Head               │  │
                          │  │ 5 Gaussians, 32-dim    │  │
                          │  └────────────────────────┘  │
                          └──────────────────────────────┘
                                         │
                          z_t + h_t ─────►  C (Controller)  ──► a_t

Component Summary

Component	Architecture	Parameters	Status
V (Vision)	ConvVAE encoder	~4.4M	Frozen after Phase 1
M (Memory)	HOPEMemory (Titans + CMS + MDN)	~2.3M	Frozen after Phase 2
C (Controller)	1-hidden-layer MLP (32 ELU units)	5,218	Trained via CMA-ES
Total		~6.7M

Environment

Task: MuJoCo Reacher-v5 — a 2-DOF planar arm reaching toward a random target
Observation: 64×64 RGB frames
Action space: 2 continuous actions in [-1, 1] (joint torques)
Episode length: 50 steps
Reward: −distance_to_target − action_penalty

Phase-by-Phase Development

Phase 0 — Environment Setup

Set up MuJoCo Reacher-v5 with the full simulation pipeline: XML physics definition, Python gym wrapper (reset(), step(), _get_frame()), batch rollout collection, and environment verification.

Stack: Python 3.12, PyTorch, MuJoCo, Mac M4 Air (16GB RAM, MPS backend)

Phase 1 — V Model (ConvVAE)

Goal: Train a convolutional VAE to compress 64×64 RGB frames into a 32-dimensional latent space.

Metric	Value
Data	100,000 frames from 2,000 rollouts (50 steps each)
Storage	HDF5, uint8, ~375 MB
Latent dim	32
Training	1 epoch, ~2 minutes on MPS
KL health	13.80 per sample (healthy range 0.5–20)
Reconstruction	Arm pose and scene structure visible
Latent interpolation	Smooth, avg_diff = 0.0108

Key fix: Hit posterior collapse (KL=0) on first two attempts. Resolved with free bits (kl_tolerance=0.5 per dimension, from the World Models paper).

Deliverable: checkpoints/vae/vae_frozen.pt — frozen encoder mapping (B, 3, 64, 64) → (B, 32)

Phase 2 — M Model (HOPE Memory)

Goal: Build the predictive world model that learns environment dynamics: P(z_{t+1} | a_t, z_t, h_t).

The M model replaces the original MDN-RNN with HOPE, a two-system architecture:

Self-Modifying Titans (high expressiveness, small capacity):

6 self-referential memory matrices (M_k, M_v, M_q, M_η, M_α, M_memory)
Each matrix generates its own keys, values, learning rates, and decay rates
Updated via L2-regression Delta Gradient Descent (Eq. 86–93 from NL paper)
Hidden dim: 128
L2-normalized q, k; local convolutions with window size 4

Continuum Memory System (high capacity, simple learning rule):

4 frequency levels [1, 16, 1,000,000, 16,000,000] (from the NL paper)
Sequential MLP chain where each level updates at its own frequency
For 50-step Reacher episodes, only L0 (freq=1) and L1 (freq=16) actively update; L2/L3 are effectively frozen, serving as stable knowledge anchors
Output dimension: 256

MDN Head:

5 Gaussian mixture components
Diagonal covariance
Temperature-controlled sampling for dream training

Training: AdamW optimizer, MDN negative log-likelihood loss, gradient clipping. VAE re-sampling per batch (z ~ N(μ, σ) not fixed z, per the World Models paper).

Deliverable: checkpoints/m_model/hope_m_frozen.pt — controller input dim = 32 (z) + 128 (h) = 160

Phase 3 — C Model (Controller)

Goal: Train a minimal controller that maps world model representations to actions.

Phase 3 v1 — Linear Controller

Metric	Value
Architecture	Single linear layer
Parameters	322
CMA-ES config	pop=16, 100 gen, 4 rollouts/eval
Training time	7.4 minutes
Best reward	−16.1
Random baseline	−42.3 ± 4.2
Improvement	+26.7 over random

Takeaway: Training time of only 7 minutes signaled under-provisioned compute — scale up.

Phase 3 v2 — Nonlinear Controller

Metric	Value
Architecture	1 hidden layer, 32 ELU units
Parameters	5,218
CMA-ES config	pop=64, 100 gen, 8 rollouts/eval
Training time	57.8 minutes
Best reward	−8.2
Random baseline	−42.3
Improvement	+34.1 over random

This became the gold standard — the target for all dream-trained controllers to match.

Deliverable: checkpoints/controller/controller.pt

Phase 4 — Dream Training (First Attempt)

Goal: Train the controller entirely inside M's hallucinated world via a DreamEnv gym wrapper around HOPEMemory.

Key elements:

DreamEnv wrapping HOPEMemory as a proper gym.Env
reset_level parameter controlling which CMS tiers reset between episodes
Temperature sweep: τ ∈ {0.5, 1.0, 1.15, 1.3}
4-step dream coherence validation pipeline before CMA-ES training

Outcome: Dream-trained controller failed to match Phase 3v2's −8.2 in real-environment transfer. The linear reward proxy (reward ≈ w^T z + b) had poor correlation with real rewards (correlation ~−0.08).

Phase 5 — Iterative Training & Reward Head

Goal: Close the dream-to-real gap by adding a learned reward head to M and running the iterative V-M-C training loop.

Key additions:

reward_head: Linear(hidden_dim, 1) added to M as a parallel branch
Iterative loop: rollout → retrain V → re-encode → retrain M → retrain C → evaluate
Warm-start CMA-ES from previous best controller parameters
Anti-exploitation measures (z-clip, reward clamping)

Result: Dream transfer improved to ~−77 (vs −42 random, vs −8.2 real-trained), but still far from the Phase 3v2 gold standard.

Phase 6 — Scaling & Optimization

Goal: Address reward head gradient starvation and add M3 optimizer, CMS ablation, scaling to harder tasks.

Diagnosis: The reward head received < 0.02% of total gradient signal because reward_weight=0.1 competed against MDN loss of ~−100. The reward head's correlation collapsed to 0.01–0.11, causing controller regression from −77 to −107.

Sub-phases validated:

CMS frequency separation (L0 drifts > L1 > L2/L3 as expected)
Continual learning (forgetting delta improved)
M3 optimizer wiring
Reward head instability identified

Phase 7 — Calibrated Reward Head

Goal: Fix the gradient starvation by decoupling reward head training entirely.

Approach: Freeze the entire M backbone (Titans + CMS + MDN), train only the 16,513-parameter reward head (MLP: 256→64→1 with ReLU) in isolation with lr=1e-3 for 30 epochs. Target: reward correlation ≥ 0.7 before proceeding to dream CMA-ES.

Result: Reward head correlation improved from 0.10 to 0.86 with 0% MDN degradation. However, full 50-step dream rollouts still suffered from compounding prediction error. Real-env transfer: −107.02 ± 2.18.

Phase 8 — Hybrid Dream-Real Training (Final)

Goal: Solve the compounding prediction error problem with a three-pronged approach grounded in recent research.

Phase 8A — Multi-Step Fine-Tuning (21.9 min)

Retrained M with a weighted multi-step loss (K=10, exponentially decaying weights) plus scheduled sampling (p_real 1.0 → 0.5) to close the train-inference gap.

Metric	Before	After
R² at horizon 25	0.9735	0.9902
Single-step loss	−119.95	−138.00

Phase 8B+C — Branched CMA-ES + Uncertainty Penalty (7.2 min)

Instead of 50-step pure dream rollouts, used short branched rollouts (k=5→10→15 steps) from real replay buffer states, with MDN variance penalty (λ_u = 0.5).

Detail	Value
Real-state buffer	24,500 transitions from 500 episodes
Branch curriculum	k=5 → k=10 → k=15 steps
CMA-ES convergence	Generation 52 (early stopped)
Best dream reward	−31.99

Phase 8D — Transfer Evaluation

Agent	Real Reward	Notes
Random	−43.17 ± 3.44	No learning
Phase 8 (branched+penalty)	−59.71 ± 2.19	Best dream-trained — worse than random
Phase 7 (calibrated reward)	−107.02 ± 2.18	Full 50-step dreams
Phase 3v2 (real CMA-ES)	−9.88 ± 3.22	Gold standard (real-env trained)

Honest disclosure (post-audit): Phase 8 currently performs worse than a random policy. The 47-point improvement over Phase 7 is real, but the dream-to-real transfer gap is not yet closed. See audit_report.md for the methodology issues being fixed.

Total Phase 8 time: 31.4 minutes

Results Summary

Phase	Agent Type	Real Reward	Training Time	Key Innovation
—	Random baseline	−43.17 ± 3.44	—	—
3v1	Real-env linear	−16.1	7.4 min	Linear controller
3v2	Real-env nonlinear	−8.2	57.8 min	ELU hidden layer
4	Dream (full rollout)	Poor transfer	~10 min	DreamEnv wrapper
5	Iterative + reward head	~−77	~45 min	Learned reward head
6	M3 + scaling	−107 (regressed)	~67 min	Gradient starvation bug
7	Calibrated reward	−107.02	~15 min	Decoupled reward training
8	Branched + penalty	−59.71	31.4 min	MBPO + scheduled sampling

Key Learnings

Keep C tiny. All intelligence should reside in V and M. The 5,218-parameter controller outperforms because CMA-ES can still search the space tractably.
Project HOPE memory to fixed dim. HOPE's memory matrix must be projected to a fixed 256-dim h_t for CMA-ES tractability.
Compounding error is the enemy of dream training. 50-step autoregressive rollouts accumulate systematic biases that controllers exploit. Short branched rollouts (k=5–15) from real states are the solution.
Decouple losses with mismatched scales. The reward head gradient starvation (Phase 6) taught us to never train a tiny head jointly against a dominant loss term.
Temperature alone doesn't fix exploitation. The World Models paper's temperature trick (τ=1.15) helps but doesn't solve the fundamental train-inference mismatch.
CMS frequency levels vs episode length. The NL paper's frequencies [1, 16, 1M, 16M] work well — for 50-step Reacher episodes, L0 and L1 actively update while L2/L3 serve as frozen knowledge anchors, naturally creating a fast/slow hierarchy.
Gradient flow through CMS requires functional_call. In-context weight updates need torch.func.functional_call to maintain gradient flow.
Always sanity-check compute budget. Phase 3v1's 7-minute training was a red flag — scaling to 58 minutes revealed much better solutions.

HOPE's Unique Contributions

What HOPE enables that a flat LSTM cannot:

Cross-episode memory persistence. The reset_level parameter lets slow CMS tiers persist across dream episodes, accumulating task-level statistics.
CMS frequency separation. Validated in Phase 6: L0 (freq=1) drifts most during training; L1 (freq=16) drifts moderately; L2/L3 (freq=1M/16M) show zero drift — the frequency hierarchy works as designed.
Self-referential adaptation. Titans' 6 self-modifying matrices generate their own learning rules, giving high expressiveness with small parameter count.
Branching advantage. Under MBPO-style short rollouts, slower CMS tiers retain cross-branch context while faster tiers adapt to each branch start state.

References

Ha, D. & Schmidhuber, J. (2018). "World Models." arXiv:1803.10122
Behrouz et al. (2025). "Nested Learning." (HOPE/NL paper)
Janner, M. et al. (2019). "When to Trust Your Model: Model-Based Policy Optimization." NeurIPS (MBPO)
Yu, T. et al. (2020). "MOPO: Model-Based Offline Policy Optimization." NeurIPS
Reference repos: obekt/HOPE-nested-learning, WindOfNature/Nested-Learning, ctallec/world-models

Navigation Menu

Uh oh!

Supreme — Project Overview

What is Supreme?

Architecture

Component Summary

Environment

Phase-by-Phase Development

Phase 0 — Environment Setup

Phase 1 — V Model (ConvVAE)

Phase 2 — M Model (HOPE Memory)

Phase 3 — C Model (Controller)

Phase 3 v1 — Linear Controller

Phase 3 v2 — Nonlinear Controller

Phase 4 — Dream Training (First Attempt)

Phase 5 — Iterative Training & Reward Head

Phase 6 — Scaling & Optimization

Phase 7 — Calibrated Reward Head

Phase 8 — Hybrid Dream-Real Training (Final)

Phase 8A — Multi-Step Fine-Tuning (21.9 min)

Phase 8B+C — Branched CMA-ES + Uncertainty Penalty (7.2 min)

Phase 8D — Transfer Evaluation

Results Summary

Key Learnings

HOPE's Unique Contributions

References

Uh oh!

FilesExpand file tree

Supreme.md

Latest commit

History

Supreme.md

File metadata and controls

Supreme — Project Overview

What is Supreme?

Architecture

Component Summary

Environment

Phase-by-Phase Development

Phase 0 — Environment Setup

Phase 1 — V Model (ConvVAE)

Phase 2 — M Model (HOPE Memory)

Phase 3 — C Model (Controller)

Phase 3 v1 — Linear Controller

Phase 3 v2 — Nonlinear Controller

Phase 4 — Dream Training (First Attempt)

Phase 5 — Iterative Training & Reward Head

Phase 6 — Scaling & Optimization

Phase 7 — Calibrated Reward Head

Phase 8 — Hybrid Dream-Real Training (Final)

Phase 8A — Multi-Step Fine-Tuning (21.9 min)

Phase 8B+C — Branched CMA-ES + Uncertainty Penalty (7.2 min)

Phase 8D — Transfer Evaluation

Results Summary

Key Learnings

HOPE's Unique Contributions

References