UARC: Uncertainty-Aware Regime-Conditioned RL for Portfolio Management

A systematic trading research project investigating whether regime-aware reinforcement learning improves portfolio allocation.

---

Summary

This project evaluates a hybrid architecture combining:

• Bayesian HMM (causal regime filtering posteriors)
• iTransformer (cross-asset representation learning)
• IQN (distributional RL with FiLM regime conditioning)

across a 20-asset diversified portfolio over a strict out-of-sample test period (2021–2024).

---

Key Results

Three findings from the ablation (seed 42, full 1000-episode training):

1. Distributional RL helps in-sample; regime conditioning does not. On the validation window (2018–2020), IQN agents reach an excess-return information ratio of 0.84–0.87 vs 0.42–0.45 for single-quantile DQN ablations — but the regime-conditioned UARC (0.84) is indistinguishable from the regime-free IQN (0.87). The value comes from the distributional head, not the HMM regime signal.

2. Learned alpha does not survive out-of-sample. Agents that beat the equal-weight benchmark on 2018–2020 validation underperform it on 2021–2024 test (Sharpe ~0.33 vs 0.61 for buy-and-hold). The cross-sectional patterns learned in the COVID-era validation window actively mislead in the subsequent market.

3. More training worsens generalization. A 40-episode version of the same agents beat buy-and-hold on test (Sharpe 0.69); the fully trained 1000-episode agents do not (0.34). Validation-driven checkpoint selection overfits to the validation regime.

Together: a clean negative result for regime-conditioned RL in liquid ETF allocation after transaction costs, from a pipeline where the agents demonstrably learn (monotonically improving validation, distinct ablation outcomes, active non-uniform allocations).

---

Performance (Test Set: 2021–2024)

System	Sharpe	Ann. Return	Max Drawdown	Avg Turnover	L1 to EqW
Equal Weight Buy-and-Hold	0.606	+7.1%	-17.6%	0.000	0.000
Equal Weight (Rebalanced)	0.539	+6.3%	-17.6%	0.000	0.000
Momentum	0.477	+5.6%	-17.9%	0.010	0.052
Risk Parity	0.407	+2.8%	-14.0%	0.002	0.758
Mean-Variance	0.278	+3.0%	-19.8%	0.053	1.330
No Regime + IQN	0.343	+3.8%	-22.7%	0.013	0.430
UARC (Ours)	0.342	+3.7%	-22.6%	0.013	0.433
HMM Posterior + DQN	0.323	+3.5%	-22.9%	0.014	0.465
HMM Hard + DQN	0.320	+3.5%	-22.9%	0.014	0.467

Seed 42, best-validation checkpoint per agent, transaction costs applied. Validation (2018–2020) information ratios: No Regime IQN 0.87, UARC 0.84, HMM Posterior DQN 0.45, HMM Hard DQN 0.42.

---

Interpretation

• The agents genuinely trade (L1 distance to equal weight ~0.43–0.47, max weight ~15%, daily turnover ~0.013) — this is a real strategy failing out-of-sample, not a degenerate policy
• Validation alpha (IR up to 0.87) inverts on test: the 2018–2020 window (dominated by the COVID crash/recovery) rewards patterns that lose money in 2021–2024
• IQN vs DQN separates clearly on validation; regime signal (none / hard / posterior) never separates — explicit regime inputs are redundant given the encoder
• Equal-weight buy-and-hold remains the strongest system tested, consistent with the difficulty of post-cost daily cross-sectional alpha in liquid ETFs

---

Architecture

State = Embedding (64) + Posterior (K) + Previous Weights (20) → IQN Agent → Portfolio Weights

---

Experimental Setup

• Assets: 20 (equities, bonds, commodities, sectors, defensive)
• Frequency: Daily
• Train: 2000–2017
• Validation: 2018–2020 (used for model selection)
• Test: 2021–2024 (strict holdout, no parameter updates)

No lookahead. HMM uses filtering posteriors p(z_t | x_{1:t}), not Viterbi.

---

Pipeline

pip install -r requirements.txt
python run_stage1.py --seed 42   # HMM + posteriors -> outputs/
python run_stage2.py             # Encoder pretraining -> outputs/
python run_stage3.py --seeds 42  # RL training (use all 5 seeds for full study)
python run_stage4.py             # Backtest + figures

Or run everything:

bash scripts/run_pipeline.sh

Note: data/raw/prices.csv is downloaded automatically on first run via yfinance. Intermediate artifacts are written to outputs/ (gitignored).

---

What Was Fixed (methodology)

• Unified outputs/ paths across all stages (stage 1 no longer writes to a separate seed folder)
• Stage 2 now pretrains the encoder (next-day EW return prediction), not just random init
• Training reward = excess return over the equal-weight benchmark − transaction costs (stationary, grounded in P&L; "do nothing" earns zero, so agents must find relative alpha)
• Action head z-normalizes per-asset scores before softmax: plain softmax over Q-values (differences ~1e-3) provably collapses every agent to equal weight regardless of what the network learns
• Per-asset credit assignment: each asset's quantile head is trained against that asset's own excess return. (The original design broadcast one scalar portfolio reward to all assets, making the cross-asset ranking unlearnable — loss converged to exactly 0.)
• Validation is one deterministic pass over the full 2018–2020 window (random-start episode sampling made checkpoint selection noisy)
• Uncertainty-awareness is applied at the policy level: IQN agents select actions using CVaR-adjusted quantile scores (risk_aversion=0.1), consistently in training and backtest. (Shaping the reward with model-predicted CVaR was tried and diverges — the reward becomes a function of the value estimate.)
• Backtests apply the same linear transaction costs as training
• Validation uses the documented 2018–2020 split (not a re-split of training data)
• Stage 4 backtests all seeds and reports mean ± std

---

Limitations

• Results reported for seed 42; the full protocol (python run_stage3.py with all 5 seeds) adds mean ± std
• Validation window (2018–2020) is dominated by the COVID regime, which generalizes poorly to 2021–2024 — a structural challenge for any walk-forward protocol with this data
• "DQN" ablations use IQN with n_tau=1 (distributional head, single quantile)
• Linear transaction cost model
• Fixed 20-asset universe

---

Reproducibility

pytest tests/

Outputs:

• outputs/backtest_results.csv
• outputs/backtest_results_multiseed.csv
• outputs/figures/

---

License

MIT