Consolidate merge#58
Conversation
Resolves the critical finding from ROADMAP_SOTA_MASSIVE.md: the
EMPIRICAL_VALIDATION_REPORT.md was measuring an AR(1) proxy, not the
real MASSIVE engine. This commit integrates the actual simulator.
## What changed
- benchmarks/massive_real.py (NEW, 195 lines): wraps simulator.simular()
with heurístico mode, deterministic seed, 2-param linear calibration
(a*opinion + b -> polarization) fit on train split.
- benchmarks/runner.py: adds --real flag that invokes the real engine.
--offline (proxy) and --llm (LLM-backed) modes preserved.
- datasets/real_cases/<12>/timeseries.csv: fixed CRLF -> LF for
consistent parsing on Linux.
## Results (n=12, 3 seeds)
| Metric | MASSIVE real | MASSIVE proxy |
|-----------------------|--------------|---------------|
| Mean MAE | 0.0929 | 0.1616 |
| Wins vs naive | 9/12 | 1/12 |
| Wins vs proxy (self) | 10/12 | 2/12 |
| Cross-seed determinism| 100% (std=0) | - |
- MASSIVE real vs naive: 9/12 wins, median MAE 0.0777 vs 0.0950 (-18%)
- MASSIVE real vs proxy: 10/12 wins, paired t-test t=+3.009, **p=0.0119** (significant at α=0.05)
- Directional accuracy: 0.674 (real) vs 0.000 (naive)
- 3 losses: hong_kong, south_korea (cascade dynamics), us_election (volatility)
## What this proves
N1 criterion (ROADMAP) PASSED: real engine > proxy in 10/12 cases.
The proxy was a flat AR(1); the real engine adds mechanistic value
through the heurístico rule selector and polarization dynamics.
## Reproducibility
export PYTHONHASHSEED=42
python3 -m benchmarks.runner --cases datasets/real_cases --real \
--out reports/sota_v0 --seed 42
## Files
- experiments/06_real_benchmark_v0/REPORT.md (157 lines, full analysis)
- experiments/06_real_benchmark_v0/{metrics.json, cross_seed_stats.json}
- experiments/sota_roadmap/ROADMAP_SOTA_MASSIVE.md (450 lines, the master plan)
- benchmarks/massive_real.py (new module)
- benchmarks/runner.py (--real flag added)
Refs: massive-benchmark-project memory
- Add strong baselines (ETS via statsmodels, ARIMA via SARIMAX, Ridge(lags))
- Keep proxy only for --offline; --real runs MASSIVE real (heurístico)
- Integrate cluster_id conditioning (light-touch) in massive_real wrapper
- EXP-003: EnKF delta script (n_ensemble=32, sigma_obs=0.02, interval=3)
- Full 12-case EnKF run: 11/12 improvements, mean ΔMAE=+0.0063 (p=0.0389)
- Consolidated report with per-case MAE and baseline comparisons
- Reproducibility commands and threat/limitations sections
Artifacts:
- experiments/08_enkf_delta/exp_003_enkf.py
- experiments/11_consolidation/CONSOLIDATED_REPORT.md
- reports/{sota_baselines,enkf_pilot,enkf_pilot_i3_s002,enkf_full_i3_s002,cluster_run}
Deps: requirements.txt adds statsmodels and scikit-learn
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (30)
📝 WalkthroughWalkthroughAdds statistical/ML forecasting baselines (ETS, ARIMA, threshold-logistic, Ridge-lags) with dependency-tolerant assembly, a new real MASSIVE simulator forecasting module with linear calibration, a ChangesBaselines and dependency management
Real MASSIVE engine forecasting and runner integration
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Runner as benchmarks/runner.py
participant MassiveReal as massive_real.py
participant Simulator as simular()
Runner->>MassiveReal: massive_real_forecast_with_calibration(train, horizon, seed)
MassiveReal->>MassiveReal: build_initial_state(train)
MassiveReal->>Simulator: simular(estado, pasos)
Simulator-->>MassiveReal: history (opinion trajectory)
MassiveReal->>MassiveReal: fit linear calibration on train slice
MassiveReal-->>Runner: clipped polarization forecast
Runner->>Runner: fallback to offline AR(1) forecast if real fails
sequenceDiagram
participant Main as exp_003_enkf.main
participant RunCase as run_case()
participant Simulator as simular()
participant EnKF as assimilate_history_observations()
Main->>RunCase: run_case(case, seed, obs_interval, sigma_obs, n_ensemble)
RunCase->>Simulator: simular(estado, pasos=n_total)
Simulator-->>RunCase: history
RunCase->>RunCase: fit linear mapping, compute pre-forecast
RunCase->>EnKF: assimilate_history_observations(history, observations)
EnKF-->>RunCase: assimilated mean opinions
RunCase-->>Main: pre/post metrics and deltas
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 Biome (2.5.1)experiments/06_real_benchmark_v0/metrics.jsonFile contains syntax errors that prevent linting: Line 63: String values must be double quoted.; Line 64: String values must be double quoted.; Line 66: String values must be double quoted.; Line 133: String values must be double quoted.; Line 134: String values must be double quoted.; Line 136: String values must be double quoted.; Line 203: String values must be double quoted.; Line 204: String values must be double quoted.; Line 206: String values must be double quoted.; Line 273: String values must be double quoted.; Line 274: String values must be double quoted.; Line 276: String values must be double quoted.; Line 343: String values must be double quoted.; Line 344: String values must be double quoted.; Line 346: String values must be double quoted.; Line 413: String values must be double quoted.; Line 414: String values must be double quoted.; Line 416: String values must be double quoted.; Line 483: String values must be double quoted.; Line 484: String values must be double quoted.; Line 486: String values must be double quoted.; Line 553: String values must be double quoted.; Line 554: String values must be double quoted.; Line 556: String values must be double quoted.; Line 623: String values must be double quoted.; Line 624: String values must be double quoted.; Line 626: String values must be double quoted.; Line 693: String values must be double quoted.; Line 694: String values must be double quoted.; Line 696: String values must be double quoted.; Line 763: String values must be double quoted.; Line 764: String values must be double quoted.; Line 766: String values must be double quoted.; Line 833: String values must be double quoted.; Line 836: String values must be double quoted. reports/cluster_run/metrics.jsonFile contains syntax errors that prevent linting: Line 96: String values must be double quoted.; Line 97: String values must be double quoted.; Line 99: String values must be double quoted.; Line 199: String values must be double quoted.; Line 200: String values must be double quoted.; Line 202: String values must be double quoted.; Line 302: String values must be double quoted.; Line 303: String values must be double quoted.; Line 305: String values must be double quoted.; Line 405: String values must be double quoted.; Line 406: String values must be double quoted.; Line 408: String values must be double quoted.; Line 508: String values must be double quoted.; Line 509: String values must be double quoted.; Line 511: String values must be double quoted.; Line 611: String values must be double quoted.; Line 612: String values must be double quoted.; Line 614: String values must be double quoted.; Line 714: String values must be double quoted.; Line 715: String values must be double quoted.; Line 717: String values must be double quoted.; Line 817: String values must be double quoted.; Line 818: String values must be double quoted.; Line 820: String values must be double quoted.; Line 920: String values must be double quoted.; Line 921: String values must be double quoted.; Line 923: String values must be double quoted.; Line 1023: String values must be double quoted.; Line 1024: String values must be double quoted.; Line 1026: String values must be double quoted.; Line 1126: String values must be double quoted.; Line 1127: String values must be double quoted.; Line 1129: String values must be double quoted.; Line 1229: String values must be double quoted.; Line 1232: String values must be double quoted. reports/sota_baselines/metrics.jsonFile contains syntax errors that prevent linting: Line 96: String values must be double quoted.; Line 97: String values must be double quoted.; Line 99: String values must be double quoted.; Line 199: String values must be double quoted.; Line 200: String values must be double quoted.; Line 202: String values must be double quoted.; Line 302: String values must be double quoted.; Line 303: String values must be double quoted.; Line 305: String values must be double quoted.; Line 405: String values must be double quoted.; Line 406: String values must be double quoted.; Line 408: String values must be double quoted.; Line 508: String values must be double quoted.; Line 509: String values must be double quoted.; Line 511: String values must be double quoted.; Line 611: String values must be double quoted.; Line 612: String values must be double quoted.; Line 614: String values must be double quoted.; Line 714: String values must be double quoted.; Line 715: String values must be double quoted.; Line 717: String values must be double quoted.; Line 817: String values must be double quoted.; Line 818: String values must be double quoted.; Line 820: String values must be double quoted.; Line 920: String values must be double quoted.; Line 921: String values must be double quoted.; Line 923: String values must be double quoted.; Line 1023: String values must be double quoted.; Line 1024: String values must be double quoted.; Line 1026: String values must be double quoted.; Line 1126: String values must be double quoted.; Line 1127: String values must be double quoted.; Line 1129: String values must be double quoted.; Line 1229: String values must be double quoted.; Line 1232: String values must be double quoted. Comment |
2 participants
Pull Request
Resumen de cambios
Describe brevemente qué se ha modificado y por qué.
Tipo de cambio
Módulos afectados
massive_core/data_assimilation/— EnKF, filtro EnKF dispersomassive_core/network_inference/— Reconstrucción de red (DE, CG)massive_core/numerics/— Steppers, estabilidad, motor multicapa dispersomassive_core/physics/— Perturbación, mecánica estadísticatests/— Pruebas unitariasfrontend/— UI, generated typesREADME.md/README_ES.md— Documentaciónmassive_engine.py— Motor principalChecklist
pytest tests/)CLAUDE.mdNotas de integración
Si este PR toca el motor principal (
massive_engine.py) o el frontend, especificar cambios en las rutas de la API o la UI.Summary by cubic
Integrates the real MASSIVE engine into the benchmark runner and adds stronger baselines and EnKF experiments. Real runs now use the actual simulator with a simple train-calibrated mapping and ship full reproducibility docs and artifacts.
New Features
benchmarks/massive_real.pywith deterministic seed and a 2-parameter train-fit mapping (a*opinion + b → polarization).--realflag to use the actual simulator;--offlineand--llmremain unchanged.benchmarks/baselines.py.experiments/08_enkf_delta/exp_003_enkf.pyand consolidated reports; 11/12 cases improve (mean ΔMAE +0.0063; p=0.0389)..gitignore.Dependencies
statsmodels>=0.14.0andscikit-learn>=1.4.0torequirements.txt.Written for commit f10930c. Summary will update on new commits.
Summary by CodeRabbit
New Features
Bug Fixes
Documentation