Skip to content

Add L1 (CRPS-consistent) ssr_bias_l1 metric to one-step ensemble aggregator#1352

Draft
mcgibbon wants to merge 3 commits into
mainfrom
feature/ssr-bias-l1-one-step-ensemble
Draft

Add L1 (CRPS-consistent) ssr_bias_l1 metric to one-step ensemble aggregator#1352
mcgibbon wants to merge 3 commits into
mainfrom
feature/ssr-bias-l1-one-step-ensemble

Conversation

@mcgibbon

@mcgibbon mcgibbon commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Adds an L1 (CRPS-consistent) analog of the existing L2 ssr_bias to the ensemble aggregator, so 1-step ensemble under-dispersion can be read in units that match the fair-CRPS / energy-score training loss rather than MSE. It is exposed on both the one-step ensemble aggregator and the train/val aggregator path.

The metric is ssr_bias_l1 = spread_L1 / skill_L1 - 1 (target 0 at calibration, same convention as the L2 metric), where the two terms are the two CRPS terms:

  • spread_L1 is the fair mean pairwise absolute difference, 1/(N(N-1)) sum_{i!=j} |x_i - x_j| (the i!=j correction plays the role unbiased variance plays for the L2 spread).
  • skill_L1 is the member MAE, mean_i |x_i - y| (member-vs-obs, not ensemble-mean-vs-obs). At calibration E|X - X'| = E|X - y|, so the ratio approaches 1. Unlike the L2 skill's mse - var/N, the member MAE is already unbiased and gets no finite-N correction: MAE-of-the-ensemble-mean has no closed /N decomposition, so there is no L2-style analog to subtract.

The numerator and denominator sums accumulate across batches and the ratio is formed once in get(), so it is not a per-batch-averaged ratio. Prescribed cells (every member equals the target: zero spread and zero error) report 0 rather than the -1 floor, matching SSRBiasMetric.

Off by default: enable_ssr_bias_l1 gates building the family, so existing runs' logged keys are unchanged.

Changes:

  • fme.ace.aggregator.one_step.ensemble.SSRBiasL1Metric: new ReducedMetric accumulating the spread and skill sums and forming the ratio in get(), with the prescribed-cell 0/0 handling carried over from SSRBiasMetric.

  • _EnsembleAggregator: enable_ssr_bias_l1 flag builds the ssr_bias_l1 family (registered in _get_variable_metrics, added to _diverging_metrics), emitting ssr_bias_l1/<var>, ssr_bias_l1/mean_map/<var>, and (for target="norm") ssr_bias_l1/channel_mean via the existing area-weighted-mean and mean-map plumbing.

  • get_one_step_ensemble_aggregator, EnsembleMetricConfig, OneStepEnsembleMetricConfig: gain enable_ssr_bias_l1: bool = False.

  • fme.ace.aggregator.train.TrainAggregatorConfig: gains enable_ssr_bias_l1: bool = False, forwarded into the _EnsembleAggregator built when ensemble_metrics is enabled, so the L1 metric is available on the train/val aggregator path (e.g. val/ensemble/ssr_bias_l1).

  • Tests added

  • If dependencies changed, "deps only" image rebuilt and "latest_deps_only_image.txt" file updated

…egator

Adds an L1 analog of the L2 ssr_bias so 1-step ensemble under-dispersion
can be read in units matching the fair-CRPS / energy-score training loss.

ssr_bias_l1 = spread_L1 / skill_L1 - 1 (target 0 at calibration), where
spread_L1 is the fair mean pairwise absolute difference
1/(N(N-1)) sum_{i!=j} |x_i - x_j| and skill_L1 is the member MAE
mean_i |x_i - y|. The member-vs-obs skill is already unbiased, so unlike
the L2 metric it carries no finite-N (-var/N) correction.

Changes:
- fme.ace.aggregator.one_step.ensemble.SSRBiasL1Metric: new ReducedMetric
  accumulating the spread and skill sums across batches and forming the
  ratio in get(), with the same prescribed-cell 0/0 handling as
  SSRBiasMetric.
- _EnsembleAggregator: enable_ssr_bias_l1 flag builds the ssr_bias_l1
  family (registered in _get_variable_metrics, added to
  _diverging_metrics) emitting ssr_bias_l1/<var>, .../mean_map/<var>, and
  .../channel_mean, reusing existing plumbing.
- get_one_step_ensemble_aggregator and EnsembleMetricConfig /
  OneStepEnsembleMetricConfig gain enable_ssr_bias_l1: bool = False, so
  existing runs' logged keys are unchanged by default.
@mcgibbon

mcgibbon commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Claude: Independent pre-review (fresh Opus agent, no authoring context; worktree/env-slot isolated on the PR head bbe493f). Verdict: ready for review — no blocking issues found, no fixes applied.

What I checked:

  1. L1 math is correct. spread_L1 is the fair pairwise mean: (gen.unsqueeze(2) - gen.unsqueeze(1)).abs().sum(dim=(1,2)) / (N*(N-1)) — the i==j terms are exactly 0 so this equals the i!=j sum, giving the unbiased E|X-X'|. skill_L1 is member-vs-obs MAE (gen - target).abs().mean(dim=(0,1,2)), not ensemble-mean, and correctly gets no -var/N correction. Independent numeric check: reference spread=5.333, skill=3.333, bias=0.6 matched the metric to float precision (0.6000001), and a 20k-sample iid calibrated ensemble gave bias -0.0017 (→0). ✓

  2. Ratio-at-end. Numerator/denominator accumulate in _total_spread/_total_skill; ratio formed once in get(). The test_..._forms_ratio_at_end_not_per_batch test genuinely distinguishes the two orderings: it asserts == (2+10)/(4+5)-1 (0.333) and != 0.25 (the per-batch-ratio mean). A per-batch implementation would fail it. ✓

  3. Cross-batch weighting matches SSRBiasMetric: per-batch means summed across batches, ratio at end, so n_batches cancels and each batch is equally weighted. Consistent and defensible. ✓

  4. Default-off invariance. The family is registered in _variable_metrics only when enable_ssr_bias_l1=True; _get_data iterates families generically, so nothing leaks when disabled. The unconditional _diverging_metrics membership is inert with no family present. Test confirms no ssr_bias_l1 keys in logs/dataset and existing ssr_bias/a unchanged. ✓

  5. Prescribed-cell 0/0 reports 0, matching SSRBiasMetric. Correctly uses linear MAE units for the floor (_PRESCRIBED_MSE_RTOL * skill.max(), not .square().max()). Test covers prescribed→0 and zero-spread-with-error→-1. ✓

  6. Plumbing/perf. enable_ssr_bias_l1 threaded through both EnsembleMetricConfig and OneStepEnsembleMetricConfig, both .build() paths, and get_one_step_ensemble_aggregator_EnsembleAggregator; mean_map/channel_mean flow through the generic loop. Pairwise materializes (batch, N, N, time, lat, lon) — O(N²), fine for the small one-step ensemble sizes and consistent with existing style.

Verification run in the env slot: test_ensemble.py 33 passed; test_main.py + test_train.py 32 passed (no regressions); pre-commit (ruff, ruff-format, mypy) all pass on both changed files.

Two cosmetic non-issues (mentioned only for completeness, no change needed): SSRBiasL1Metric._n_batches is tracked but unused in get() — but SSRBiasMetric does the same; and the shared _PRESCRIBED_MSE_RTOL constant name carries "MSE" while reused as a generic rtol for the L1 floor.

@mcgibbon mcgibbon marked this pull request as ready for review July 9, 2026 16:15
@mcgibbon

mcgibbon commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

I'll make use of this in an experiment before we consider reviewing/merging.

@mcgibbon mcgibbon marked this pull request as draft July 9, 2026 16:27
mcgibbon added 2 commits July 9, 2026 16:33
The new enable_ssr_bias_l1 field on EnsembleMetricConfig changes the
auto-generated default aggregator config, so refresh the committed
regression fixture (docs/test_default_aggregator_config.py).
Add enable_ssr_bias_l1 to TrainAggregatorConfig and pass it into the
_EnsembleAggregator built when ensemble_metrics is enabled, so the L1
(CRPS-consistent) ssr_bias_l1 metric can be logged on the train/val
aggregator path (e.g. val/ensemble/ssr_bias_l1). Off by default.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant