Add L1 (CRPS-consistent) ssr_bias_l1 metric to one-step ensemble aggregator#1352
Add L1 (CRPS-consistent) ssr_bias_l1 metric to one-step ensemble aggregator#1352mcgibbon wants to merge 3 commits into
Conversation
…egator
Adds an L1 analog of the L2 ssr_bias so 1-step ensemble under-dispersion
can be read in units matching the fair-CRPS / energy-score training loss.
ssr_bias_l1 = spread_L1 / skill_L1 - 1 (target 0 at calibration), where
spread_L1 is the fair mean pairwise absolute difference
1/(N(N-1)) sum_{i!=j} |x_i - x_j| and skill_L1 is the member MAE
mean_i |x_i - y|. The member-vs-obs skill is already unbiased, so unlike
the L2 metric it carries no finite-N (-var/N) correction.
Changes:
- fme.ace.aggregator.one_step.ensemble.SSRBiasL1Metric: new ReducedMetric
accumulating the spread and skill sums across batches and forming the
ratio in get(), with the same prescribed-cell 0/0 handling as
SSRBiasMetric.
- _EnsembleAggregator: enable_ssr_bias_l1 flag builds the ssr_bias_l1
family (registered in _get_variable_metrics, added to
_diverging_metrics) emitting ssr_bias_l1/<var>, .../mean_map/<var>, and
.../channel_mean, reusing existing plumbing.
- get_one_step_ensemble_aggregator and EnsembleMetricConfig /
OneStepEnsembleMetricConfig gain enable_ssr_bias_l1: bool = False, so
existing runs' logged keys are unchanged by default.
|
Claude: Independent pre-review (fresh Opus agent, no authoring context; worktree/env-slot isolated on the PR head bbe493f). Verdict: ready for review — no blocking issues found, no fixes applied. What I checked:
Verification run in the env slot: Two cosmetic non-issues (mentioned only for completeness, no change needed): |
|
I'll make use of this in an experiment before we consider reviewing/merging. |
The new enable_ssr_bias_l1 field on EnsembleMetricConfig changes the auto-generated default aggregator config, so refresh the committed regression fixture (docs/test_default_aggregator_config.py).
Add enable_ssr_bias_l1 to TrainAggregatorConfig and pass it into the _EnsembleAggregator built when ensemble_metrics is enabled, so the L1 (CRPS-consistent) ssr_bias_l1 metric can be logged on the train/val aggregator path (e.g. val/ensemble/ssr_bias_l1). Off by default.
Adds an L1 (CRPS-consistent) analog of the existing L2
ssr_biasto the ensemble aggregator, so 1-step ensemble under-dispersion can be read in units that match the fair-CRPS / energy-score training loss rather than MSE. It is exposed on both the one-step ensemble aggregator and the train/val aggregator path.The metric is
ssr_bias_l1 = spread_L1 / skill_L1 - 1(target 0 at calibration, same convention as the L2 metric), where the two terms are the two CRPS terms:spread_L1is the fair mean pairwise absolute difference,1/(N(N-1)) sum_{i!=j} |x_i - x_j|(thei!=jcorrection plays the role unbiased variance plays for the L2 spread).skill_L1is the member MAE,mean_i |x_i - y|(member-vs-obs, not ensemble-mean-vs-obs). At calibrationE|X - X'| = E|X - y|, so the ratio approaches 1. Unlike the L2 skill'smse - var/N, the member MAE is already unbiased and gets no finite-N correction: MAE-of-the-ensemble-mean has no closed/Ndecomposition, so there is no L2-style analog to subtract.The numerator and denominator sums accumulate across batches and the ratio is formed once in
get(), so it is not a per-batch-averaged ratio. Prescribed cells (every member equals the target: zero spread and zero error) report 0 rather than the -1 floor, matchingSSRBiasMetric.Off by default:
enable_ssr_bias_l1gates building the family, so existing runs' logged keys are unchanged.Changes:
fme.ace.aggregator.one_step.ensemble.SSRBiasL1Metric: newReducedMetricaccumulating the spread and skill sums and forming the ratio inget(), with the prescribed-cell 0/0 handling carried over fromSSRBiasMetric._EnsembleAggregator:enable_ssr_bias_l1flag builds thessr_bias_l1family (registered in_get_variable_metrics, added to_diverging_metrics), emittingssr_bias_l1/<var>,ssr_bias_l1/mean_map/<var>, and (fortarget="norm")ssr_bias_l1/channel_meanvia the existing area-weighted-mean and mean-map plumbing.get_one_step_ensemble_aggregator,EnsembleMetricConfig,OneStepEnsembleMetricConfig: gainenable_ssr_bias_l1: bool = False.fme.ace.aggregator.train.TrainAggregatorConfig: gainsenable_ssr_bias_l1: bool = False, forwarded into the_EnsembleAggregatorbuilt whenensemble_metricsis enabled, so the L1 metric is available on the train/val aggregator path (e.g.val/ensemble/ssr_bias_l1).Tests added
If dependencies changed, "deps only" image rebuilt and "latest_deps_only_image.txt" file updated