Bc cvar updates by bcakire · Pull Request #107 · ReEDS-Model/ReEDS

bcakire · 2026-05-26T18:30:14Z

Summary

This PR adds CVAR/NCVAR reporting and target checks to the PRAS resource adequacy workflow.

This implementation builds on the aa/metrics_extension / MultiMetricRA workflow. Standard RA metrics are still used for stress-period selection, while CVAR/NCVAR are added as check-only metrics.

Main idea is:

PRAS can write CVAR/NCVAR-related outputs.
Python can read sample-level PRAS shortfall outputs.
CVAR/NCVAR can be calculated by hierarchy level.
CVAR/NCVAR can be checked against user-defined thresholds.
For now, CVAR/NCVAR are check-only metrics:
- they do not add stress periods,
- they do not update PRM,
- they do not replace the standard MultiMetricRA stress-period metrics.

Standard stress-period metrics remain:

NEUE / LOLH / LOLE / LOLD / Duration / Depth

CVAR/NCVAR are evaluated separately after the standard RA metrics are written.

What changed

`run_pras.jl`

Added:

--cvar_alpha
--write_shortfall_samples_totals

When PRAS.ShortfallSamples() is available, run_pras.jl can now write:

PRAS_{t}i{iteration}-risk_metrics.csv
PRAS_{t}i{iteration}-shortfall_totals_by_sample.h5

risk_metrics.csv includes CVAR/NCVAR-related PRAS risk metrics such as:

CVAR in MWh
NCVAR in ppm
alpha
estimate
standard error
VaR

shortfall_totals_by_sample.h5 includes total shortfall by PRAS sample for USA and regions. These values are totals over the full PRAS resource adequacy period, not annual values.

`stress_periods.py`

Added CVAR/NCVAR handling with:

get_cvar_alpha()
get_shortfall_totals_by_sample()
_sample_cvar()
get_annual_cvar_stress_metric()
evaluate_cvar_target_check()

CVAR/NCVAR are calculated by hierarchy level.

NCVAR is normalized by total PRAS load and reported in ppm.

CVAR is written as an annualized MWh value.

The standard ra_metrics_{t}i{iteration}.csv output remains focused on the MultiMetricRA stress-period metrics. CVAR/NCVAR are written separately as:

cvar_{t}i{iteration}.csv
ncvar_{t}i{iteration}.csv
cvar_target_check_{t}i{iteration}.csv

CVAR/NCVAR are intentionally kept out of the normal stress-period selection loop. They are only used for reporting and target checks.

`ra_calcs.py`

Added logic so write_shortfall_samples_totals is turned on automatically when CVAR/NCVAR checks are requested.

This output is also kept on when PRAS-informed PRM updates need sample-level shortfall totals.

`runreeds.py`

Updated compatibility checks so CVAR and NCVAR are allowed in:

GSw_PRM_StressThresholdMetrics

but treated as check-only metrics instead of standard stress-period metrics.

Switches / cases

Added or updated these switches:

GSw_PRM_CVARAlpha
GSw_PRM_StressThresholdCVAR
GSw_PRM_StressThresholdNCVAR

Updated the description of:

GSw_PRM_StressThresholdMetrics

so it includes the standard MultiMetricRA stress-period metrics and the new check-only CVAR/NCVAR metrics.

The standard MultiMetricRA threshold format follows the aa/metrics_extension format:

GSw_PRM_StressThresholdNEUE = transgrp_1
GSw_PRM_StressThresholdLOLH = transgrp_2.4
GSw_PRM_StressThresholdLOLE = transgrp_0.1
GSw_PRM_StressThresholdLOLD = transgrp_0.1
GSw_PRM_StressThresholdDuration = transgrp_12
GSw_PRM_StressThresholdDepth = transgrp_0.1

CVAR/NCVAR threshold examples:

GSw_PRM_StressThresholdCVAR = country_50000_CVAR_cvar
GSw_PRM_StressThresholdNCVAR = transgrp_10_NCVAR_cvar

Example metric list:

GSw_PRM_StressThresholdMetrics = NEUE/LOLH/LOLE/LOLD/Duration/Depth/CVAR/NCVAR

The default remains unchanged unless CVAR/NCVAR are explicitly requested.

Testing

Added and ran a MultiMetricRA + CVAR/NCVAR test case instead of relying on the Pacific-only test case.

Test case:

NY_CVAR_debug_merged_MultiMetricRA

The case produced the expected standard MultiMetricRA outputs:

ra_metrics_{t}i{iteration}.csv
eue_events_{t}i{iteration}.csv

and the new CVAR/NCVAR check-only outputs:

cvar_{t}i{iteration}.csv
ncvar_{t}i{iteration}.csv
cvar_target_check_{t}i{iteration}.csv

Example result interpretation from the test case:

The country-level CVAR target passed because the annualized absolute tail-risk shortfall was below the MWh threshold.
The transgrp-level NCVAR target failed because the normalized tail-risk shortfall exceeded the ppm threshold.
This confirms that CVAR/NCVAR target checks are being evaluated and reported separately from stress-period selection.

Because CVAR/NCVAR are check-only in this PR, a failed CVAR/NCVAR target check does not add stress periods or trigger a PRM update.

Checklist for author

Details to double-check

General information to guide review

Zero impact on results of default case
No large data file(s) added/modified
No substantive impact on runtime for full-US reference case when CVAR/NCVAR are not enabled
No substantive impact on folder size for full-US reference case when CVAR/NCVAR are not enabled
No change to process flow (runreeds.py, reeds/core/solve/solve.py)
No change to code organization
No change to package requirements (environment.yml or Project.toml)

Did you use LLM tools (chatbot or copilot) in the preparation of this PR? If so, describe how

Used ChatGPT to help draft PR text and troubleshoot merge-conflict resolution. Code changes and test results were reviewed by the author.

Tag points of contact here if you would like additional review of the relevant parts of the model

bsergi

Here are some initial thoughts from a quick look. Happy to touch base when you get back to discuss any of these. Let me know when you've taken another pass here and I can review again.

One high-level comment: can you organize the PR summary text a bit more? The technical details section has a lot of information but is a bit difficult to follow. Also, I didn't quite understand your test output GSw_PRM_StressThreshold = country_-1_NCVAR_cvar failed. What does it mean to have a -1 threshold? Ultimately for the PR we'll also want a comparison of default case from cases.csv from both this branch and main to show there weren't any changes.

bsergi · 2026-06-18T14:54:30Z

revert the changes to this file before merging

bsergi · 2026-06-18T14:56:39Z

            required = false
        "--write_shortfall_samples"
-            help = "Write the sample-level shortfall"
+            help = "Write the sample-level shortfall (hourly, large)"


What are the large and small designations? I might suggest just "Write per-sample hourly shortfall by region" for this one and "Write per-sample total shortfall by region" for the one below.

bsergi · 2026-06-18T14:57:02Z

            default = 0
            required = false
+        "--write_shortfall_samples_totals"
+            help = "Write per-sample total shortfall by region (small)"


this is total over the entire PRAS time period and not annual, right?

bsergi · 2026-06-18T14:58:35Z

+        _,_,_,_,energyunit = PRAS.get_params(sys)
+        alpha = Float64(args["cvar_alpha"])
+        cvar_obj = PRAS.CVAR(energyunit, results["short_samples"], alpha)
+        @info(cvar_obj)


I think PRAS is setup so that the above info statements provide context (e.g., @info "$(PRAS.EUE(results["short"]))" results in EUE = 851000±5000 MWh/131400h MWh) being printed). Does this do something similar?

bsergi · 2026-06-18T15:47:19Z

-        end
-        ## Write it
+        sf = results["short_samples"]
+        region_names = sf.regions.names


the previous code filtered using sys.regions.names. I think this was because there are some pseudo regions for DC converter stations that don't have load and thus don't need to be included here, so we may want to continue to drop them (@patrickbrown4 might be able to weigh in on this one).

bsergi · 2026-06-18T16:27:28Z

+    return x.sort_values(ascending=False).iloc[:n_tail].mean()
+
+
+def get_annual_cvar_stress_metric(case, t, stress_metric='NCVAR', iteration=0, alpha=0.95):


this function and the next one get pretty long, so it would be good to add comments throughout to provide some documentation on what they are doing.

bsergi · 2026-06-18T16:31:40Z

+    x = pd.Series(samples).dropna().astype(float)
+    if x.empty:
+        return np.nan
+    if not (0 <= alpha < 1):


Feels like this check might fit better in get_cvar_alpha. An even better approach might be to add a check before the ReEDS run which would avoid having runs get this far and then running into an error (best place for that would probably be here:

ReEDS/runreeds.py

Line 222 in 1a8ab63

def check_compatibility(sw):

)

bsergi · 2026-06-18T16:39:35Z

+
+def get_annual_cvar_stress_metric(case, t, stress_metric='NCVAR', iteration=0, alpha=0.95):
+    stress_metric = stress_metric.upper()
+    if stress_metric not in CVAR_METRICS:


The multi-metrics PR (#99) now drops the metric tag in the switch value; for example, GSw_PRM_StressThresholdNEUE now just has a value of transgrp_1_sum instead of transgrp_1_EUE_sum (see here). I think we could do the same in this PR and then you could drop the CVAR_METRICS and the check and just iterate over whichever switches are activated.

bsergi · 2026-06-18T16:48:21Z

-                                                                   )
+        for criterion in sw[f'GSw_PRM_StressThreshold{metric}'].split('/'):
+            print(f"Evaluating GSw_PRM_StressThreshold {metric} with criterion: {criterion}")
+            stress_criteria = _evaluate_stress_threshold_criterion(stress_criteria, criterion, sw, t, iteration, dfenergy_r, stressperiods_this_iteration,


This line looks like just a formatting change--I'd suggest keeping as the original had it.

bsergi · 2026-06-18T16:53:02Z

+    return pd.concat(_metric, names=['level','metric','region']).rename(stress_metric)
+
+
+def evaluate_cvar_target_check(sw, t, iteration=0, stress_metrics=None):


Much of this function, including the core check of failed = this_test.loc[this_test > threshold], seems to replicate behavior in _evaluate_stress_threshold_criterion. Do you think we could merge the CVAR treatment into that existing function and just have some control statements to turn off adding new stress periods for now when using CVAR metrics? It's possible having a second function makes the most sense here, but I worry a little bit about maintaining two similar functions.

Bumps the actions-non-breaking group with 2 updates: [actions/checkout](https://github.com/actions/checkout) and [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv). Updates `actions/checkout` from 6.0.2 to 6.0.3 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@de0fac2...df4cb1c) Updates `astral-sh/setup-uv` from 8.1.0 to 8.2.0 - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](astral-sh/setup-uv@0880764...fac544c) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 6.0.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions-non-breaking - dependency-name: astral-sh/setup-uv dependency-version: 8.2.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions-non-breaking ... Signed-off-by: dependabot[bot] <[email protected]>

…_actions/actions-non-breaking-0dffccaecd build(deps): bump the actions-non-breaking group with 2 updates

…alculations and differentiate LOLE and LOLD

…outputs

…ents

…get_longest_events()

…_stressperiods()

…ss_periods plot

…tify new stress periods

…EDS-Model#123) * Remove instances of GSw_RegionResolution and agglevel variables * Delete hierarchy_from134.csv files * Clarify disaggregation docstrings and rename PCA_REG to legacy_ba * Cleanup * Undo rounding * Undo rounding * Revert changes * Rename PCA_REG to legacy_ba * bugfix: unitdata.csv already has 'r' column now * Misc. bugfixes/cleanup * Cleanup * Cleanup * bugfix: drop md5 and lat/lon columns from hierarchy in get_dfmap * Delete code writing hierarchy_with_res.csv * add hurdlereg for z90/hierarchy.csv; add z90 to allowed GSw_ZoneSet choices * z90 hierarchy.csv: fix El Paso NERC region * z90 hierarchy.csv: change '&' to 'and' in hurdlereg for GAMS * Update reeds/input_processing/copy_files.py Co-authored-by: Patrick Brown <[email protected]> * Remove 'regions' argument from main() * Update reeds/input_processing/copy_files.py Co-authored-by: Patrick Brown <[email protected]> * Delete commented out itlgrp constraint code * Remove remaining references to hierarchy_from134.csv * Delete 'aggreg' column * Delete modeled_regions.csv and references * Replace load allocation method switch compatibility check with warning in zones/README.MD * Add zoneset config and broaden definition of county-level zones * Add type indicators * Fix formatting * describe zoneset_config.yaml in inputs/zones/README.md * Remove now-inaccurate comment * Remove now-redundant special handling for county-level runs * Revert "Remove now-redundant special handling for county-level runs" This reverts commit a01d4be. * Remove now-redundant special handling for county-level runs --------- Co-authored-by: Patrick Brown <[email protected]>

…t_eue_events(); silence map_outagerate_new_stressperiods() if empty

…ull failures in plot_eue_events()

…EDS-Model#132) * bugfix: get_zonemap() output index is no longer nameless * Refactor workflow to create outputs with old format * Fix handling of kwargs in get_zonemap() * Remove unused row from level map * Bugfix --------- Co-authored-by: Kodi Obika <[email protected]> Co-authored-by: Kodi Obika <[email protected]>

Updated format of documentation on the setup page

…opower (ReEDS-Model#90) * Duplicate all changes from branch in private repo: - Add PSH data from ORNL (to replace data in repo) - Change relevant instances of `storage_duration(i)` with `storage_duration_m(i,v,r)` * Remove IHA dataset and replace with ORNL dataset * Update data source of cap_existing_psh.csv in sources.csv * Add storage_duration_m parameter to report_params.csv

Bumps the actions-major group with 1 update: [actions/checkout](https://github.com/actions/checkout). Updates `actions/checkout` from 6.0.3 to 7.0.0 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@df4cb1c...9c091bb) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions-major ... Signed-off-by: dependabot[bot] <[email protected]>

…_actions/actions-major-2217aebe03 build(deps): bump actions/checkout from 6.0.3 to 7.0.0 in the actions-major group

Burcin Cakir Erdener and others added 3 commits May 20, 2026 05:38

add CVaR reporting to run_pras.jl

29b6be0

fix fix

33d2a02

Add CVaR target checks for stress periods

e0ae25e

bcakire marked this pull request as draft May 28, 2026 06:19

merged with aa/multimetrics/06_02_26

9bc6424

patrickbrown4 self-requested a review June 2, 2026 23:50

abdelrahman-ayad added 6 commits June 17, 2026 11:46

initial implementation of outage duration

0a0d986

add OutageDuration and OutageMagnitude metrics

360cda4

add metrics to cases.csv and extend validation

704a0b1

extend stress_metric_units dict

047caf2

minor change

3afcdac

add LOLE

3c74b66

bsergi reviewed Jun 18, 2026

View reviewed changes

abdelrahman-ayad and others added 17 commits June 18, 2026 11:51

add LOLE

a0e0c0c

restore EUE for shoulder_periods calculation

8e539ab

Merge pull request ReEDS-Model#126 from ReEDS-Model/dependabot/github…

62f6381

…_actions/actions-non-breaking-0dffccaecd build(deps): bump the actions-non-breaking group with 2 updates

stress_periods.py: combine ra metrics into one csv; refactor metric c…

475a94a

…alculations and differentiate LOLE and LOLD

stress_periods.py: whitespace

d7f5dae

remove get_and_write_neue() and simplified national neue.csv/eue.csv …

60ed855

…outputs

calc_ra_metrics(): aggregate before the ra calculation functions

c3c4400

reformat RA threshold switches

dc8c59e

stress_periods.py: write EUE event list

7f4e779

add MultiMetricRA case to cases_test.csv

7d6a1a2

stress_periods.py: fix get_events() and calc_max_duration() for no ev…

f9da333

…ents

replace get_stress_metric_periods() with get_shortfall_periods() and …

cf7b5c8

…get_longest_events()

stress_periods.py: fix stress/shoulder format

3d525f2

move plot_stress_diagnostics() to diagnostic_plots.map_outagerate_new…

fc09c9a

…_stressperiods()

stress_periods.py: fix handling of no new stress periods

ff29cc5

stress_periods.py: streamline logs; diagnostic_plots.py: fix new_stre…

5388a3d

…ss_periods plot

patrickbrown4 and others added 30 commits June 24, 2026 15:50

fix some plots

50f0dc1

docs: adapt ra-flowcharts.png for multimetric RA

b125b1d

small formatting updates

3c530b4

move the internal doc up a level so it's more visible

93fed14

update DOI link

e6e004d

remove reference to POC list

cfe2093

remove broken link

3ab91b8

add profile to eue_events output

0a936a4

Merge aa/multi_metrics and integrate CVAR/NCVAR checks

6aec186

drop already-modeled stress hours from eue/lole profiles used to iden…

dfa9d27

…tify new stress periods

add reedsplots.plot_eue_events()

d88d298

Revert unrelated PR changes

65fd43a

Address CVAR/NCVAR review comments

e41316f

adapt plot_stressperiod_evolution() for different RA metrics; fix plo…

85acca5

…t_eue_events(); silence map_outagerate_new_stressperiods() if empty

revert compare_cases.py bypass

7780385

reedsplots.py: fix scaling in plot_stressperiod_evolution(); bypass h…

2bfecac

…ull failures in plot_eue_events()

fix get_longest_events() when there are no events

3eaf1fd

Merge branch 'main' into km/setup-docs

6db432b

Merge branch 'aa/metrics_extension' into pb/multimetric

92243bb

add figure to mac instructions for python setup

3873441

Merge branch 'main' into km/setup-docs

f11f125

Merge pull request ReEDS-Model#133 from ReEDS-Model/km/setup-docs

4d9607f

Updated format of documentation on the setup page

Merge pull request ReEDS-Model#136 from ReEDS-Model/dependabot/github…

a906387

…_actions/actions-major-2217aebe03 build(deps): bump actions/checkout from 6.0.3 to 7.0.0 in the actions-major group

Merge branch 'main' into aa/metrics_extension

f945065

merged with aa/metrics_extension

61146cb

runreeds.py updated and CVAR NCVAR added to cases_test

87826f9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Bc cvar updates#107

Bc cvar updates#107
bcakire wants to merge 60 commits into
ReEDS-Model:aa/multi_metricsfrom
bcakire:bc_cvar_updates

bcakire commented May 26, 2026 •

edited

Loading

Uh oh!

bsergi left a comment •

edited

Loading

Uh oh!

bsergi Jun 18, 2026

Uh oh!

bsergi Jun 18, 2026

Uh oh!

bsergi Jun 18, 2026

Uh oh!

bsergi Jun 18, 2026

Uh oh!

bsergi Jun 18, 2026

Uh oh!

bsergi Jun 18, 2026

Uh oh!

bsergi Jun 18, 2026

Uh oh!

bsergi Jun 18, 2026

Uh oh!

bsergi Jun 18, 2026

Uh oh!

bsergi Jun 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

		return x.sort_values(ascending=False).iloc[:n_tail].mean()


		def get_annual_cvar_stress_metric(case, t, stress_metric='NCVAR', iteration=0, alpha=0.95):

		return pd.concat(_metric, names=['level','metric','region']).rename(stress_metric)


		def evaluate_cvar_target_check(sw, t, iteration=0, stress_metrics=None):

Uh oh!

Conversation

bcakire commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What changed

run_pras.jl

stress_periods.py

ra_calcs.py

runreeds.py

Switches / cases

Testing

Checklist for author

Details to double-check

General information to guide review

Did you use LLM tools (chatbot or copilot) in the preparation of this PR? If so, describe how

Tag points of contact here if you would like additional review of the relevant parts of the model

Uh oh!

bsergi left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

bcakire commented May 26, 2026 •

edited

Loading

`run_pras.jl`

`stress_periods.py`

`ra_calcs.py`

`runreeds.py`

bsergi left a comment •

edited

Loading