Add custom-constraints-from-PLEXOS templater#112
Open
nick-gorman wants to merge 11 commits into
Open
Conversation
Splits the constraint definition (id + direction) from RHS values (per timeslice) and LHS terms. Membership and coefficients are taken from the PLEXOS constraint formulation (data/plexos/), with the IASR workbook "Build limits - REZs" tab as the cross-reference for narrative context. Hydrogen constraints are excluded as ISPyPSA does not currently support electrolyser load. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Both the parallel-path expansion rows and the new custom-constraints templater need the same representative timeslice vocabulary (peak_demand, summer_typical, winter_reference). Promote it to a single _CANONICAL_TIMESLICES constant in mappings so the two stay in lockstep. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Translates the PLEXOS extract into ISPyPSA's three custom-constraint tables in two passes: a literal PLEXOS->ISPyPSA translation, then a "common sense" injection of IASR new-entrant batteries for every REZ/sub-region whose generators participate in a constraint. PLEXOS' own battery participation is an opaque LP-variable layout we deliberately don't mirror (see #110 and #111). Ships a workbook-vs-PLEXOS validation harness and caches the two IASR generator-summary tables the templater depends on. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Splice the three custom-constraint tables into the new-format template, gated to sub_regions granularity: the constraints reference sub-region nodes, sub-regional flow paths and REZ-located units that have no meaningful representation once sub-regions are collapsed. Threads iasr_workbook_version through the orchestrator and its callers to select the PLEXOS extract. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
notes/ holds local-only exploration scripts and drafts that should never be committed. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Comments and docs must not point at gitignored paths (e.g. notes/); reference a GitHub issue instead. Codifies the convention so future contributions don't create dead links. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The PLEXOS extract script is now shared by the custom-constraints templater, where this coefficient appears as a constraint relaxation variable rather than a network-expansion decision variable. Re-applies the wording lost when the duplicate extract commit was dropped during the rebase onto main. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The custom-constraints templater adds new_entrants_summary and existing_committed_anticipated_additional_generator_summary to the new-format required-tables list, so the create_ispypsa_inputs doit task now declares a file dependency on them. main's frozen-7.5-cache CLI tests (test_create_ispypsa_inputs_new_table_formats.py) failed the dependency check because these two tables weren't in the committed cache. Copied from the real parsed 7.5 workbook tables (byte-identical provenance to the other 80 cache files). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
5895ffe to
91d060b
Compare
Codecov Report✅ All modified and coverable lines are covered by tests.
🚀 New features to boost your workflow:
|
create_ispypsa_inputs writes the custom-constraint tables at sub_regions (write_csvs is dict-driven), but list_templater_output_files only listed the five network tables, so the doit task never declared them as targets. They were written-but-untracked: deleting one would not trigger a rebuild, and downstream tasks consuming get_ispypsa_input_files() did not see them as dependencies. Make the new-format output list granularity-aware so the three tables are declared only at sub_regions, matching the templater's own gate (and never expected at coarser granularities, where they are not written). Adds a unit test pinning the granularity-aware output list and extends the new-format CLI test to assert the tables are written (no orphan LHS/RHS rows) at sub_regions and absent otherwise. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Locate the shipped extract with files("ispypsa.templater") rather than Path(__file__).parent, matching the resource lookup already used in local_cache.py and not relying on __file__.
Add a direct test for _plexos_extract_dir. The default-path branch was previously only reached through the create_ispypsa_inputs CLI, which runs in a subprocess and so never showed as covered; the new test exercises it in-process and guards that the three extract CSVs ship with the package.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The new-format path generates the three custom-constraint tables from the PLEXOS extract and returns before the only line that splices in manually_extracted_tables (create_template.py), so these hand-extracted 7.5 custom_constraints / _lhs / _rhs files were loaded by load_manually_extracted_tables and then ignored. They were a precursor the PLEXOS templater superseded. Full suite passes without them. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
ISPyPSA needs AEMO's REZ / sub-region export limits as custom constraints, but
they don't come out of the IASR workbook in a usable shape. AEMO does express
them in its PLEXOS model, as export-group constraints, so we extract them from
there. This PR adds the templater that turns that extract into ISPyPSA's three
custom-constraint tables:
custom_constraints: one row per constraint (id + direction)custom_constraints_lhs: the left-hand-side terms (which unit, what coefficient)custom_constraints_rhs: the per-timeslice limit on the rightIt sits behind the
use_new_table_formatflag (committed off) and only runs atsub_regionsgranularity.The bit worth understanding first
The RHS and most of the LHS are a straight translation from the PLEXOS extract.
The one judgement call is batteries, and it's worth knowing before you read
the code.
PLEXOS encodes battery participation as per-constraint LP variables (e.g.
SWQLD1 Battery - 2h) whose layout doesn't map cleanly onto any IASR unit (see#110). Rather than mirror that, the templater builds the LHS in
two passes:
nodes) into ISPyPSA's term schema, dropping anything that doesn't map to a
known unit.
new-entrant generator terms sit (REZ / sub-region), then add every IASR
new-entrant battery at those same locations.
One format note: the constraints come out with region-prefixed timeslices
(
qld_peak_demand) rather than the bare names (peak_demand) the rest of thetemplater uses. A follow-up PR will bring the rest of the transmission
functionality onto this format so the constraints flow through end-to-end.
Where it lives
Supporting changes
A few small things ride along: the templater is wired into
create_ispypsa_inputs_template(gated tosub_regions, its three outputstracked as task targets); the two IASR summary tables it needs are added to the
required-tables list; and the canonical timeslice vocabulary is promoted into
mappingsso the templater and the existing transmission code share one copy.Validation
Pulling these constraints out of PLEXOS takes enough translation that we didn't
want to take the output on trust. So the PR cross-checks the templater against a
second, fully independent reconstruction of each constraint built straight from
the IASR workbook (
scripts/workbook_extract_constraint.py). The goal is bothto confirm the PLEXOS route produces the right LHS and to understand exactly
where and why the two sources differ. The two routes share no code:
rez_group_constraint_summary, parses its Terms into(coefficient, body)pairs, and expands them. A REZ id (e.g.
Q1) becomes every generator,battery and electrolyser tagged with that REZ ID across the IASR unit summary
tables; an interconnector path id (e.g.
CQ-NQ) becomes a singlelink_flowterm. Coefficients can be implicit (
Q1is 1.0), signed (- CQ-NQis -1.0)or explicit (
0.78 * V7).test_custom_constraints_validation.pyruns both routes over five constraints(
NQ1,NET1,WV1,CQ1,MN1) and asserts the LHS term sets matchexactly, save for an explicit
EXPECTED_DELTAStable. A new mismatch in eitherdirection fails the test, and so does an expected one going missing.
The documented differences come down to two things:
fine-grained REZ ID and a coarser sub-region, and they usually agree. The
workbook route expands by REZ ID; PLEXOS includes existing plant by
sub-region. The handful of boundary plants where the two tags point
different places are the only generator diffs. For example
EMERASF1(REZQ4, sub-regionNQ) lands inNQ1under PLEXOS butCQ1under theworkbook, so the one fact shows up on both sides of the comparison.
Purchaserclass, which the templater drops by design (hydrogen demand isn't modelled),
so they appear as workbook-only
loadterms.Batteries don't appear as a difference either: the templater rebuilds battery
participation from the IASR new-entrant set rather than translating PLEXOS' own
battery layout, so neither PLEXOS' constraint-scoped battery variants nor its
quiet exclusion of 4h-duration batteries surface in the comparison.