Skip to content

Add custom-constraints-from-PLEXOS templater#112

Open
nick-gorman wants to merge 11 commits into
mainfrom
new-manually-extracted-tables
Open

Add custom-constraints-from-PLEXOS templater#112
nick-gorman wants to merge 11 commits into
mainfrom
new-manually-extracted-tables

Conversation

@nick-gorman
Copy link
Copy Markdown
Member

@nick-gorman nick-gorman commented May 28, 2026

What this does

ISPyPSA needs AEMO's REZ / sub-region export limits as custom constraints, but
they don't come out of the IASR workbook in a usable shape. AEMO does express
them in its PLEXOS model, as export-group constraints, so we extract them from
there. This PR adds the templater that turns that extract into ISPyPSA's three
custom-constraint tables:

  • custom_constraints: one row per constraint (id + direction)
  • custom_constraints_lhs: the left-hand-side terms (which unit, what coefficient)
  • custom_constraints_rhs: the per-timeslice limit on the right

It sits behind the use_new_table_format flag (committed off) and only runs at
sub_regions granularity.

The bit worth understanding first

The RHS and most of the LHS are a straight translation from the PLEXOS extract.
The one judgement call is batteries, and it's worth knowing before you read
the code.

PLEXOS encodes battery participation as per-constraint LP variables (e.g.
SWQLD1 Battery - 2h) whose layout doesn't map cleanly onto any IASR unit (see
#110). Rather than mirror that, the templater builds the LHS in
two passes:

  1. PLEXOS pass: translate the raw LHS rows (generators, batteries, lines,
    nodes) into ISPyPSA's term schema, dropping anything that doesn't map to a
    known unit.
  2. Battery injection: for each constraint, find where its surviving
    new-entrant generator terms sit (REZ / sub-region), then add every IASR
    new-entrant battery at those same locations.

One format note: the constraints come out with region-prefixed timeslices
(qld_peak_demand) rather than the bare names (peak_demand) the rest of the
templater uses. A follow-up PR will bring the rest of the transmission
functionality onto this format so the constraints flow through end-to-end.

Where it lives

ISPyPSA/
├─ src/ispypsa/templater/
│  ├─ custom_constraints_from_plexos.py   the templater (PLEXOS extract → the 3 tables)
│  ├─ plexos/7.5/*.csv                     the extract it reads at runtime (already on main)
│  ├─ create_template.py                   wires it in at sub_regions; tracks the 3 outputs
│  └─ mappings.py, transmission.py         now share one canonical-timeslice vocabulary
├─ scripts/
│  ├─ extract_plexos_constraints.py        PLEXOS XML → the extract above (already on main)
│  └─ workbook_extract_constraint.py       independent workbook-route LHS, validation only
└─ tests/test_templater/
   ├─ test_custom_constraints_from_plexos.py   templater unit tests
   ├─ test_custom_constraints_validation.py    PLEXOS route vs workbook route reconciliation
   └─ data/custom_constraints_validation/      fixtures (plexos/ + workbook/)

Supporting changes

A few small things ride along: the templater is wired into
create_ispypsa_inputs_template (gated to sub_regions, its three outputs
tracked as task targets); the two IASR summary tables it needs are added to the
required-tables list; and the canonical timeslice vocabulary is promoted into
mappings so the templater and the existing transmission code share one copy.

Validation

Pulling these constraints out of PLEXOS takes enough translation that we didn't
want to take the output on trust. So the PR cross-checks the templater against a
second, fully independent reconstruction of each constraint built straight from
the IASR workbook (scripts/workbook_extract_constraint.py). The goal is both
to confirm the PLEXOS route produces the right LHS and to understand exactly
where and why the two sources differ. The two routes share no code:

  • The PLEXOS route is the production templater reading the PLEXOS extract.
  • The workbook route looks each constraint up in
    rez_group_constraint_summary, parses its Terms into (coefficient, body)
    pairs, and expands them. A REZ id (e.g. Q1) becomes every generator,
    battery and electrolyser tagged with that REZ ID across the IASR unit summary
    tables; an interconnector path id (e.g. CQ-NQ) becomes a single link_flow
    term. Coefficients can be implicit (Q1 is 1.0), signed (- CQ-NQ is -1.0)
    or explicit (0.78 * V7).

test_custom_constraints_validation.py runs both routes over five constraints
(NQ1, NET1, WV1, CQ1, MN1) and asserts the LHS term sets match
exactly, save for an explicit EXPECTED_DELTAS table. A new mismatch in either
direction fails the test, and so does an expected one going missing.

The documented differences come down to two things:

  1. Geographic lookup for existing plant. Every IASR unit carries both a
    fine-grained REZ ID and a coarser sub-region, and they usually agree. The
    workbook route expands by REZ ID; PLEXOS includes existing plant by
    sub-region. The handful of boundary plants where the two tags point
    different places are the only generator diffs. For example EMERASF1 (REZ
    Q4, sub-region NQ) lands in NQ1 under PLEXOS but CQ1 under the
    workbook, so the one fact shows up on both sides of the comparison.
  2. Electrolysers. PLEXOS routes electrolyser load through its Purchaser
    class, which the templater drops by design (hydrogen demand isn't modelled),
    so they appear as workbook-only load terms.

Batteries don't appear as a difference either: the templater rebuilds battery
participation from the IASR new-entrant set rather than translating PLEXOS' own
battery layout, so neither PLEXOS' constraint-scoped battery variants nor its
quiet exclusion of 4h-duration batteries surface in the comparison.

nick-gorman and others added 8 commits June 3, 2026 11:21
Splits the constraint definition (id + direction) from RHS values (per
timeslice) and LHS terms. Membership and coefficients are taken from the
PLEXOS constraint formulation (data/plexos/), with the IASR workbook
"Build limits - REZs" tab as the cross-reference for narrative context.

Hydrogen constraints are excluded as ISPyPSA does not currently support
electrolyser load.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Both the parallel-path expansion rows and the new custom-constraints
templater need the same representative timeslice vocabulary
(peak_demand, summer_typical, winter_reference). Promote it to a single
_CANONICAL_TIMESLICES constant in mappings so the two stay in lockstep.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Translates the PLEXOS extract into ISPyPSA's three custom-constraint
tables in two passes: a literal PLEXOS->ISPyPSA translation, then a
"common sense" injection of IASR new-entrant batteries for every
REZ/sub-region whose generators participate in a constraint. PLEXOS' own
battery participation is an opaque LP-variable layout we deliberately
don't mirror (see #110 and #111). Ships a
workbook-vs-PLEXOS validation harness and caches the two IASR
generator-summary tables the templater depends on.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Splice the three custom-constraint tables into the new-format template,
gated to sub_regions granularity: the constraints reference sub-region
nodes, sub-regional flow paths and REZ-located units that have no
meaningful representation once sub-regions are collapsed. Threads
iasr_workbook_version through the orchestrator and its callers to select
the PLEXOS extract.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
notes/ holds local-only exploration scripts and drafts that should never
be committed.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Comments and docs must not point at gitignored paths (e.g. notes/);
reference a GitHub issue instead. Codifies the convention so future
contributions don't create dead links.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The PLEXOS extract script is now shared by the custom-constraints templater, where this coefficient appears as a constraint relaxation variable rather than a network-expansion decision variable. Re-applies the wording lost when the duplicate extract commit was dropped during the rebase onto main.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The custom-constraints templater adds new_entrants_summary and existing_committed_anticipated_additional_generator_summary to the new-format required-tables list, so the create_ispypsa_inputs doit task now declares a file dependency on them. main's frozen-7.5-cache CLI tests (test_create_ispypsa_inputs_new_table_formats.py) failed the dependency check because these two tables weren't in the committed cache. Copied from the real parsed 7.5 workbook tables (byte-identical provenance to the other 80 cache files).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@nick-gorman nick-gorman force-pushed the new-manually-extracted-tables branch from 5895ffe to 91d060b Compare June 3, 2026 02:01
@codecov
Copy link
Copy Markdown

codecov Bot commented Jun 3, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines Coverage Δ
src/ispypsa/iasr_table_caching/local_cache.py 74.28% <ø> (ø)
src/ispypsa/templater/create_template.py 92.50% <100.00%> (+3.31%) ⬆️
...spypsa/templater/custom_constraints_from_plexos.py 100.00% <100.00%> (ø)
src/ispypsa/templater/mappings.py 100.00% <100.00%> (ø)
src/ispypsa/templater/transmission.py 98.70% <100.00%> (-0.01%) ⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

create_ispypsa_inputs writes the custom-constraint tables at sub_regions (write_csvs is dict-driven), but list_templater_output_files only listed the five network tables, so the doit task never declared them as targets. They were written-but-untracked: deleting one would not trigger a rebuild, and downstream tasks consuming get_ispypsa_input_files() did not see them as dependencies. Make the new-format output list granularity-aware so the three tables are declared only at sub_regions, matching the templater's own gate (and never expected at coarser granularities, where they are not written).

Adds a unit test pinning the granularity-aware output list and extends the new-format CLI test to assert the tables are written (no orphan LHS/RHS rows) at sub_regions and absent otherwise.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
nick-gorman and others added 2 commits June 4, 2026 11:31
Locate the shipped extract with files("ispypsa.templater") rather than Path(__file__).parent, matching the resource lookup already used in local_cache.py and not relying on __file__.

Add a direct test for _plexos_extract_dir. The default-path branch was previously only reached through the create_ispypsa_inputs CLI, which runs in a subprocess and so never showed as covered; the new test exercises it in-process and guards that the three extract CSVs ship with the package.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The new-format path generates the three custom-constraint tables from the PLEXOS extract and returns before the only line that splices in manually_extracted_tables (create_template.py), so these hand-extracted 7.5 custom_constraints / _lhs / _rhs files were loaded by load_manually_extracted_tables and then ignored. They were a precursor the PLEXOS templater superseded. Full suite passes without them.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant