Feat/metadata v2 schema - #898
Draft
AlexAxthelm wants to merge 6 commits into
Draft
Conversation
Foundation for epic #860 (best-effort / inheritance search): each of the 11 keyFeatures fields becomes an array of {sector, geography, value} entries so a pathway can hold different values for different parts of its coverage, and the #869 resolver can serve the most specific value for a search scope and fall back to broader ones. Also adds coreDrivers, dependencies, pathwayDescription and transitionAssessment. Additive only. v1 stays present and loadable, PathwayMetadataType still points at v1, and no data file changes here — the loader is repointed at v2 in a later commit once data carries the v2 $schema. Nothing consumes v2 yet.
Migrates the 4 ASEAN Centre for Energy and 3 IEA metadata files to pathwayMetadata.v2 via a new re-runnable codemod. src/data now holds 7 v2 and 49 v1 documents, which coexist because validateData routes each by its own $schema $id. Nothing reads v2 yet — the loader still points at v1, so the app is unchanged. All 7 resolve to a single widest-scope entry per keyFeature: cross-sector/South East Asia for ACE, cross-sector/Global for IEA. Both IEA files that carry pathwayOverview fold it into pathwayDescription as the lead paragraph; it has no readers in the app, so nothing observable moves. - scripts/codemod-v1-to-v2.ts splits v1's expertOverview into its three sections, wraps each keyFeature as one scoped entry, and scaffolds coreDrivers/dependencies. It skips files already on v2, so the remaining 49 are a re-run rather than a rewrite. A development tool only: there is no runtime v1 conversion, so un-migrated files simply will not load once the loader moves to v2. - The splitter accepts a bare line matching a section title as a heading. That exists for ACE-CNS-2024, whose "Core Drivers" heading lost its #### markers; without it, 1.5 KB of core-drivers prose folds into pathwayDescription and pushes it from 1204 to 2727 chars. - coreDrivers is scaffolded all-null per #858 rather than populated. The v1 "#### Core Drivers" prose does not map onto the 7 named fields mechanically: four paragraphs already exceed the 500-char cap, the italic labels ("Technology shifts", "Falling energy demand", "Economic growth") do not correspond 1:1 to the field names, and every section has unlabeled paragraphs with no destination. The codemod prints the prose it is not carrying so the hand-authoring ticket starts from the text. - transitionAssessment's maxLength goes 2500 -> 3000. 2500 was chosen for symmetry with pathwayDescription rather than measured; the longest section in the corpus is 2655 chars (ACE-RAS-2024), which made the codemod's own output invalid. pathwayDescription's 2500 is confirmed correct — the longest across all 56 files is 2459. Fixtures are added rather than converted, so the v1 fixtures stay v1 and the new coexistence tests can assert both halves. pathwayMetadata_v2_full carries several entries per field at different scopes, which #869 and #859 will need; _v2_minimal proves an all-empty keyFeatures document validates. One coexistence test documents a sharp edge deliberately: validateDataCollect filters entries to the single $id it is handed, so documents of the other version are dropped as neither valid nor invalid. That is what makes a mixed corpus work, and it is why repointing the loader has to report the count it skipped. Refs #858, #801. Co-Authored-By: Claude Opus 5 <[email protected]>
Points the loader at pathwayMetadata.v2 and moves every consumer onto the
scoped {sector, geography, value} shape. Only v2 documents load, so the app
now shows the 7 migrated ACE/IEA pathways; the 49 still on v1 are skipped
by $schema routing until they migrate.
That skip is silent by construction — validateDataCollect drops non-matching
documents as neither valid nor invalid — so pathwayMetadata.ts counts and
logs them. Without it, 49 missing pathways look like a data bug.
New src/utils/keyFeatureScope.ts answers "which entries apply to what the
user is looking at": containment on both axes, where cross-sector means the
union of the pathway's own declared sectors (not a universal match), and a
geography scope contains a query when the query's ISO set is a subset of the
entry's. Broader answers narrower, never the reverse. Deliberately no cost
model, no ranking, no fallback — that is #869, and it is what will turn a
non-match at the queried scope into a ranked broader-scope match rather than
an exclusion.
The emissionsTrajectory and policyAmbition facets now match like the sector
and metric facets — ANY/ALL over a value list, empty list meaning absent —
restricted to the entries whose scope contains the active sector/geography
selection. The two near-identical 30-line arms collapse into one helper.
concrete.includes(v) against an array is always false, so selecting either
returned zero pathways, and option building emitted "[object Object]".
Neither arm had any test coverage before — no filterPathways test passed
either filter — which is why the whole suite stayed green while both were
broken. Adding that coverage caught a regression that would otherwise have
shipped: in v1 a missing field contributed undefined, which
buildOptionsFromValues read as the absent bucket, but in v2 an empty entry
array contributes no elements, so the "None" option disappeared from both
dropdowns while the filter still honoured the token. Fixed with
withAbsentOption, matching how the sector facet does it.
Rendering keeps its current output. KeyFeatures reads through widestValue,
a deliberately provisional stand-in for #869's resolver: it picks the value
at the broadest declared scope, which reproduces v1 exactly for
codemod-migrated data (one entry, at its widest scope). #859 replaces it and
adds the badge naming the scope.
PathwayDetailPage renders pathwayDescription and transitionAssessment under
separate subheadings. v1's single expertOverview blob was three sections, so
rendering only the description would have visibly dropped the Application
to Transition Assessment text. The "Expert Overview" heading is left alone;
naming is #859's call.
Verified against the running app: 7 pathways load, the skip warning fires
without error, and all 11 key features on IEA-NZE render values matching the
source file — including the multi-select branch, which degrades silently
rather than throwing when handed the wrong shape.
Refs #858. Enables #869, #859.
Co-Authored-By: Claude Opus 5 <[email protected]>
Closes out #858's checklist. src/data/README.md described a format that no longer exists — and in the R example's case, one that never validated. Beyond the expected v1 leftovers (expertOverview, npm run json:check, a pbtar_schema.json link), the example used `name` as a bare string, the pre-#783 flat geography array, top-level publisher/publicationYear, and a `dataSource` field absent from every version of the schema. It would have failed against v1 as readily as against v2. Rewritten around v2: the two coexisting schema versions and the fact that only v2 documents are loaded, the scoped keyFeatures shape with its sentinels and the widest-scope rule, coreDrivers/dependencies/pathwayDescription/ transitionAssessment, the codemod for migrating an existing file, and the commands that actually exist. The R example is now verified rather than asserted: its blocks were extracted from this file, executed, and the resulting JSON validated against v2. Doing that corrected a wrong claim in an earlier draft — R's list() preserves NULL elements; the actual pitfall is jsonlite writing NULL as {}, which is why the helper passes null = "null". The validate_json R helper is dropped rather than repaired. The schema is split across common/*.json with absolute $refs that a single-URL jsonvalidate::json_validate() cannot resolve, so it documented a validation route that cannot succeed. Authors are pointed at npm run schema:check, which resolves the refs and additionally runs the cross-field scope checks that JSON Schema draft-07 cannot express. Tests: adds v2 counterparts to the existing v1 required-field cases — all 12 required fields, each of the 7 coreDrivers keys, unknown keys, and the dependencies enums. Also pins that pathwayDescription accepts null but not absence, the nullable-but-required distinction v2 relies on. The v1 REQ array still lists expertOverview on purpose: those cases validate v1 documents against v1, where it remains required. Finally, migrates the seven `keyFeatures: { emissionsTrajectory: "foo" }` stubs in ComparisonPage and PathwaySearch tests to the v2 shape. They kept their deliberately-invalid values, which exist to test degradation; the point is that a v1-shaped scalar in a v2 fixture silently exercises nothing. Refs #858. Co-Authored-By: Claude Opus 5 <[email protected]>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| <div class="card"> | ||
| <div | ||
| class="card-body" | ||
| id="" |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces pathway metadata schema v2 (scoped keyFeatures entries plus coreDrivers / dependencies / pathwayDescription), migrates a first slice of pathway JSON to v2, and updates validation, search, and rendering so the app can operate on v2-shaped data.
Changes:
- Add
pathwayMetadata.v2.json+ v2 scope subschemas, generated TS types, and generated HTML schema docs. - Add cross-field validation for scoped entries (sector/geography references) and update search facets + rendering to handle scoped keyFeature values.
- Migrate fixtures + a subset of real data files to v2, and add a codemod + tests to support migrating the remaining corpus.
Reviewed changes
Copilot reviewed 35 out of 39 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| testdata/valid/pathwayMetadata_v2_minimal.json | Adds a minimal v2-valid fixture for schema/validator coverage. |
| testdata/valid/pathwayMetadata_v2_full.json | Adds a comprehensive v2 fixture exercising scoped keyFeatures and new fields. |
| src/utils/validateScopes.ts | Implements cross-field scope reference validation for v2 (sector/geography must be declared). |
| src/utils/validateScopes.test.ts | Unit tests for scoped entry validation behavior and error reporting. |
| src/utils/validateData.test.tsx | Extends validation tests to cover v2 routing/coexistence and v2 required fields. |
| src/utils/searchUtils.ts | Updates facet option building and filtering to support scoped keyFeature entries. |
| src/utils/searchUtils.scopedFacets.test.ts | Adds targeted test coverage for scoped facets (emissionsTrajectory, policyAmbition). |
| src/utils/keyFeatureScope.ts | Adds helpers for reading scoped keyFeature entries (containment + widest-value fallback). |
| src/utils/keyFeatureScope.test.ts | Unit tests for scope containment, ISO resolution, and widest-value selection. |
| src/types/pathwayMetadata.v2.d.ts | Adds generated TS types for the v2 schema. |
| src/types/index.ts | Switches PathwayMetadataType to v2 and exports both v1/v2 types during migration. |
| src/types/common/scopeSector.v2.d.ts | Adds generated TS type for the v2 sector scope sentinel enum. |
| src/types/common/scopeGeography.v2.d.ts | Adds generated TS type for v2 geography scope (open string). |
| src/schema/pathwayMetadata.v2.test.ts | Adds schema self-guard tests to prevent keyFeatures wrapper/value drift. |
| src/schema/pathwayMetadata.v2.json | Introduces the v2 pathway metadata JSON Schema definition. |
| src/schema/common/scopeSector.v2.json | Adds schema for the v2 sector scope axis (sector names + cross-sector). |
| src/schema/common/scopeGeography.v2.json | Adds schema for the v2 geography scope axis (open string with basic guards). |
| src/schema/common/index.ts | Registers new v2 common schemas so AJV can resolve $refs. |
| src/pages/PathwaySearch.test.tsx | Updates integration test fixtures to use scoped v2 keyFeatures. |
| src/pages/PathwayDetailPage.tsx | Renders v2 pathwayDescription / transitionAssessment in place of v1 expertOverview. |
| src/pages/ComparisonPage.test.tsx | Updates comparison page test fixtures to use scoped v2 keyFeatures. |
| src/data/README.md | Updates contributor docs for v2 format, migration approach, and validation commands. |
| src/data/pathwayMetadata.ts | Switches loader to v2 schema and warns when v1 metadata files are skipped. |
| src/data/iea/IEA-STEPS-2024.json | Migrates this pathway metadata file to v2 structure. |
| src/data/iea/IEA-NZE-2024.json | Migrates this pathway metadata file to v2 structure. |
| src/data/iea/IEA-APS-2024.json | Migrates this pathway metadata file to v2 structure. |
| src/data/asean-centre-for-energy/ACE-RAS-2024.json | Migrates this pathway metadata file to v2 structure. |
| src/data/asean-centre-for-energy/ACE-CNS-2024.json | Migrates this pathway metadata file to v2 structure. |
| src/data/asean-centre-for-energy/ACE-BAS-2024.json | Migrates this pathway metadata file to v2 structure. |
| src/data/asean-centre-for-energy/ACE-ATS-2024.json | Migrates this pathway metadata file to v2 structure. |
| src/components/KeyFeatures.tsx | Updates rendering to read v2 scoped entries via a “widest value” helper. |
| src/components/KeyFeatures.test.tsx | Updates keyFeatures rendering tests to use v2 scoped-entry fixtures. |
| scripts/schema-check-files.ts | Adds a second-pass v2 scope reference check after AJV validation. |
| scripts/codemod-v1-to-v2.ts | Adds a codemod to migrate v1 metadata docs to v2 format. |
| scripts/codemod-v1-to-v2.test.ts | Adds tests for the codemod’s section splitting and scope selection logic. |
| public/schema/scopeSector.v2.html | Adds generated HTML documentation for scopeSector.v2.json. |
| public/schema/scopeGeography.v2.html | Adds generated HTML documentation for scopeGeography.v2.json. |
| PLAN.md | Adds an implementation/migration plan and rationale for v2 schema + rollout. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+8
to
+12
| "minLength": 1, | ||
| "allOf": [ | ||
| { "pattern": "^(?!\\s*$).+" }, | ||
| { "not": { "pattern": "^[A-Za-z]{3}$" } } | ||
| ], |
Comment on lines
+43
to
+55
| function allowedGeographies(pathway: PathwayMetadataV2): Set<string> { | ||
| const allowed = new Set<string>([GLOBAL_SCOPE, CROSS_REGION]); | ||
| const geo = pathway.geography; | ||
| if (!geo || typeof geo !== "object") return allowed; | ||
| if (geo.regions) { | ||
| for (const [label, members] of Object.entries(geo.regions)) { | ||
| allowed.add(label); | ||
| if (Array.isArray(members)) members.forEach((m) => allowed.add(m)); | ||
| } | ||
| } | ||
| if (Array.isArray(geo.country)) geo.country.forEach((c) => allowed.add(c)); | ||
| return allowed; | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Related issues
Closes:
Testing
Checklist