Feat/metadata v2 schema - #898
Open
AlexAxthelm wants to merge 11 commits into
Open
Conversation
Foundation for epic #860 (best-effort / inheritance search): each of the 11 keyFeatures fields becomes an array of {sector, geography, value} entries so a pathway can hold different values for different parts of its coverage, and the #869 resolver can serve the most specific value for a search scope and fall back to broader ones. Also adds coreDrivers, dependencies, pathwayDescription and transitionAssessment. Additive only. v1 stays present and loadable, PathwayMetadataType still points at v1, and no data file changes here — the loader is repointed at v2 in a later commit once data carries the v2 $schema. Nothing consumes v2 yet.
Migrates the 4 ASEAN Centre for Energy and 3 IEA metadata files to pathwayMetadata.v2 via a new re-runnable codemod. src/data now holds 7 v2 and 49 v1 documents, which coexist because validateData routes each by its own $schema $id. Nothing reads v2 yet — the loader still points at v1, so the app is unchanged. All 7 resolve to a single widest-scope entry per keyFeature: cross-sector/South East Asia for ACE, cross-sector/Global for IEA. Both IEA files that carry pathwayOverview fold it into pathwayDescription as the lead paragraph; it has no readers in the app, so nothing observable moves. - scripts/codemod-v1-to-v2.ts splits v1's expertOverview into its three sections, wraps each keyFeature as one scoped entry, and scaffolds coreDrivers/dependencies. It skips files already on v2, so the remaining 49 are a re-run rather than a rewrite. A development tool only: there is no runtime v1 conversion, so un-migrated files simply will not load once the loader moves to v2. - The splitter accepts a bare line matching a section title as a heading. That exists for ACE-CNS-2024, whose "Core Drivers" heading lost its #### markers; without it, 1.5 KB of core-drivers prose folds into pathwayDescription and pushes it from 1204 to 2727 chars. - coreDrivers is scaffolded all-null per #858 rather than populated. The v1 "#### Core Drivers" prose does not map onto the 7 named fields mechanically: four paragraphs already exceed the 500-char cap, the italic labels ("Technology shifts", "Falling energy demand", "Economic growth") do not correspond 1:1 to the field names, and every section has unlabeled paragraphs with no destination. The codemod prints the prose it is not carrying so the hand-authoring ticket starts from the text. - transitionAssessment's maxLength goes 2500 -> 3000. 2500 was chosen for symmetry with pathwayDescription rather than measured; the longest section in the corpus is 2655 chars (ACE-RAS-2024), which made the codemod's own output invalid. pathwayDescription's 2500 is confirmed correct — the longest across all 56 files is 2459. Fixtures are added rather than converted, so the v1 fixtures stay v1 and the new coexistence tests can assert both halves. pathwayMetadata_v2_full carries several entries per field at different scopes, which #869 and #859 will need; _v2_minimal proves an all-empty keyFeatures document validates. One coexistence test documents a sharp edge deliberately: validateDataCollect filters entries to the single $id it is handed, so documents of the other version are dropped as neither valid nor invalid. That is what makes a mixed corpus work, and it is why repointing the loader has to report the count it skipped. Refs #858, #801. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Points the loader at pathwayMetadata.v2 and moves every consumer onto the
scoped {sector, geography, value} shape. Only v2 documents load, so the app
now shows the 7 migrated ACE/IEA pathways; the 49 still on v1 are skipped
by $schema routing until they migrate.
That skip is silent by construction — validateDataCollect drops non-matching
documents as neither valid nor invalid — so pathwayMetadata.ts counts and
logs them. Without it, 49 missing pathways look like a data bug.
New src/utils/keyFeatureScope.ts answers "which entries apply to what the
user is looking at": containment on both axes, where cross-sector means the
union of the pathway's own declared sectors (not a universal match), and a
geography scope contains a query when the query's ISO set is a subset of the
entry's. Broader answers narrower, never the reverse. Deliberately no cost
model, no ranking, no fallback — that is #869, and it is what will turn a
non-match at the queried scope into a ranked broader-scope match rather than
an exclusion.
The emissionsTrajectory and policyAmbition facets now match like the sector
and metric facets — ANY/ALL over a value list, empty list meaning absent —
restricted to the entries whose scope contains the active sector/geography
selection. The two near-identical 30-line arms collapse into one helper.
concrete.includes(v) against an array is always false, so selecting either
returned zero pathways, and option building emitted "[object Object]".
Neither arm had any test coverage before — no filterPathways test passed
either filter — which is why the whole suite stayed green while both were
broken. Adding that coverage caught a regression that would otherwise have
shipped: in v1 a missing field contributed undefined, which
buildOptionsFromValues read as the absent bucket, but in v2 an empty entry
array contributes no elements, so the "None" option disappeared from both
dropdowns while the filter still honoured the token. Fixed with
withAbsentOption, matching how the sector facet does it.
Rendering keeps its current output. KeyFeatures reads through widestValue,
a deliberately provisional stand-in for #869's resolver: it picks the value
at the broadest declared scope, which reproduces v1 exactly for
codemod-migrated data (one entry, at its widest scope). #859 replaces it and
adds the badge naming the scope.
PathwayDetailPage renders pathwayDescription and transitionAssessment under
separate subheadings. v1's single expertOverview blob was three sections, so
rendering only the description would have visibly dropped the Application
to Transition Assessment text. The "Expert Overview" heading is left alone;
naming is #859's call.
Verified against the running app: 7 pathways load, the skip warning fires
without error, and all 11 key features on IEA-NZE render values matching the
source file — including the multi-select branch, which degrades silently
rather than throwing when handed the wrong shape.
Refs #858. Enables #869, #859.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Closes out #858's checklist. src/data/README.md described a format that no longer exists — and in the R example's case, one that never validated. Beyond the expected v1 leftovers (expertOverview, npm run json:check, a pbtar_schema.json link), the example used `name` as a bare string, the pre-#783 flat geography array, top-level publisher/publicationYear, and a `dataSource` field absent from every version of the schema. It would have failed against v1 as readily as against v2. Rewritten around v2: the two coexisting schema versions and the fact that only v2 documents are loaded, the scoped keyFeatures shape with its sentinels and the widest-scope rule, coreDrivers/dependencies/pathwayDescription/ transitionAssessment, the codemod for migrating an existing file, and the commands that actually exist. The R example is now verified rather than asserted: its blocks were extracted from this file, executed, and the resulting JSON validated against v2. Doing that corrected a wrong claim in an earlier draft — R's list() preserves NULL elements; the actual pitfall is jsonlite writing NULL as {}, which is why the helper passes null = "null". The validate_json R helper is dropped rather than repaired. The schema is split across common/*.json with absolute $refs that a single-URL jsonvalidate::json_validate() cannot resolve, so it documented a validation route that cannot succeed. Authors are pointed at npm run schema:check, which resolves the refs and additionally runs the cross-field scope checks that JSON Schema draft-07 cannot express. Tests: adds v2 counterparts to the existing v1 required-field cases — all 12 required fields, each of the 7 coreDrivers keys, unknown keys, and the dependencies enums. Also pins that pathwayDescription accepts null but not absence, the nullable-but-required distinction v2 relies on. The v1 REQ array still lists expertOverview on purpose: those cases validate v1 documents against v1, where it remains required. Finally, migrates the seven `keyFeatures: { emissionsTrajectory: "foo" }` stubs in ComparisonPage and PathwaySearch tests to the v2 shape. They kept their deliberately-invalid values, which exist to test degradation; the point is that a v1-shaped scalar in a v2 fixture silently exercises nothing. Refs #858. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces pathway metadata schema v2 (scoped keyFeatures entries plus coreDrivers / dependencies / pathwayDescription), migrates a first slice of pathway JSON to v2, and updates validation, search, and rendering so the app can operate on v2-shaped data.
Changes:
- Add
pathwayMetadata.v2.json+ v2 scope subschemas, generated TS types, and generated HTML schema docs. - Add cross-field validation for scoped entries (sector/geography references) and update search facets + rendering to handle scoped keyFeature values.
- Migrate fixtures + a subset of real data files to v2, and add a codemod + tests to support migrating the remaining corpus.
Reviewed changes
Copilot reviewed 35 out of 39 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| testdata/valid/pathwayMetadata_v2_minimal.json | Adds a minimal v2-valid fixture for schema/validator coverage. |
| testdata/valid/pathwayMetadata_v2_full.json | Adds a comprehensive v2 fixture exercising scoped keyFeatures and new fields. |
| src/utils/validateScopes.ts | Implements cross-field scope reference validation for v2 (sector/geography must be declared). |
| src/utils/validateScopes.test.ts | Unit tests for scoped entry validation behavior and error reporting. |
| src/utils/validateData.test.tsx | Extends validation tests to cover v2 routing/coexistence and v2 required fields. |
| src/utils/searchUtils.ts | Updates facet option building and filtering to support scoped keyFeature entries. |
| src/utils/searchUtils.scopedFacets.test.ts | Adds targeted test coverage for scoped facets (emissionsTrajectory, policyAmbition). |
| src/utils/keyFeatureScope.ts | Adds helpers for reading scoped keyFeature entries (containment + widest-value fallback). |
| src/utils/keyFeatureScope.test.ts | Unit tests for scope containment, ISO resolution, and widest-value selection. |
| src/types/pathwayMetadata.v2.d.ts | Adds generated TS types for the v2 schema. |
| src/types/index.ts | Switches PathwayMetadataType to v2 and exports both v1/v2 types during migration. |
| src/types/common/scopeSector.v2.d.ts | Adds generated TS type for the v2 sector scope sentinel enum. |
| src/types/common/scopeGeography.v2.d.ts | Adds generated TS type for v2 geography scope (open string). |
| src/schema/pathwayMetadata.v2.test.ts | Adds schema self-guard tests to prevent keyFeatures wrapper/value drift. |
| src/schema/pathwayMetadata.v2.json | Introduces the v2 pathway metadata JSON Schema definition. |
| src/schema/common/scopeSector.v2.json | Adds schema for the v2 sector scope axis (sector names + cross-sector). |
| src/schema/common/scopeGeography.v2.json | Adds schema for the v2 geography scope axis (open string with basic guards). |
| src/schema/common/index.ts | Registers new v2 common schemas so AJV can resolve $refs. |
| src/pages/PathwaySearch.test.tsx | Updates integration test fixtures to use scoped v2 keyFeatures. |
| src/pages/PathwayDetailPage.tsx | Renders v2 pathwayDescription / transitionAssessment in place of v1 expertOverview. |
| src/pages/ComparisonPage.test.tsx | Updates comparison page test fixtures to use scoped v2 keyFeatures. |
| src/data/README.md | Updates contributor docs for v2 format, migration approach, and validation commands. |
| src/data/pathwayMetadata.ts | Switches loader to v2 schema and warns when v1 metadata files are skipped. |
| src/data/iea/IEA-STEPS-2024.json | Migrates this pathway metadata file to v2 structure. |
| src/data/iea/IEA-NZE-2024.json | Migrates this pathway metadata file to v2 structure. |
| src/data/iea/IEA-APS-2024.json | Migrates this pathway metadata file to v2 structure. |
| src/data/asean-centre-for-energy/ACE-RAS-2024.json | Migrates this pathway metadata file to v2 structure. |
| src/data/asean-centre-for-energy/ACE-CNS-2024.json | Migrates this pathway metadata file to v2 structure. |
| src/data/asean-centre-for-energy/ACE-BAS-2024.json | Migrates this pathway metadata file to v2 structure. |
| src/data/asean-centre-for-energy/ACE-ATS-2024.json | Migrates this pathway metadata file to v2 structure. |
| src/components/KeyFeatures.tsx | Updates rendering to read v2 scoped entries via a “widest value” helper. |
| src/components/KeyFeatures.test.tsx | Updates keyFeatures rendering tests to use v2 scoped-entry fixtures. |
| scripts/schema-check-files.ts | Adds a second-pass v2 scope reference check after AJV validation. |
| scripts/codemod-v1-to-v2.ts | Adds a codemod to migrate v1 metadata docs to v2 format. |
| scripts/codemod-v1-to-v2.test.ts | Adds tests for the codemod’s section splitting and scope selection logic. |
| public/schema/scopeSector.v2.html | Adds generated HTML documentation for scopeSector.v2.json. |
| public/schema/scopeGeography.v2.html | Adds generated HTML documentation for scopeGeography.v2.json. |
| PLAN.md | Adds an implementation/migration plan and rationale for v2 schema + rollout. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Expected version change and release notes:1.16.0-dev.16 (v1.16.0-dev.15...feat/metadata-v2-schema ) (2026-08-19T12:08 UTC)Features
DocsOther
|
|
Azure Static Web Apps: Your stage site is ready! Visit it here: https://proud-glacier-0f640931e-898.westus2.2.azurestaticapps.net |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 35 out of 39 changed files in this pull request and generated no new comments.
Suppressed comments (2)
src/schema/pathwayMetadata.v2.json:37
- Schema description has a duplicated word: "Type of the pathway pathway." This will propagate into generated TypeScript types and the HTML schema docs on the next regen, so it’s worth correcting in the source schema.
"pathwayType": {
"description": "Type of the pathway pathway.",
"type": "string",
"enum": ["Normative", "Exploratory", "Predictive"]
src/schema/pathwayMetadata.v2.json:340
- The policyAmbition field description is missing punctuation/wording between “ones” and “Scoped”, reading “… beyond currently legislated ones Scoped: …”. This is grammatically incorrect and will flow into generated types/docs.
"policyAmbition": {
"description": "Represents the overall stringency and intent of modeled policies relative to climate targets, often reflecting if and how far the included policies go beyond currently legislated ones Scoped: see keyFeatures.",
"type": "array",
added 2 commits
August 19, 2026 13:50
|
Azure Static Web Apps: Your stage site is ready! Visit it here: https://proud-glacier-0f640931e-898.westus2.2.azurestaticapps.net |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 34 out of 38 changed files in this pull request and generated no new comments.
Suppressed comments (1)
src/utils/validateScopes.ts:55
cross-regionis treated as an always-allowed geography sentinel here (new Set([GLOBAL_SCOPE, CROSS_REGION])). But per the scopeGeography.v2 schema comment,cross-regionis reserved for non-global multi-region aggregates. Allowing it unconditionally means a v2 doc can passvalidateScopedEntrieswithgeography: "cross-region"even when the pathway isglobal: trueor has no declared ISO coverage; in that case the app-side resolver (keyFeatureScope.entryISOSet) resolves it to the pathway’s ISO coverage (often empty for global-only pathways) and the entry matches nothing.
Consider only allowing cross-region when the pathway is not global and declares more than one region/country (i.e. when it can actually represent a meaningful aggregate scope).
|
Azure Static Web Apps: Your stage site is ready! Visit it here: https://proud-glacier-0f640931e-898.westus2.2.azurestaticapps.net |
AlexAxthelm
marked this pull request as ready for review
August 19, 2026 15:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Alex's TL;DR here, Claude's summary below:
publicchanges (generated docs)src/schemahave useful docstrings that you may want to look at in particularAt this point, this is a "plumbing" PR. I'm intending that this will have the UI one, and one for a better import of the touched data files (not just placeholder values), extended on top of it and then we can merge that whole block as one (using the new stacks feature on GH)
Summary
Introduces
pathwayMetadata.v2— the foundation for epic #860 (best-effort /inheritance search). Each of the 11
keyFeaturesbecomes an array of{sector, geography, value}entries so a pathway can hold different values fordifferent parts of its coverage, plus new
coreDrivers,dependencies,pathwayDescriptionandtransitionAssessmentfields.Scope was deliberately narrowed from the ticket: 7 of 56 data files (4 ACE,
3 IEA) are migrated here; the other 49 follow in a separate PR. v1 and v2 coexist
via
$schema$idrouting, but only v2 documents are loaded, so the appcurrently shows 7 pathways and logs how many were skipped.
Reviewing this
The diff is ~27.6k added lines, but 23.4k of that is generated — please skim
rather than read:
.d.tsThe parts worth real attention:
src/schema/pathwayMetadata.v2.json+common/scope{Sector,Geography}.v2.jsonsrc/utils/keyFeatureScope.ts— scope containment for searchsrc/utils/validateScopes.ts— the cross-field check draft-07 can't expressscripts/codemod-v1-to-v2.ts— the v1→v2 migrationsrc/utils/searchUtils.ts— the two facets that had to changeKnown gaps, deliberately left
coreDriversis scaffolded all-null. The v1 "Core Drivers" prose doesn't maponto the 7 named fields mechanically (4 paragraphs exceed the 500-char cap, the
labels don't correspond 1:1). Consequence: those 7 detail pages show ~1 KB less
text than before. Needs an authoring pass.
entry at its widest scope, so
keyFeatureScopenever actually narrows. The testsare the only place the mechanism is currently observable.
Related issues
Refs: #858, #801 · Enables: #869, #859
Testing
search facets, which previously had none — that's why the v1→v2 shape change
broke them silently.
npm run schema:checkvalidates all 81documents including the cross-field scope checks.
values matching the source files, both new prose fields display.
Checklist
src/data/README.mdrewritten for v2)Claude Code, reviewed commit-by-commit.