Psy6 - #8
Open
jd-lara wants to merge 10 commits into
Open
Conversation
- replace the psy5 Matpower parser with PowerFlowFileParser's in-memory OpenAPI document pipeline (psy6 dropped PSY's own parser) - add explicit unit tags to every convertible getter/setter call (psy6 removed the stateful unit system) - export via to_file(...; unit_system=:device_base) instead of native to_json - add Python/load_cats.py to validate the export against power_openapi_models - correct ramp rate and min up/down time assumptions in generator_types.jl against RTS-GMLC's public unit data; document nuclear's intentional deviation from that data
GitHub blocks new LFS object uploads on this repo (a fork of a public repo), and detaching the fork requires a GitHub Support ticket, so keep the time series data external and automate fetching it instead. Verified against the Google Drive folder linked in the README: the HourlyProduction file's checksum matches what's already in data/.
[sources] pointed at /home/jdlara/Sienna_work/psy6/..., which only resolved on this machine. Switched to git url+rev (matching the branches each package is actually developed on) so anyone can instantiate this env from a fresh clone. Slower first instantiate (clones the stack from GitHub instead of reusing local checkouts), but self-contained. Re-verified end to end: build_CATS.jl exits 0, Python loader still validates all 35876 components.
Names, prime movers, plant grouping and hydro reservoirs derived from the EIA identifiers already carried in GIS/CATS_gens.csv, plus the CAISO reactive inventory replacing the MATPOWER placeholders. Every derived decision lands in a checked-in CSV under data/ with a markdown audit report, so the result is reviewable and diffable rather than buried in build code; .gitignore whitelists those CSVs specifically. Provenance in CHANGELOG.md. test/runtests.jl reads the exported case back with `from_file(System, ...)` and checks it against those CSVs rather than a snapshot, so regenerating the CSVs and rebuilding keeps the test valid while changing one without the other fails it. HydroPowerPlant is asserted at the promoted-plant count rather than every hydro plant: PSY rejects `add_supplemental_attribute!(sys, ::HydroDispatch, ::HydroPowerPlant, ...)`, so `attach_plant_groups!` attaches one only where the units were promoted to HydroTurbine/HydroPumpTurbine. 32 of the 182 hydro plant codes qualify. The equality keeps that boundary honest in both directions. Loading the case needs PowerSystems at 82c2c5961 or later, which adds the ImportExportCost OpenAPI reader the 26 import sources require.
The build is now a first-class directory rather than a subdirectory of the
Sienna code. `build/` holds everything that constructs the system —
build_CATS.jl and its includes, the hydro and reactive enrichment scripts — and
`Sienna/` holds only the model. The two carry separate environments on purpose:
the build needs PowerSystems and the parsers, the model needs
PowerOperationsModels and a solver, and neither imports the other.
`Sienna/cats_model.jl` replaces the psy5 PowerSimulations driver. It solves a
single DecisionModel instance rather than a multi-step Simulation, so the
rolling-horizon machinery is gone; `interval` survives only because
`auto_transform_time_series!` needs both it and `horizon` to convert the
SingleTimeSeries to a forecast. The security behaviour is unchanged: contingencies
are the highest-rated non-radial 500 kV lines, monitored lines the non-radial
230 kV ones, and duals are requested on the base-case and post-contingency
capacity limits. PowerOperationsModels derives the MODF from the prebuilt PTDF's
factorization core, so there is no separate MODF matrix and no second reduction.
The OpenAPI bundle is now the only serialization. The psy5-era CATS_Sienna.json,
its metadata and its 337 MB time-series store are deleted; they carry
Transformer2W and NATURAL_UNITS and cannot load into psy6 PowerSystems. Script/
goes with them — those drivers target PowerSimulations, which does not exist on
this line — along with GIS/counties, whose only consumer was the map script.
GIS/CATS_{buses,gens}.csv and CATS_lines.json stay: despite the directory name
they are build inputs.
Both environments pin their Sienna dependencies by git branch rather than by
local development path, so the tree resolves from a clean clone. The manifests
lock the exact commits this was validated against: PowerSystems psy6 a73f8b071,
InfrastructureSystems IS4 6dc80f32, PowerNetworkMatrices psy6 3a635ca,
PowerFlows psy6 cdf02bc, PowerOperationsModels main a5a5468,
InfrastructureOptimizationModels main 4fd56fb, PowerFlowFileParser psy6 e2987b2,
PowerOpenAPIModels main 0568d41.
README.md documents the full rebuild: download_data.sh for the two large
time-series files, instantiate, build, then test/runtests.jl. The 1.2 GB
Load_Agg_Post_Assignment .jld2 is a cache the build regenerates from the CSV, not
an input, so only ~580 MB needs downloading.
Verified end to end on the pinned versions: build exits clean, the round-trip
test passes 492 assertions, and the model solves with three binding base-case
duals and none post-contingency.
Runs the CATS N-1 SCUC end-to-end against PowerOperationsModels main (with the jd/hydro_fixes reservoir fixes validated locally): PTDF network model with slacks, run-of-river hydro (reservoir formulations disabled pending fuller inflow-series coverage), and updated hydro/reactive enrichment scripts feeding the build. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011FD965hAbJu5wYPCMZFTMH
Pins PowerOperationsModels to the jd/hydro_fixes branch (unreleased reservoir-hydro fixes) instead of main, so the model runs against the fix until it merges. Revert to main once merged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011FD965hAbJu5wYPCMZFTMH
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR enhances the CATS dataset to have better hydro and also uses the data sources attribute as an example