Update PyPSA dependency >= V1 - #762
Closed
apigott wants to merge 20 commits into
Closed
Conversation
apigott
marked this pull request as ready for review
June 19, 2026 03:50
Collaborator
Author
|
The biggest concern I have is around changing the unit commitment implementation. The default attr isn't available as part of pypsa v1 so I borrowed the implementation mostly from the pypsa-eur implementation. Open to discussion about the change of method and addition of |
for more information, see https://pre-commit.ci
ktehranchi
requested changes
Jun 25, 2026
ktehranchi
left a comment
ktehranchi
left a comment
Collaborator
There was a problem hiding this comment.
A few things:
In PyPSA-USA we always merge new developments to the develop branch first before moving to master. We make releases from develop -> master when there are large feature bumps. So i would change this to merge into develop.
RE: Unit-committment. I'm fine to move toward the data-source and method used in pypsa-eur for UC... we currently take it from the public version of the wecc ads, matched on the unit, then fill missing data with the averages of the WECC dataset... but would be fine to use a flat csv to fill any missing data... But i think this can be a different PR.
Collaborator
Author
|
Target branch has been changed. Can you clarify the preference for how to do the UC in a different PR? |
Collaborator
|
Thanks for the work on this, @apigott! Quick question, have you run the sector network with this, or just the electrical network? If you haven't actually run the sector network, I can run it over the next couple of days to make sure nothing broke with sector studies. |
Collaborator
Author
|
@trevorb1 I haven't had a chance to test the sectoral coupling |
Collaborator
|
@apigott Sorry, this PR slipped my mind! Il try to do a test of it this weekend! |
This was referenced Aug 23, 2026
Collaborator
|
closing since opened new PR in the v1 epic on this |
ktehranchi
added a commit
that referenced
this pull request
Aug 28, 2026
…edo of #762) (#778) * Pre-aggregate SCD tables in build_powerplants to fix memory blow-up The LEFT JOINs in load_eia_operable_data and load_heat_rates_data joined the multi-year EIA-860 SCD tables (yearly_generators, scd_plants, scd_generators_energy_storage) without any date constraint. Each generator carries ~24 years of annual snapshots, so the intermediate join produced ~7,000 rows per generator (~200M rows for load_heat_rates_data) before the final aggregation collapsed them. Pre-aggregating each SCD table to one row per plant/generator using the same array_agg(... ORDER BY report_date DESC) FILTER (WHERE ... IS NOT NULL)[1] "latest non-null" pattern already used elsewhere in the file keeps the projection identical while reducing the join cardinality from ~24x24 to 1x1. Measured against s3://pudl.catalyst.coop/v2025.5.0: - peak RSS: 17.3 GB -> 2.2 GB - runtime: 503 s -> 24 s Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add scatter plot assets for PR description (drop before merge) These PNGs are referenced in the PR body to visualize the original-vs- refactored marginal_cost and heat_rate distributions. Safe to remove once the PR is reviewed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Remove PR-assets PNGs (not opening that PR yet) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Split simplify_network into aggregate_to_substations + cluster_simpl Phase 1 of the simplify-early refactor. Splits the existing simplify_network rule into two rules with no behavior change for the default kmeans algorithm: - aggregate_to_substations: pure-topology aggregation to substations (convert_to_voltage_level, remove_transformers, busmap-by-sub_id). Writes resources/{interconnect}/elec_b.nc + busmap_b.csv. - cluster_simpl: optional k-means/modularity reduction to {simpl} clusters. Reads elec_b.nc, writes elec_s{simpl}.nc + simpl regions, matching the former simplify_network output interface so downstream rules (cluster_network etc.) are untouched. HAC clustering is dropped from both rules (cluster_network now raises if algorithm='hac') along with the unused 'to_substations' and 'feature' config knobs. simplify_network.py is removed; its functions move into aggregate_to_substations.py. Snakemake dry-run on the default Western config produces the expected chain: aggregate_to_substations -> cluster_simpl -> cluster_network. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Remove unused config keys and unused snakemake rule params Dead-code cleanup audit against v1-epic. No behavior change. YAML config keys removed (no code reference): - config.plotting.yaml: costs_max, costs_threshold, energy_max, energy_min, energy_threshold (with matching docs/source/configtables/plotting.csv rows) - config.tutorial.yaml: sector.natural_gas.allow_imports_exports - config.common.yaml: renewable.hydro.{PHS_max_hours, resource.hydrobasins, resource.flowspeed, hydro_max_hours, clip_min_inflow, normalization, multiplier}; atlite.default_cutout; electricity.prm regional block - config.default.yaml: model_topology.interface_transmission_limits; solving.mem Snakemake params: declarations removed (declared but never read by the target script): - build_bus_regions: focus_weights - add_extra_components: ucap (script reads via snakemake.config instead) - prepare_network: adjustments, co2base - add_sectors: electricity, costs, plotting - plot_network_maps: plotting, retirement - plot_statistics: plotting - plot_natural_gas: plotting - solve_network: planning_horizons, transmission_network, sector_config - solve_network_validation: planning_horizons, co2_sequestration_potential Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Reorganize resources/ into category-first subfolders Outputs under resources/ previously sat at a single interconnect-rooted level (e.g. resources/texas/elec_b.nc, resources/texas/busmap_s50.csv, resources/texas/profile_solar_s50.nc). Files of unrelated types were intermingled, making the tree hard to navigate. This change introduces a category-first layout: the top level under resources/ is what the file IS (networks/, busmaps/, profiles/, ...) and {interconnect} becomes the next level down. resources/ networks/{interconnect}/elec_*.nc busmaps/{interconnect}/busmap_*.csv, bus2sub.csv, sub.csv profiles/{interconnect}/profile_*.nc, nrel_mapping_cache geospatial/{interconnect}/*.geojson, bus_gis.csv, lines_gis.csv costs/costs_{year}.csv, sector_costs_{year}.csv prices/{interconnect}/{state,ba}_*_prices.csv, pudl_fuel_costs.csv demand/{interconnect}/{end_use}_*.csv|pkl population/{interconnect}/pop_layout_*.nc|csv temperature/{interconnect}/temp_{soil,air}_*.nc heating_cop/{interconnect}/cop_{soil,air}_*.nc co2/{interconnect}/co2_storage_*.csv powerplants/powerplants.csv Implementation: - workflow/Snakefile defines twelve category constants (NETWORKS, BUSMAPS, PROFILES, GEOSPATIAL, COSTS, PRICES, POWERPLANTS, DEMAND, HEATING_COP, TEMPERATURE, POPULATION, CO2) composed from RESOURCES, so the RDIR (run name) and shared_resources prefixes still work. - Every RESOURCES + "{interconnect}/<file>" reference in the .smk files swapped to the matching category constant. No filenames change. - "Geospatial/" (capital G) renamed to "geospatial/" (lowercase) for consistency with the new top-level subfolders. - "resources/powerplants.csv" hardcoded literal moved to "resources/powerplants/powerplants.csv" (still shared across runs). - Two off-rule literal paths updated: a __main__ test path in build_natural_gas.py and argparse defaults in plot_caps_summary.py. - Doc reference in about-usage.md sharpened to resources/networks/. Verified by snakemake -n: full DAG resolves end-to-end for resources/Default/networks/texas/elec_base_network_l_pp.pkl, with each intermediate input/output routed to the new category folder. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Merge stacked PRs #9 and #11 into v1-epic Brings in Phase 2 (move simplify/cluster_simpl ahead of demand+RE) and Phase 3 (aggregate EGS supply curves through cluster_simpl busmap). Both PRs were originally merged into their stacked-base branches rather than v1-epic; this merge propagates their changes up. Resolved conflicts in build_electricity.smk by taking the post-#12 resources/ layout (category-first constants: NETWORKS, GEOSPATIAL, PROFILES, DEMAND, BUSMAPS) and applying Phase 2's path repointing (network refs to elec_s{simpl}.nc, _s{simpl} suffix on demand/profile outputs). Repointed aggregate_egs outputs and add_electricity EGS inputs to PROFILES/BUSMAPS per the new layout. * Add design spec for PyPSA network schema tracking Two-part approach: a curated docs/network-schema.md catalog of custom columns plus a _helpers.log_network_schema helper that logs per-script column entry/exit diffs. Motivated by recurring consense aggregation crashes (most recently LAF_state in aggregate_to_substations). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add implementation plan for network schema tracking Six tasks: add helper + tests, wire into topology / add-* / sectors chains, smoke test on test_small western, seed catalog from logs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add log_network_schema helper for per-script column tracking Logs component row count and column set on entry; on exit emits row and column diffs vs. the entry snapshot. Logging only — no asserts. Wires into scripts in follow-up tasks; tested in isolation here. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Wire log_network_schema into topology chain scripts Adds entry/exit schema logging to build_base_network, aggregate_to_substations, cluster_simpl, cluster_network. Logging only; no behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Wire log_network_schema into add_electricity / add_extra_components / add_demand Logging only; no behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Wire log_network_schema into prepare_network / add_sectors / solve_network Logging only; no behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add testing strategy design spec Tiered test pyramid (static <30s + integration <5min) targeting the classes of breakage that the simplify-early refactor (PRs #7-#12) made visible: path/wiring drift, dead config keys, silent artifact-shape regressions. Schema-catalog assertions deferred to a follow-up PR after the schema-tracking spec lands. Brainstorming output. Five-PR migration plan included. * Seed docs/network-schema.md from observed pipeline logs Initial catalog of custom columns on PyPSA components, populated from [schema ...] log output of build_base_network and aggregate_to_substations entry on the small_mh western config. Sector- and electricity-stage columns will be added as those rules get exercised under the schema logger. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Tighten LAF_state catalog row tone Drop editorial framing ("the bug class this catalog was created to address"). Keep the factual note that LAF_state is missing from bus_strategies and the fix is tracked separately. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add implementation plan for testing strategy Five-PR migration: scaffolding → Tier A static checks → Tier B fixture → Tier B full assertions → CI wiring. Plan includes exact code, file paths, verification commands per step. PR 6 (schema assertions) deferred pending the schema-tracking initiative. * Add pytest config and test extras * Add tests/ directory skeleton * Wire existing unit tests into the 'fast' marker via repo-root conftest Adds a repo-root conftest.py that auto-marks every test collected from workflow/scripts/test/ with the 'fast' marker, so pytest -m fast picks them up without per-file changes. Also skips 9 pre-existing test failures unrelated to this PR (RPS constraint helper signature drift and ERM API drift) so pytest -m fast exits 0. Each skip carries a reason pointing back to v1-epic. * progress in refactor * Add cluster_simpl county fast-path design spec Approved-design doc for a new simpl="county" wildcard value that bypasses k-means and uses the substation network's county FIPS (prefixed with reeds_zone) as a direct busmap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Draft config-cleanup formatting spec Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add cluster_simpl county fast-path implementation plan Four-task TDD plan: extract resolve_simpl_mode and build_county_busmap helpers, wire them into __main__ dispatch, and document the new simpl="county" wildcard value. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add resolve_simpl_mode dispatch helper to cluster_simpl Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add build_county_busmap helper with missing-county guard Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Document and standardize repo_data config files Rewrite the three docs-source configs (default, tutorial, common) so a new user can read top-to-bottom and understand every option without cross-referencing scripts. Pure formatting + commenting; no key renames or hierarchy changes, no behavior changes. - Per-section banners above each Sphinx `# docs :` sentinel - Inline comment on every option (allowed values, units, acronym glosses) - File-header block per config (purpose, audience, merge order) - Removed in-section comment lines that bled into adjacent Sphinx slices - Normalized False→false - Verbose missing-key surface in default.yaml (commented-out examples) - Added minimal disabled co2/dac blocks to tutorial.yaml so it loads (build_electricity.smk unconditionally reads config["co2"]["storage"]) Verified: all three YAMLs parse; Sphinx slicer ranges still resolve; `snakemake -n data_model` builds DAG cleanly with both default and tutorial configs. Spec: docs/superpowers/specs/2026-05-21-config-cleanup-design.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Revert "Test scaffolding: pytest config + fast-marker for existing unit tests" * Revert "Merge pull request #18 from ktehranchi/revert-13-tests/pr1-scaffolding" This reverts commit 21c3e95294898ba60ed606f1e4f3d102b6698b22, reversing changes made to 5e351fb15b6d0593cd1a9864018f62eb9380b6b7. * Add pipeline-equivalence master spec, glossary, and v1-epic change-log Approved via grilling session: anchor = upstream/develop e7f8bd70, two-prong pinned-busmap equivalence protocol, CA harness (USA deferred), deltas ledger + waivers, profile-first performance phase. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Land Tier A + Tier B fixture into v1-epic (#14 + #15) (#20) * Add Tier A: path validator + fix surfaced RESOURCES violations Adds tests/static/test_paths.py which scans every .smk file (and the root Snakefile) for ``RESOURCES + "{interconnect}/..."`` literals, the forbidden form after PR #12 reorganized resources/ into category-first subfolders (NETWORKS, HEATING_COP, POPULATION, CO2, ...). Routes every rule input/output through the right category constant so the new test passes: - workflow/Snakefile data_model rule: RESOURCES -> NETWORKS (clustered network output). - workflow/rules/build_sector.smk sector_input_files() + add_sectors: RESOURCES -> NETWORKS (elec_*_ec_l*_opts*.nc inputs and the add_sectors output), HEATING_COP (cop_soil/cop_air *.nc inputs), POPULATION (pop_layout_elec_*.csv input), CO2 (co2_storage_*.csv input). - workflow/rules/solve_electricity.smk solve_network rule: RESOURCES -> NETWORKS (network input). - workflow/rules/validate.smk solve_network_validation rule: RESOURCES -> NETWORKS (network input). These changes align with the producer rules, which already write to the correct category-prefixed paths; the bug was on the consumer side referencing stale RESOURCES + {interconnect}/ layouts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add Tier A: snakemake -n dry-run resolves on tutorial + default configs Adds tests/static/test_dag_dryrun.py which runs ``snakemake -n`` against config.tutorial.yaml (cluster_network + solve_network) and config.default.yaml (cluster_network) to catch typo'd rule names, wildcard mismatches, syntactically broken .smk files, and unresolved config keys before they reach a real build. A session-scoped autouse fixture mirrors init_pypsa_usa.sh, copying any missing config templates (and policy_constraints/) from workflow/repo_data/config/ into workflow/config/ so the test runs cleanly in CI without depending on the init script. Surfaced two real KeyErrors in the merged config that blocked the dry-run; fixed by extending workflow/config/config.common.yaml (and the canonical template at workflow/repo_data/config/config.common.yaml) with sensible defaults so every loaded config exposes the keys the rules read unconditionally: - ``co2.storage`` / ``co2.network.*`` — read in build_electricity.smk, build_sector.smk, add_sectors.py, add_extra_components.py without an ``enable`` guard. Defaults to ``false`` so behavior is unchanged. - ``renewable.EGS.drilling_cost`` — read in add_electricity.py to switch between the ``capex_usd_kw`` and ``advanced_capex_usd_kw`` columns of the EGS supply curve. Defaults to ``base`` (selects the non-advanced column). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add Tier A: config-key existence validator Adds tests/static/test_config_keys.py which AST-walks every workflow/scripts/*.py module, extracts subscript chains rooted at ``snakemake.config[...][...]``, and asserts each chained key path exists in the merged default config. Catches the class of bug fixed in PR #10 — scripts dereferencing config keys that no live YAML defines (typos, dropped flags, dead branches). The merged config is built in the same order the Snakefile loads configfiles (cluster -> common -> plotting -> api -> sector -> default), with each file preferred from workflow/config/ if present and falling back to the canonical workflow/repo_data/config/ template otherwise. The tutorial config is excluded — it's a minimal opt-in subset, not a key-vocabulary source of truth. All 57 parametrized scripts pass after the ``co2`` / ``renewable.EGS. drilling_cost`` defaults added in the preceding commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add minimal CA-only test config for tests/integration Adds workflow/repo_data/config/config.test.yaml — a minimal CA-only Western slice (simpl=20, clusters=4m, 1-day snapshots, 2030 horizon) sized for <5 min builds. Mirrors the seeding pattern used for other config templates in repo_data/config (the autouse fixture in tests/static/test_dag_dryrun.py copies missing files from there into workflow/config/ at test time). Includes explicit electricity, conventional, lines, links, costs, clustering, custom_files, and solving blocks needed for the DAG to resolve to cluster_network without pulling config.default.yaml inheritance. * Add Tier B fixture + smoke test (cluster_simpl bus count) - `tests/integration/conftest.py` provides a session-scoped `built` fixture that runs `snakemake --until cluster_network` once per session using config.test.yaml. Skips gracefully when `workflow/data`, `workflow/cutouts`, or `workflow/repo_data` are missing, letting developers run `pytest -m fast` without setting up the data deps. Uses `--scheduler greedy` to avoid snakemake's ILP-based scheduler (which can fail when CBC isn't available). Includes its own autouse `_seed_runtime_configs` fixture mirroring Tier A so it works regardless of collection order. - `tests/integration/test_artifacts.py` has one placeholder assertion (`test_post_cluster_simpl_bus_count`) that loads elec_s20.nc and asserts bus count == 20. PR 4 will flesh out the full per-stage assertion suite. * PR3 quality fixes: TimeoutExpired handling, xdist guard, solving cleanup - Catch subprocess.TimeoutExpired and surface stderr tail in the fail message (previously the hang case dropped helpful output). - Add pytest_collection_modifyitems hook that exits cleanly if pytest-xdist is active for integration tests (avoids snakemake lock races). - Strip the dead solving block from config.test.yaml — fixture stops at cluster_network, solver never invoked. - Inline comment for --scheduler greedy rationale. - BuiltArtifacts docstring cross-referencing config.test.yaml constants. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Tier B: per-stage artifact assertions (#16) * Tier B: per-stage shape assertions for the five build artifacts Replaces the single smoke test from PR 3 with a class-per-stage layout covering aggregate_to_substations, cluster_simpl, add_demand, add_electricity, and cluster_network. Asserts file existence, bus counts matching wildcards, no-NaN in load-bearing columns, netCDF roundtrip, busmap coverage, and filename wildcard propagation. * Replace broken ./test.sh with fast-tests + e2e-tests CI jobs (#17) Two parallel jobs gate every push/PR to master and v1-epic: - fast-tests: no data deps, ~1min wall-clock, runs pytest -m fast - e2e-tests: cached data+cutouts, ~10min wall-clock, runs pytest -m integration Plus a weekly Tuesday cron (upstream-regression) that pulls latest PyPSA/atlite/linopy from master and runs the full suite. The previous `./test.sh` step was a silent no-op (file didn't exist). Manual post-merge step: mark fast-tests and e2e-tests as required status checks in branch protection for master and v1-epic. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Revise equivalence protocol to config-only determinism (no pinned busmap) cluster_network kmeans is seeded (random_state=0) and unchanged on both branches, so identical inputs yield identical busmaps from basic config alone; custom_busmap repair drops off the critical path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix four post-refactor breakages found during validation runs - add_electricity: map buses to states via reeds_state->full name for capital-cost multipliers; guard match_plant_to_bus when plants lack a 'country' column; group renewable capacities by bus identity (bus == substation after aggregate_to_substations) - build_demand: state-zone lookup via reeds_state->full name - build_powerplants: fill missing EIA-860 summer/winter derates with 1.0 (multi-unit CC sub-units otherwise propagate NaN into p_max_pu) - cluster_network: preserve 'country' column through cluster_regions dissolve (consumed by match_plant_to_bus) Equivalence verdict pending (Phase 3 of the 2026-08-07 master spec): the derate fill plausibly changes results vs anchor and needs a ledger entry or a fix-to-parity. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix upstream-merge fallout caught by Tier A checks - postprocess.smk: repoint upstream's new rule block from the old {interconnect}/Geospatial/ layout to the GEOSPATIAL category constant - common.smk: materialize constants.py into the snakemake source cache so _helpers.py's module-level import resolves under a fresh cache (broken since upstream #764 on any machine without a warm cache) - change-log: record the upstream sync merge and its resolutions Tier A: 101 passed, 0 failed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add CA equivalence harness implementation plan (Phase 2-4) Grounded in a 5-reader research workflow over both branches: artifact path pairing table, simpl='' viability, shared-config key requirements, anchor worktree provisioning steps, loader/report tooling inventory. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * T1: add shared equivalence-harness config (both branches) CA-only Western slice, Jan 6-19 2019, clusters=10, seeded gurobi barrier. Carries the anchor-required keys (co2.storage, clustering feature keys) and non-null renewable_land_access for godeeep. Both prong targets dry-run-resolve on v1-epic. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * T2-T4: equivalence harness package + AEO scenario config fix Harness: paths.py (verified pairing map), build.py (dual-side driver, anchor worktree provisioning, documented infra patch), compare.py (D2/D7 tolerances, waivers), report.py (self-contained HTML), run.py CLI, tier-C pytest marker. Config fix (debug agent): build_cost_data crashed on both branches because the script's 'Reference' AEO fallback matches zero PUDL rows; costs.aeo.scenario: reference added to the shared harness config. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Change-log: PR #21 bugfixes merged during harness bringup Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Equivalence config: clusters 10 -> 4m (reeds transport-model constraint) cluster_network's reeds transport-model path requires clusters == the footprint's ReEDS zone count (4 for CA); discovered by the anchor build assert. Even better for prong 1: the zonal busmap is pure attribute membership - no kmeans anywhere in the chain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Equivalence config: add hard-required costs sub-keys (itc/ptc/max_growth/emission_prices/social_discount_rate/min_year) Direct-subscript reads in add_extra_components and prepare_network on both branches; values mirror config.default.yaml. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix simpl='' identity branch: normalize region bus names (harness catch) cluster_simpl's pass-through branch copied regions geojson verbatim, keeping float-formatted names ('35827.0') while the network/busmap use bare IDs ('35827'). The godeeep profile path's three-way bus intersection came up empty, silently writing 0-bus profiles that crashed add_electricity two rules later. Fix normalizes names in the identity branch (same as cluster_regions does for kmeans), adds a fail-loud disjoint-bus-spaces guard in build_renewable_profiles, and skips carriers with no profile buses in attach_wind_and_solar. Results effect: simpl='' runs regain silently-dropped RE generators; simpl=N unchanged. Found by the first candidate equivalence build. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Equivalence config: solve-stage policy-constraint keys (both branches) regional_Co2_limits (read by add_regional_co2limit, opts/policy.py:548, fired by the REM opt), plus sibling TCT/RPS keys. Dead keys (transmission_interface_limits, SAFE_*, agg_p_nom_limits) deliberately omitted - zero readers on either branch. Both prong-1 solves now complete optimally: candidate obj -2.56292e9, anchor -2.55060e9 (0.48% delta - goes to the harness comparison for adjudication). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix double-applied length_factor in transmission capital costs aggregate_to_substations folds lines.length_factor into length; the post-reorder add_electricity call multiplied it again (hav x 1.5625 instead of x 1.25 -> 25% line CAPEX inflation in line-preserving runs). Harness catch: assembled-stage comparison, ratio 1.2486 on 2810/2811 lines vs anchor. Clustered/solved outputs of reeds transport runs were unaffected (ITL costs recomputed identically on both branches). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Comparator: semantic normalization + fan-out suppression Float-label normalization, Load re-keyed on bus attr, demand pair corrected to system-total check (anchor CSV is nodal), per-frame finding cap with suppression marker, Generator[bus,carrier] p_nom aggregate view. Prong-1 findings: 16882 -> 379; profiles fully clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Deltas ledger + first waivers (DL-1..DL-4, provisional pending user sign-off) Stage-ordering capital-cost residues (clustered stage verified identical), plotting palette, slack-bus bookkeeping. Open findings (demand -6.3%, hydro attachment, storage sets, naming, objective 0.48%) remain unwaived pending investigation verdicts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Ledger: DL-5..DL-7 (Catalina placement, naming stage-ordering, aggregation composition) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix hydro attachment and demand conservation (harness catches #3 and #4) Hydro: attach_breakthrough_renewable_plants filtered raw plant.csv bus_ids against the substation-level network index - 12,848 MW of CA hydro silently dropped, 128.6 MW of Eastern hydro attached at colliding ids. Fix remaps through bus2sub + busmap_s{simpl} (patterned after aggregate_egs; busmap_s added to rule inputs). Verified: hydro p_nom 12,976.8 MW == anchor exactly; zero Eastern collisions. Demand: remove_transformers dropped 420 trafo-secondary buses with their Pd/LAF_state before demand existed; build_demand allocates by LAF_state without renormalizing -> -6.281% total demand and 95 zero- load substations. Fix transfers Pd/LAF_state through the trafo map and composes it into busmap_b.csv (full 4,248-bus coverage). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Prong 1 GREEN: finalize waivers DL-5..DL-8; ledger resolution note All fixes landed; solved objective rel diff 6.6e-05 (gate 1e-3); comparison: 0 live / 305 total findings. Every waived class carries a ledger row with quantified evidence; content-bearing aggregates (Generator[bus,carrier], Load_t.p_set, StorageUnit totals) remain unwaived as guards. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Spec: implementation-outcomes addendum (prong 1 green, benchmarks) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Prong 2 GREEN: DL-9 (existing-RE discretization) + prong-scoped waivers Investigation closed the MW accounting exactly: the +15%/+8% existing onwind/solar delta is the shared silent-drop of plants in profile-less groups (upstream issue #16) interacting with by-design different cluster geometries; candidate strictly closer to ground truth. Waivers now match on prong so prong-1's objective gate stays live. Both prongs PASS; pytest -m equivalence: 2 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Report: explicit white background / dark text (dark-mode viewers rendered dark-on-dark) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Harness: clusters 4m -> 4 (user choice: aggregate all carriers to zones) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Report: p_nom-by-attribute plots + harness-only benchmark summary table New section compares generator p_nom grouped by carrier / reeds_zone / build-year decade for two pairings: each side's add_electricity output and the simplify-stage networks. Benchmark table now filters to this harness's rules only (historic runs polluted the shared cluster_network benchmark path), shows wall time + max RSS per rule for both sides, and ends with a TOTAL row (wall sum, RSS peak) with cross-side caveats. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Revamp equivalence report into one explanatory artifact Grilling-session design: executive summary (verdict banner, provenance fingerprint, delta index from the ledger, stage timeline strip), unified diff-DAG built from both sides' rulegraphs with per-rule walltime annotations, stage-ordered narrative spine with born/inherited/masked delta-class tracking, 3-panel maps (V1-epic | anchor | diff) reusing plot_network_maps choropleths at the assembled + solved stages (identical quantities collapse to a sentence; the only capacity map rendered is oil - the 11.3 MW DL-5 Catalina placement), and a rule-grouped benchmark table with instance subrows and cross-side caveats. candidate -> V1-epic in all user-facing text. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Percentage differences everywhere + rename rule cluster_simpl -> cluster_resources Comparator findings carry max_rel_pct / per-example rel_pct; report narratives, map captions, and delta quotes state % alongside absolutes (DC links 17.86%, zonal Pd 7.02%, prong-2 onwind 15.15%, objective 0.2228%). Rule rename covers rule name, log path, walltime key, living docs; script file unchanged. Tier A green (102 passed) - DAG dry-runs validate the rename. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Ledger amendments DL-2/DL-3/DL-9 + CF-coverage trace; CF map captions DL-2: anchor-side link-aggregation bug (inverter-pair term deflated 1/1.25), not base-stage pricing; V1-epic DC-link km-term follow-up flagged. DL-3: palette diff was an uncommitted local plotting config (co2_emissions 0.000% different) - resynced from template. DL-9: MW-exact confirmation; candidate drops are out-of-footprint snap artifacts; s385 production exposure 1.54%/0.12%; prototype fix recovers 100.0%. CF maps: 72.4% onwind-less regions are NREL land-access exclusions, identical on both sides - captions updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Record user decision: keep length_factor=1.0; note TAMU-only cost effect The factor only modifies transmission costs on the TAMU line-preserving network; reeds transport runs drop lines/DC links at clustering and rebuild ITL costs from ReEDS distance-cost tables. DC-link km-term difference (~6% of link cost, 2 links) accepted under this decision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * NREL source-data verification: split verdict on missing CF regions In-state CA exclusions are genuine (NREL raster: median 0.0%% developable in missing regions vs 8.1%% kept; sjoin verified clean, 0.12%% sites dropped nationally; only 450 MW / 0.17%% in missing CA regions). REAL GAP found out of state: caps rolled up on the national substation tessellation are silently dropna'd by the CA-focus busmap (build_renewable_profiles.py:50) - two border regions holding 13.4%% of the West's developable wind get 0 MW. Shared identically by both sides (no equivalence delta); follow-up chip filed. CF map caption updated with the nuanced explanation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Harness: interconnect parameterization (EQ_INTERCONNECT) + whole-US data config EQ_INTERCONNECT env switches paths/config/report filenames (western default, fully backward-compatible); EQ_UNTIL=assembled truncates the comparison at the data stages. config.equivalence-usa.yaml = shared config with interconnect usa and no reeds_state filter, for the national-scale data-equivalence run (stop before solve). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * USA data-stage equivalence: DL-10 state-assignment split, usa-scoped waivers National run (both sides to assembled stage, 16m10s candidate / 17m13s anchor concurrent on one Mac): system demand conserves to 0.0485%, state totals shift <=1.6% (KS) because candidate allocates by reeds_state membership vs anchor's raw breakthrough state column - a scientific decision flagged for user sign-off. Same family: 93 capital_cost diffs (max 8.4%), one 296 MW CCGT placement flip. Waivers now match on interconnect so usa waivers cannot blind the western gates. USA prong 1 (data stages): PASS 0 live / 113 total. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix benchmark rule mapping: expand {IC} in patterns at match time The interconnect parameterization left literal '{IC}' in the regex pattern lists, so no usa benchmark path matched and only stale western solve_network rows (whose pattern lacked the interconnect) leaked through. Patterns now .format(IC=...) at match time and the solve pattern carries the interconnect. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Maps: thorough national-scale comparison suite Per user request for the 41k-region national run: 3-panel maps (V1-epic | anchor | difference) for EVERY carrier's existing capacity (identical-vs-differing summary line retained, no more collapse to text), maximum installable capacity (profile p_nom_max + finite extendable generator p_nom_max), capacity-weighted mean p_max_pu per carrier, and demand; profile-file CF maps retained. Zero-linewidth regions and capped dpi keep the 41k-polygon renders legible and the HTML at 3.6MB (29 map figures, 0 failures). Western report regression verified. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Ledger: DL-1..DL-10 countersigned by ktehranchi (2026-08-18) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Out-of-footprint NREL caps: loud accounting + opt-in nearest-bus reassignment remap_caps_to_cluster now WARNS unconditionally with dropped entry count / MW / % of national total (CA prong-1: 17,340/17,890 entries, 9.43 of 9.70 TW onwind p_nom_max, 97.3%). New default-off config nrel_caps_reassign {enable, max_km:100} reassigns unmapped entries to the nearest in-footprint bus within max_km (haversine, chunked); published caps carry no x/y so enabling raises a clear config error until the HPC rollup regenerates - build_nrel_bus_capacities.py now writes capacity-weighted site-centroid x/y for that regeneration. Flag-off output verified byte-identical; 7 new unit tests; fast gate 108 passed. Ledger: no new delta while default-off. Long-term option (a) - re-roll caps per run geometry - stays HPC-side. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add HPC handoff plan: regenerate NREL caps with per-bus coordinates Self-contained brief for an HPC-side agent: verify generation geometry against the published 17,890-bus tessellation before building, back up and force-regenerate caps only (avail unchanged), per-file identity verification (x/y additions only), CA recovery quantification with max_km sensitivity, and an explicitly human-gated Zenodo publish step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Ledger: record footprint-scoped bus-regions prototype findings Documents the interconnect-wide empty-county sweep root cause (regions 7x CA, 215.5 GW WECC fleet attached to a CA-demand-only model) and the quantified before/after from branch proto/footprint-scoped-regions (ccfe4b77). Docs only; no pipeline behavior change on v1-epic. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Migrate to pypsa 1.2.4 / linopy 0.9.1 (redo of upstream #762 on v1-epic) Every removed pre-1.0 API is migrated: madd/mremove -> add/remove (names= -> name=; v1 add returns None), clustering.network -> clustering.n, n.df/n.pnl/iterate_components -> n.components[c].static/.dynamic, pypsa.descriptors imports -> Network/component methods and pypsa.common.expand_series, component_attrs -> n.components[c].defaults, copy(with_time=False) -> copy(snapshots=[]), statistics groupers -> string lists (comps/aggregate_time -> components/groupby_time), pypsa.pf logger -> pypsa.network.power_flow. Semantics pinned, not just renamed: - linopy vars live on dim "name": fixed selectors/groupers in opts/ (policy, reserves, land, sector, interchange); RESERVES operational constraints rewritten to mirror pypsa v1 internals (xarray get_bounds_pu). - MultiIndex coefficient frames wrapped in DataArray() so linopy keeps the flat snapshot dim instead of unstacking to period x timestep. - e_cyclic_per_period / cyclic_state_of_charge_per_period defaults flipped True->False in v1: pinned True at all 15 cyclic adds (results equivalence). - Bus index renamed to "name": bus2sub.csv keeps legacy "Bus" header via index_label; nearest-bus matching made index-positional (fixes two latent .Bus-attribute bugs in match_missing_buses on n.buses-derived frames). - Empty component frames carry int64 index in v1: guarded .str use in freeze_prior_periods. - Network.copy() drops the hidden snapshot index name (pypsa bug through 1.3.0, breaks c.da on multi-period copies): healed in test conftest; netCDF round-trips unaffected. Tests: unit tier 45 passed / 1 skipped; 9 stale "pre-existing failure on v1-epic" skips removed (now passing under the new stack); the remaining skip (e2e myopic) bisected as pre-existing on the old stack too. Static tier 72 passed. tables==3.10.2 pinned explicitly (re-lock dropped the implicit transitive). Dead load_network helper deleted. Migration map and pypsa-v1 data-storage adoption candidates in docs/pypsa-v1-migration.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Scope empty-county sweep to model_topology.include footprint When a run is scoped with model_topology.include (e.g. reeds_state: [CA]), build_bus_regions previously swept every busless county in the FULL interconnect and glued its geometry onto the nearest retained bus: a CA-only run's onshore regions covered 2.93M km2 (86% outside CA, ~7x the state). Those inflated polygons then passed the whole WECC fleet through filter_plants_by_region's sjoin (215.5 GW existing capacity attached to a CA-demand-only model, incl. 22.6 GW coal and 5.4 GW out-of-state nuclear) and skewed border-bus godeeep CF aggregation. Now the sweep is restricted to the ReEDS zones present in the (already include-filtered) base network. Gated on include being set: unfiltered interconnect runs are byte-identical. After the fix a CA run tiles 409.8k km2 (0.2% out-of-state slivers) and attaches an 84.5 GW fleet that matches California's actual one carrier-by-carrier. Results-changing for scoped runs; pending ledger countersignature. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * CI: trigger tiered workflow on develop pushes/PRs All PRs now target develop (never master), so fast-tests/e2e-tests must run there once v1-epic merges. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DL-11: adopt footprint-scoped regions on both harness sides Adds the ADOPTED-FIX anchor-patch category to the equivalence harness (apply_adopted_fix_patches in tests/equivalence/build.py): mirrors the DL-11 empty-county-sweep fix onto the anchor worktree, marker-idempotent, plus a one-shot .eq-force-rerun marker that build_side turns into -R build_bus_regions (--rerun-triggers mtime neither reruns on code changes nor revisits missing intermediates when the final target looks current). Ledger gains the countersigned DL-11 row (with the known plants_must_add seam residual and forced-rerun mechanics); CHANGELOG and spec D10 document the two anchor-patch categories. Post-adoption CA prong-1 rerun: data stages clean (80/82 findings waived, matching baseline); the 2 live solved-stage findings trace to a pre-existing build_powerplants.py source-data divergence (diagnosed 2026-08-23, previously absorbed by DL-7's waiver), addressed separately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Bump to pypsa 1.3.0 on a pandas 3 stack (pandas 3.0.5, xarray 2026.7.0) Step two of the v1 landing: the API port was proven on 1.2.4/pandas 2.2.2 (previous commit), now the pins move to pypsa==1.3.0 which requires pandas>=3.0. Pin moves: pandas 2.2.2 -> 3.0.5; xarray 2024.9.0 -> 2026.7.0 (floor forced by pandas 3); geopandas 1.0.1 -> 1.1.4 (pandas-3 line). Deliberately held: numpy 1.26.0 (rasterio/atlite numpy-1.x ABI), dask/distributed 2024.12.0 (import-verified under pandas 3), linopy 0.9.1. xarray 2026 fallout: - StorageUnit RESERVES energy balance rewritten fully in model space, mirroring pypsa 1.3's internal define_storage_unit_constraints (n.optimize._window + c.da accessors). The old pypsa-0.30-era copy mixed pandas-converted DataArrays with model coords and raised AlignmentError under xarray 2026's strict alignment. - ERM nodal-balance RHS columns axis pinned to "Bus": region_buses.index is named "name" under v1, which broadcast the constraint over a spurious dim and broke dual extraction in store_ERM_duals. pandas 3 sweep (audited class-by-class; both suites green): - groupby(axis=1) removed -> transpose-group-transpose (_helpers aggregate_p_nom, plot_network_maps emissions map). - Offset aliases: 'H' -> 'h' (add_sectors), deprecated 'd' -> 'D' (plot_statistics, plot_statistics_sector, summary), and the config time_resolution knob (documented as "int H") lowercased before resampling in prepare_network ('4H' is a hard error in pandas 3). - read_csv(squeeze=) removed -> .squeeze("columns") (cluster_network). - geopandas: .unary_union -> .union_all() (build_renewable_profiles). - pypsa.options.api.legacy_string_dtype = True pinned wherever networks are loaded (_helpers, unit-test conftest, tests/integration conftest, equivalence harness load_network) so component frames stay on object dtype until the deliberate pypsa-2.0-era flip. - Audited clean: chained-assignment/CoW hazards (63 candidate sites), str-dtype checks (0 sites), 20+ removed pandas API families, all freq literals + dynamic freq paths + 14 YAML configs. Tests unchanged on the final stack: unit 45 passed / 1 skipped, static 72 passed. Tier-C equivalence not rerun (env moved twice; re-baseline needed). CLAUDE.md env section, docs/pypsa-v1-migration.md, and the changelog entry updated for the 1.3.0 landing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DL-12: adopt EIA-860 pre-aggregation on both harness sides Second ADOPTED-FIX anchor patch (apply_powerplants_adoption): dynamic whole-file adoption of the candidate's build_powerplants.py onto the anchor, sentinel-gated (ges_latest CTE), interface-checked against the pristine e7f8bd70 file, idempotent by content comparison; .eq-force-rerun writing is now merge-safe (mark_force_rerun). Root cause of the 2.34% prong-1 solve divergence: upstream joins EIA-860 tables raw so ~24 years of report_date vintages reweight its means (Watson Cogen at an impossible 62% efficiency); v1-epic pre-aggregates. PUDL release and all tracked inputs identical — the query was the sole divergence. Harness after adoption (both sides rebuilt, candidate's stale powerplants.csv regenerated too): prong 1 PASS 0 live/72, objective rel 2.34e-2 -> 2.1e-6, per-carrier p_nom_opt equal to 0.01 MW; prong 2 PASS 0 live/3 (all DL-9-class, recalibrated: solar gap unchanged 3,586.6 MW, onwind gap 3,680.1 MW after DL-11 decontamination). DL-7 re-scoped: efficiency + marginal_cost members were DL-12 source data (waivers deleted as dead); only the non-composable unweighted fuel_cost mean survives, metadata-only. Countersigned (ktehranchi, 2026-08-23). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Bound the seam-plant fallback to the model footprint in scoped runs filter_plants_by_region filters plants by sjoin against regions_onshore / regions_offshore. Since DL-11 (88bede47) those layers tile only the model footprint in scoped runs — a model_topology.include: {reeds_state: [CA]} run tiles 409.8k km2 instead of 2.93M km2. The plants_must_add fallback, however, stayed unconditional: it selects plants lying outside every ReEDS shape of the run's interconnect whose ReEDS-membership interconnect disagrees with their EIA `interconnection`, and concatenates them into plants_filt with no test against the (now footprint-sized) regions. match_plant_to_bus's second pass then assigns any plant without a zone match to the nearest network bus with no distance bound. Measured on the CA equivalence run (western, 2030, simpl=''), that fallback leaks 23 plants / 1,887.4 MW onto California buses — NM 1,112.1 MW of wind and solar, MT 370.3 MW (incl. Fort Peck hydro 162.4 MW across 4 units), Buffalo Ridge II SD 210.0 MW, and Hardy Hills Solar IN 195.0 MW from 2,508 km away. Every entry is at least 890 km from the CA footprint, so 50/100/200 km all drop the whole population. Fix: when the run is footprint-scoped, keep only must-add plants within SEAM_PLANT_MAX_KM = 100 km of the union of regions_onshore + regions_offshore, measured in EPSG:5070. Plants inside the footprint have distance 0 and are always kept, so genuine near-seam plants still attach — match_plant_to_bus is deliberately left alone, since its unbounded second pass is correct once the leak population is filtered upstream. Gated on model_topology.include being truthy (read from snakemake.config in main(), threaded as the filter_plants_by_region(footprint_scoped=...) parameter). With the gate off not a single statement changes, so unfiltered interconnect/usa runs are byte-identical BY CONSTRUCTION. That gate is load-bearing, not cosmetic: against a full-western footprint the same population is mostly legitimate, and an unconditional 100 km bound would delete 8 plants / 694.9 MW (incl. Hardy Hills and Buffalo Ridge II) from an unfiltered western run. Every dropped plant is logged at WARNING with name, carrier, state, MW and distance, plus a summary line with count and total MW. New unit tests in workflow/scripts/test/test_seam_plants.py build synthetic regions/ReEDS shapes/membership so an in-footprint plant, a ~50 km seam plant and a ~500 km far plant all land in plants_must_add, then pin gate-off keeping all three (legacy behavior) and gate-on keeping the first two while dropping and loudly logging the third. workflow/scripts/test/: 41 passed, 10 skipped (pre-existing skips), no regressions. Full quantification, method and caveats: docs/superpowers/specs/2026-08-23-seam-plant-quantification.md — including three findings adjacent to this fix and deliberately NOT changed here: the primary sjoin leaks zero out-of-state plants; the plants_nearshore sjoin_nearest path compares EPSG:4326 degrees against EPSG:3857 metres and so matches nothing (11 SD plants / 230.0 MW silently dropped today); and 46 plants / 3,326.3 MW (CA 41, WA 5) fall outside every national ReEDS shape before the fallback runs. Results-changing for scoped runs; pending ledger countersignature. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DL-13: adopt the seam-plant bound on both harness sides Mirrors v1-epic d98cb93f onto the pinned anchor e7f8bd70 as the third ADOPTED-FIX patch, so the harness keeps comparing like-for-like instead of scoring the candidate against a known-leaking anchor. apply_seam_adoption does targeted string surgery rather than DL-12's whole-file adoption: v1-epic's add_electricity.py legitimately differs from the anchor's in the simplify-early bus2sub/sub_id removals, the length_factor=1.0 decision (DL-1/DL-2) and the schema-logging calls, so copying it wholesale would smuggle those unrelated deltas onto the anchor. The whole filter_plants_by_region body is byte-identical between e7f8bd70 and v1-epic, so the anchor takes the same footprint_scoped parameter plumbing as the candidate rather than an inlined-config variant. Verified AST-identical across both sides: the helper body (docstrings stripped), the SEAM_PLANT_MAX_KM value, the filter_plants_by_region signature, the gated call block and the main() wiring. The constant block and the helper body are sliced out of the LIVE candidate file so the numeric logic both sides run is the same text and any drift in v1-epic's helper re-triggers the forced rerun; only the four wiring edits, which must adapt to the anchor's own shape, are hardcoded. Safety rails match apply_powerplants_adoption: the candidate must carry the sentinel and yield both slices, all four needles are verified to occur exactly once against the PRISTINE anchor file fetched from git (not the possibly already-patched worktree), the pristine anchor must not already contain the sentinel, and the result is checked for end-to-end footprint_scoped wiring before anything is written. Idempotent on the sentinel; marks add_electricity for the one-shot forced rerun on newly-applied. Also wires EQ_UNTIL=assembled into run.py's build targets. paths.py has stopped the COMPARED pairs at the assembled stage since the harness was written, and exported assembled_target/anchor_assembled_target for the matching build targets, but run.py never used them — so an EQ_UNTIL=assembled run still drove the whole chain through a solve it then did not compare. At usa scope that is a national solve. Inert for the CA runs, which do not set the variable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Seam bound: measure distance per region instead of unioning the footprint The equivalence harness's CA prong 2 (simpl=20) died in add_electricity with GEOSException: TopologyException: side location conflict, raised by _drop_distant_seam_plants' pd.concat(region_geoms).union_all(). Root cause is the reprojection, not the source data: regions_onshore_s20 and regions_offshore_s20 are 100% valid as stored in EPSG:4326, but converting them to EPSG:5070 for the distance measurement leaves 9 of 29 polygons self-intersecting or degenerate ("Too few points in geometry component"), and GEOS union_all refuses invalid input. Prong 1 never tripped it because the simpl='' layer reprojects to 2,014 fine-grained polygons that all stay valid -- so the bug was latent behind the granularity of the first prong tested. The union was only ever a means to a distance, and dist(p, union(R_i)) equals min_i dist(p, R_i), so take the minimum over regions directly and skip the union. Pairwise distance is robust to self-intersection where the union is not. Verified equal, not merely similar: on the simpl='' layer where union_all does succeed, the two methods agree to 0.000000000 m over 60 probe points; on the simpl=20 layer the union raises and the per-region path returns clean distances. Cost is 0.215 s at prong-1 scale (23 plants x 2,014 regions). Also drops empty/NA geometries before measuring, and returns early if nothing survives. New regression test builds a self-intersecting bowtie plus an overlapping box -- the minimal shape that reproduces the GEOS failure -- placed far from every test plant so the expected keep/drop set is unchanged and only the distance path is under test. Confirmed to fail with the original union_all implementation (identical TopologyException) and pass with this one. workflow/scripts/test/: 42 passed, 10 skipped. Harness side: apply_seam_adoption's idempotence is now by CONTENT rather than by the sentinel string, matching apply_powerplants_adoption. Sentinel-based skipping would have left the anchor running the pre-fix helper forever, since the sentinel is present either way; content comparison re-applies the patch and re-arms the one-shot forced-rerun marker whenever the candidate's helper changes. Verified: the anchor picked up this fix automatically on the next provision, and both sides remain AST-identical in helper body, gated block, signature and main() wiring. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DL-13 docs: ledger row, CHANGELOG entry, gitignore the rerun marker Records the countersigned seam-plant bound adoption (d98cb93f + 103f2194 + anchor mirror 85cda599) with the corrected symmetric magnitude (1,725.0 MW leaves the assembled network on both sides, not the 27 MW name-count estimate), the harness verdicts (prong 1 PASS 0 live, objective rel 2.46e-06; prong 2 PASS with DL-9 absolute gaps exactly unchanged), and the GEOS per-region-distance robustness fix. Also gitignores the transient .eq-force-rerun marker. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Ledger: refresh DL-7 restatement bound to post-DL-13 artifacts fuel_cost residual is 8.1277 $/MWh (40.24%) on current artifacts (8.4498 when first measured post-DL-12); adds the three-fix invariance of the DL-9 absolute gaps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DL-13: record the completed usa validation leg usa data-stage harness PASS 0 live/113 - identical count and class structure to the pre-adoption baseline, confirming DL-11/12/13 left the national comparison untouched. Seam gate verified on all axes (source AST, config eval, 0 seam-drop lines in both sides' usa logs vs 24 at CA). usa report regenerated, superseding the stale 2026-08-22 artifacts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Raise pins to pandas 3 optional-dependency floors; real NaN assert in MECS path pandas 3 raised its optional-dependency floors, which only error at use time (import_optional_dependency), so tests missed them: openpyxl 3.1.2 -> 3.1.5 (caught via build_demand's MECS read_excel path), matplotlib 3.8.0 -> 3.9.3, scipy 1.11.3 -> 1.14.1. All 36 floors in pandas.compat._optional.VERSIONS now audited clean against the lock. pyproject, environment.yaml and uv.lock updated in sync. build_demand.py: replace the vacuous `assert not (mecs == np.NaN)` (NaN never compares equal, so it always passed; np.NaN is also removed in numpy 2) with a real `mecs.isna()` check. Verified passing against the actual MECS workbook (table3_2.xlsx, 405x9, zero NaNs) -- safe by construction too, since dropna/replace run before astype(float). No other capitalized np.NaN usages repo-wide. Results effect: none. Unit tier 45 passed / 1 skipped, static tier 72 passed on the bumped matplotlib/scipy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix two silent pandas-3 astype(str) NaN regressions caught by the harness pandas 3 preserves missing values through .astype(str) from every source dtype (object, float64, str, Int64, datetime64, category all verified) instead of stringifying them to "nan" as pandas 2 did. Two load-bearing sites relied on the old behavior: - build_powerplants merge_ads_data: ADS "Long Name" match keys (343 NaN in GeneratorList.csv) crashed the re.sub name normalization (loud). - build_powerplants set_parameters: build_decade = build_year.astype(str) feeds impute_missing_plant_data's groupby + inner merge as a key. Under pandas 3 the 923 plants with no generator_operating_date (767 proposed units / 74.7 GW: 411 solar, 173 batteries, 47 onwind, 19 CCGT) get a NaN key, match no group, and are silently dropped from powerplants.csv; under pandas 2 they landed in the "nan0s" bucket and survived to have build_year backfilled from current_planned_generator_operating_date in add_electricity (silent, results-changing). Both pinned byte-identical to pandas-2 semantics with .fillna("nan"). Found during the Tier-C equivalence re-baseline (first full pipeline run under the new stack) plus an 82-site audit of every pandas astype(str)/ map(str) call; all other sites verified SAFE empirically, two latent config-gated sites flagged in the changelog. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Untrack local harness symlinks; changelog note for the astype(str) fixes The previous commit accidentally tracked three machine-local symlinks (workflow/data, workflow/cutouts, .worktrees/anchor-e7f8bd70) created to run the equivalence harness from this worktree; remove them from the index and gitignore them. Add the changelog entry for the two pandas-3 astype(str) NaN regressions (ADS Long Name crash; 74.7 GW silent proposed-generator drop via the build_decade imputation key). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Changelog: record the two pandas-3 astype(str) regressions caught by the re-baseline Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Tier-C re-baseline adjudication: both prongs PASS under pypsa 1.3/pandas 3 Clean from-scratch candidate rebuild of both CA prongs against the anchor (develop e7f8bd70, pypsa 0.30 env). Pre-adjudication: prong 1 16 live / 88 findings, prong 2 0 live / 3. All 16 live findings verified as zero-physics artifacts of the 0.30 -> 1.3 conventions, in 3 classes: - sub_network topology metadata v1 serializes and 0.30 left empty (11 findings; Bus/Line values + SubNetwork rows) -> waived (DL-15). - StorageUnit.cyclic_state_of_charge_per_period: the attribute is not stored in the anchor netCDFs at all (0.30 omitted default-valued attrs); the pypsa-1.3 reader backfills the NEW default False while the anchor's solve-time behavior was True, and the candidate pins True explicitly. Comparison manufactures the diff on read (4 findings) -> waived (DL-15) with the file-level proof in the justifications. - Network.objective: candidate folds objective_constant in (linopy 0.9), anchor split it out. Like-for-like (objective + objective_constant) agrees to rel 5.3e-07 -> fixed as a comparator NORMALIZATION in compare.py (_total_objective), not a waiver; the objective gate stays live and correctly measured. Prong-2's DL-9 objective waiver (a real physics delta) is unaffected. Post-adjudication: prong 1 PASS 0 live / 87 (one fewer total because the objective finding is normalized away, not waived); prong 2 PASS 0 live / 3. DL-15 drafted in the deltas ledger, PENDING COUNTERSIGNATURE (DL-14 is reserved by proto/nearshore-crs-fix). Known follow-up: extend the three classes' verification + waivers to the usa leg when it is rebuilt on the 1.3 stack. Changelog updated; its stale "harness not re-run" clause corrected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #701
Changes proposed in this Pull Request
Updates PyPSA-USA to track changes made in PyPSA V1
Checklist
envs/environment.yaml.config.default.yaml.doc/configtables/*.csv.