Untrack workflow/config/config.common.yaml (unbreaks init_pypsa_usa.sh) by ktehranchi · Pull Request #791 · PyPSA/pypsa-usa

ktehranchi · 2026-08-28T20:53:42Z

Changes proposed in this Pull Request

Untracks workflow/config/config.common.yaml. One deletion from the index; no file content changes, no .gitignore change.

Why it is wrong to track it

That directory is user-generated by design:

.gitignore:15 already contains config/, which covers workflow/config/.
README.md:23 tells users to run bash init_pypsa_usa.sh to populate it from workflow/repo_data/config/.
All 9 other templates in that directory are untracked. Only this one is tracked.

It was committed by accident in 16813a4 (#745).

What tracking it breaks

init_pypsa_usa.sh refuses to copy anything when workflow/config/ is non-empty:

existing_files=$(ls "$destination" | grep -v ".gitkeep")
if [ -z "$existing_files" ]; then
    cp -r "$templates"/* "$destination"
else
    echo "Existing config files found in '$destination'. Delete the following files and rerun."

The tracked file is present in every fresh clone, so new users always hit the error branch. Reproduced on current develop:

$ bash init_pypsa_usa.sh
Existing config files found in 'workflow/config'. Delete the following files and rerun.
config.common.yaml
$ ls workflow/config/
config.common.yaml

They receive none of the other 9 configs — including config.plotting.yaml, config.api.yaml, and config.sector.yaml, which workflow/Snakefile:85-89 loads unconditionally.

With this change, on a fresh checkout of this branch:

$ bash init_pypsa_usa.sh
Copying config files from 'workflow/repo_data/config' to 'workflow/config'...
$ ls workflow/config/
config.api.yaml   config.default.yaml       config.equivalence.yaml  config.sector.yaml    config.tutorial.yaml
config.cluster.yaml  config.equivalence-usa.yaml  config.plotting.yaml     config.test.yaml      policy_constraints
config.common.yaml

Second problem this fixes

The stranded copy has drifted from its template. It still pins:

pudl_path: s3://pudl.catalyst.coop/v2025.2.0

against the template's v2025.5.0. Anyone whose run picked up the tracked copy has been silently building on an older PUDL/EIA release.

Relationship to #790

#790 adds tests/docs/test_docs_config.py::test_config_tree_sync, which skips only when workflow/config/ holds no YAML. The tracked file defeats that skip, so the test fails on any fresh clone. This change makes that test behave as intended. #790 also edits workflow/config/config.common.yaml; that edit should be dropped when #790 is refreshed onto this.

Checklist

(Deletion from the index only — no source or config content changed.)

I tested my contribution locally and it seems to work fine.
Code and workflow changes are sufficiently documented.
Changed dependencies are added to envs/environment.yaml.
Changes in configuration options are added in all of config.default.yaml.
Changes in configuration options are also documented in doc/configtables/*.csv.

Verification performed: ran init_pypsa_usa.sh on a fresh checkout of develop (fails as shown) and on a fresh checkout of this branch (copies all 10 entries). No pipeline run.

This file is user-generated: .gitignore already ignores `config/` (line 15), README tells users to run `bash init_pypsa_usa.sh` to populate workflow/config/ from workflow/repo_data/config/, and every other template in that directory is untracked. It was committed by accident in 16813a4 (PyPSA#745). Tracking it broke the documented setup flow. init_pypsa_usa.sh refuses to copy anything when workflow/config/ is non-empty: existing_files=$(ls "$destination" | grep -v ".gitkeep") if [ -z "$existing_files" ]; then cp -r ... else echo "Existing config files found ... Delete the following files and rerun." fi Since the tracked file is always present in a fresh clone, new users hit the error branch and receive NONE of the 8 templates. The stranded copy has also drifted from its template — it still pins `pudl_path: s3://pudl.catalyst.coop/v2025.2.0` against the template's v2025.5.0, so anyone relying on it silently builds on older EIA data. No .gitignore change is needed; the existing `config/` rule covers it.

Refresh the docs overhaul onto develop after the pypsa 1.3.0 / linopy 0.9.1 / pandas 3.0.5 migration (PyPSA#778), the population-based demand allocation (PyPSA#777), and the untracking of workflow/config/config.common.yaml (PyPSA#791). Conflicts: - docs/source/configtables/electricity.csv: kept the overhaul's restructured table and re-added develop's electricity.demand.bus_allocation row under the branch's '-- ' nesting convention. - workflow/config/config.common.yaml: resolved to develop's deletion; the file is user-generated and now gitignored, so the overhaul's edit is dropped (workflow/repo_data/config/config.common.yaml remains canonical). Also corrected docs/source/about-install.md, which still advertised the pre-migration pins and claimed PyPSA 1.x was unsupported.

Refreshes the cleanliness sweep onto develop after the pypsa 1.3.0 / linopy 0.9.1 / pandas 3 migration (PyPSA#778), the config.common.yaml untrack (PyPSA#791) and the equivalence-harness recalibration (PyPSA#792). Conflicts, all in favour of develop's migrated code: - _helpers.py: develop deleted load_network (pypsa.descriptors.Dict and override_components are gone in v1); the sweep's own deletions of the unreferenced pdbcast / aggregate_* / get_aggregation_strategies / load_network_for_plots / setup_custom_logger helpers still apply, so the file is a pure deletion relative to develop. - add_extra_components.py: attach_stores stays deleted (still dead on develop -- its only call site is a comment); the sweep's copy_timeseries_for_suffix helper is kept. - summary.py: keeps develop's migrated imports (pypsa.statistics.get_bus_and_carrier no longer exists) and drops the unreferenced _helpers.configure_logging import along with the __main__ block that used it; get_energy_total / get_demand_base / get_capacity_base / get_capacity_brownfield / get_capital_costs stay deleted. _iter_components remains in use by get_energy_timeseries. No pre-1.3 API is reintroduced and no cleanup hunk was applied on top of code the migration rewrote. Dependency files are untouched and workflow/config/ holds only .gitkeep.

…#793) #791 untracked workflow/config/config.common.yaml on the correct grounds that the directory is user-generated. But that file had drifted from its repo_data template and was silently supplying three keys the template never had: renewable: dataset: godeeep renewable_weather_years: [2019] renewable_scenarios: ["rcp85cooler"] So after #791, `init_pypsa_usa.sh` copies a template that omits them, and retrieve.smk:231 dereferences config["renewable"]["dataset"] whenever enable.build_cutout is false. Four of the five shipped configs (tutorial, test, equivalence, equivalence-usa) set build_cutout: false and define no dataset of their own, so a fresh clone crashes at Snakefile parse time with KeyError: 'dataset'. Only config.default.yaml defines it. Simulating Snakemake's layered load (common + main) before this commit: 2bc715f renewable.dataset='godeeep' renewable_scenarios=['rcp85cooler'] develop 125994a KeyError 'dataset' <- parse crash Values are copied verbatim from the file #791 removed, so this restores exactly the behaviour every run has had; it is not a new default. The same keys also guard add_electricity.py:646, which indexes renewable_scenarios[0] directly once the dataset is godeeep. After this commit all five shipped configs resolve dataset=godeeep and renewable_scenarios[0]=rcp85cooler.

- data-demand: describe both `electricity: demand: bus_allocation:` modes (`population`, the default since PyPSA#777, and `breakthrough`) and the `load_weight` column they populate, instead of claiming the weight is always Breakthrough `Pd`. - model-workflow: `add_sectors` runs `add_sectors.py` (not `build_sector.py`), and `export_statistics` runs `plot_statistics.py` in `export` mode (there is no `export_statistics.py`). - model-components: `model_topology.topological_boundaries` accepts `county`, `reeds_zone`, and `state`; `balancing_area` raises in `aggregate_to_substations`. - about-introduction: 2030/2040/2050 are the EFS horizons; EER covers 2021, 2025, 2030, 2035, 2040, 2045, 2050 (matches data-demand and `ReadEer.MODEL_YEARS`). - data-policies / release-notes: after PyPSA#791 `workflow/config/` is untracked; the policy CSVs are tracked in `workflow/repo_data/config/policy_constraints/` and copied by `init_pypsa_usa.sh`. - about-usage: the default config uses 33 clusters, not 30. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ction, generated figures (#790) * Pre-aggregate SCD tables in build_powerplants to fix memory blow-up The LEFT JOINs in load_eia_operable_data and load_heat_rates_data joined the multi-year EIA-860 SCD tables (yearly_generators, scd_plants, scd_generators_energy_storage) without any date constraint. Each generator carries ~24 years of annual snapshots, so the intermediate join produced ~7,000 rows per generator (~200M rows for load_heat_rates_data) before the final aggregation collapsed them. Pre-aggregating each SCD table to one row per plant/generator using the same array_agg(... ORDER BY report_date DESC) FILTER (WHERE ... IS NOT NULL)[1] "latest non-null" pattern already used elsewhere in the file keeps the projection identical while reducing the join cardinality from ~24x24 to 1x1. Measured against s3://pudl.catalyst.coop/v2025.5.0: - peak RSS: 17.3 GB -> 2.2 GB - runtime: 503 s -> 24 s Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add scatter plot assets for PR description (drop before merge) These PNGs are referenced in the PR body to visualize the original-vs- refactored marginal_cost and heat_rate distributions. Safe to remove once the PR is reviewed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Remove PR-assets PNGs (not opening that PR yet) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Split simplify_network into aggregate_to_substations + cluster_simpl Phase 1 of the simplify-early refactor. Splits the existing simplify_network rule into two rules with no behavior change for the default kmeans algorithm: - aggregate_to_substations: pure-topology aggregation to substations (convert_to_voltage_level, remove_transformers, busmap-by-sub_id). Writes resources/{interconnect}/elec_b.nc + busmap_b.csv. - cluster_simpl: optional k-means/modularity reduction to {simpl} clusters. Reads elec_b.nc, writes elec_s{simpl}.nc + simpl regions, matching the former simplify_network output interface so downstream rules (cluster_network etc.) are untouched. HAC clustering is dropped from both rules (cluster_network now raises if algorithm='hac') along with the unused 'to_substations' and 'feature' config knobs. simplify_network.py is removed; its functions move into aggregate_to_substations.py. Snakemake dry-run on the default Western config produces the expected chain: aggregate_to_substations -> cluster_simpl -> cluster_network. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Remove unused config keys and unused snakemake rule params Dead-code cleanup audit against v1-epic. No behavior change. YAML config keys removed (no code reference): - config.plotting.yaml: costs_max, costs_threshold, energy_max, energy_min, energy_threshold (with matching docs/source/configtables/plotting.csv rows) - config.tutorial.yaml: sector.natural_gas.allow_imports_exports - config.common.yaml: renewable.hydro.{PHS_max_hours, resource.hydrobasins, resource.flowspeed, hydro_max_hours, clip_min_inflow, normalization, multiplier}; atlite.default_cutout; electricity.prm regional block - config.default.yaml: model_topology.interface_transmission_limits; solving.mem Snakemake params: declarations removed (declared but never read by the target script): - build_bus_regions: focus_weights - add_extra_components: ucap (script reads via snakemake.config instead) - prepare_network: adjustments, co2base - add_sectors: electricity, costs, plotting - plot_network_maps: plotting, retirement - plot_statistics: plotting - plot_natural_gas: plotting - solve_network: planning_horizons, transmission_network, sector_config - solve_network_validation: planning_horizons, co2_sequestration_potential Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Reorganize resources/ into category-first subfolders Outputs under resources/ previously sat at a single interconnect-rooted level (e.g. resources/texas/elec_b.nc, resources/texas/busmap_s50.csv, resources/texas/profile_solar_s50.nc). Files of unrelated types were intermingled, making the tree hard to navigate. This change introduces a category-first layout: the top level under resources/ is what the file IS (networks/, busmaps/, profiles/, ...) and {interconnect} becomes the next level down. resources/ networks/{interconnect}/elec_*.nc busmaps/{interconnect}/busmap_*.csv, bus2sub.csv, sub.csv profiles/{interconnect}/profile_*.nc, nrel_mapping_cache geospatial/{interconnect}/*.geojson, bus_gis.csv, lines_gis.csv costs/costs_{year}.csv, sector_costs_{year}.csv prices/{interconnect}/{state,ba}_*_prices.csv, pudl_fuel_costs.csv demand/{interconnect}/{end_use}_*.csv|pkl population/{interconnect}/pop_layout_*.nc|csv temperature/{interconnect}/temp_{soil,air}_*.nc heating_cop/{interconnect}/cop_{soil,air}_*.nc co2/{interconnect}/co2_storage_*.csv powerplants/powerplants.csv Implementation: - workflow/Snakefile defines twelve category constants (NETWORKS, BUSMAPS, PROFILES, GEOSPATIAL, COSTS, PRICES, POWERPLANTS, DEMAND, HEATING_COP, TEMPERATURE, POPULATION, CO2) composed from RESOURCES, so the RDIR (run name) and shared_resources prefixes still work. - Every RESOURCES + "{interconnect}/<file>" reference in the .smk files swapped to the matching category constant. No filenames change. - "Geospatial/" (capital G) renamed to "geospatial/" (lowercase) for consistency with the new top-level subfolders. - "resources/powerplants.csv" hardcoded literal moved to "resources/powerplants/powerplants.csv" (still shared across runs). - Two off-rule literal paths updated: a __main__ test path in build_natural_gas.py and argparse defaults in plot_caps_summary.py. - Doc reference in about-usage.md sharpened to resources/networks/. Verified by snakemake -n: full DAG resolves end-to-end for resources/Default/networks/texas/elec_base_network_l_pp.pkl, with each intermediate input/output routed to the new category folder. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Merge stacked PRs #9 and #11 into v1-epic Brings in Phase 2 (move simplify/cluster_simpl ahead of demand+RE) and Phase 3 (aggregate EGS supply curves through cluster_simpl busmap). Both PRs were originally merged into their stacked-base branches rather than v1-epic; this merge propagates their changes up. Resolved conflicts in build_electricity.smk by taking the post-#12 resources/ layout (category-first constants: NETWORKS, GEOSPATIAL, PROFILES, DEMAND, BUSMAPS) and applying Phase 2's path repointing (network refs to elec_s{simpl}.nc, _s{simpl} suffix on demand/profile outputs). Repointed aggregate_egs outputs and add_electricity EGS inputs to PROFILES/BUSMAPS per the new layout. * Add design spec for PyPSA network schema tracking Two-part approach: a curated docs/network-schema.md catalog of custom columns plus a _helpers.log_network_schema helper that logs per-script column entry/exit diffs. Motivated by recurring consense aggregation crashes (most recently LAF_state in aggregate_to_substations). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add implementation plan for network schema tracking Six tasks: add helper + tests, wire into topology / add-* / sectors chains, smoke test on test_small western, seed catalog from logs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add log_network_schema helper for per-script column tracking Logs component row count and column set on entry; on exit emits row and column diffs vs. the entry snapshot. Logging only — no asserts. Wires into scripts in follow-up tasks; tested in isolation here. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Wire log_network_schema into topology chain scripts Adds entry/exit schema logging to build_base_network, aggregate_to_substations, cluster_simpl, cluster_network. Logging only; no behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Wire log_network_schema into add_electricity / add_extra_components / add_demand Logging only; no behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Wire log_network_schema into prepare_network / add_sectors / solve_network Logging only; no behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add testing strategy design spec Tiered test pyramid (static <30s + integration <5min) targeting the classes of breakage that the simplify-early refactor (PRs #7-#12) made visible: path/wiring drift, dead config keys, silent artifact-shape regressions. Schema-catalog assertions deferred to a follow-up PR after the schema-tracking spec lands. Brainstorming output. Five-PR migration plan included. * Seed docs/network-schema.md from observed pipeline logs Initial catalog of custom columns on PyPSA components, populated from [schema ...] log output of build_base_network and aggregate_to_substations entry on the small_mh western config. Sector- and electricity-stage columns will be added as those rules get exercised under the schema logger. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Tighten LAF_state catalog row tone Drop editorial framing ("the bug class this catalog was created to address"). Keep the factual note that LAF_state is missing from bus_strategies and the fix is tracked separately. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add implementation plan for testing strategy Five-PR migration: scaffolding → Tier A static checks → Tier B fixture → Tier B full assertions → CI wiring. Plan includes exact code, file paths, verification commands per step. PR 6 (schema assertions) deferred pending the schema-tracking initiative. * Add pytest config and test extras * Add tests/ directory skeleton * Wire existing unit tests into the 'fast' marker via repo-root conftest Adds a repo-root conftest.py that auto-marks every test collected from workflow/scripts/test/ with the 'fast' marker, so pytest -m fast picks them up without per-file changes. Also skips 9 pre-existing test failures unrelated to this PR (RPS constraint helper signature drift and ERM API drift) so pytest -m fast exits 0. Each skip carries a reason pointing back to v1-epic. * progress in refactor * Add cluster_simpl county fast-path design spec Approved-design doc for a new simpl="county" wildcard value that bypasses k-means and uses the substation network's county FIPS (prefixed with reeds_zone) as a direct busmap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Draft config-cleanup formatting spec Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add cluster_simpl county fast-path implementation plan Four-task TDD plan: extract resolve_simpl_mode and build_county_busmap helpers, wire them into __main__ dispatch, and document the new simpl="county" wildcard value. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add resolve_simpl_mode dispatch helper to cluster_simpl Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add build_county_busmap helper with missing-county guard Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Document and standardize repo_data config files Rewrite the three docs-source configs (default, tutorial, common) so a new user can read top-to-bottom and understand every option without cross-referencing scripts. Pure formatting + commenting; no key renames or hierarchy changes, no behavior changes. - Per-section banners above each Sphinx `# docs :` sentinel - Inline comment on every option (allowed values, units, acronym glosses) - File-header block per config (purpose, audience, merge order) - Removed in-section comment lines that bled into adjacent Sphinx slices - Normalized False→false - Verbose missing-key surface in default.yaml (commented-out examples) - Added minimal disabled co2/dac blocks to tutorial.yaml so it loads (build_electricity.smk unconditionally reads config["co2"]["storage"]) Verified: all three YAMLs parse; Sphinx slicer ranges still resolve; `snakemake -n data_model` builds DAG cleanly with both default and tutorial configs. Spec: docs/superpowers/specs/2026-05-21-config-cleanup-design.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Revert "Test scaffolding: pytest config + fast-marker for existing unit tests" * Revert "Merge pull request #18 from ktehranchi/revert-13-tests/pr1-scaffolding" This reverts commit 21c3e95, reversing changes made to 5e351fb. * Add pipeline-equivalence master spec, glossary, and v1-epic change-log Approved via grilling session: anchor = upstream/develop e7f8bd7, two-prong pinned-busmap equivalence protocol, CA harness (USA deferred), deltas ledger + waivers, profile-first performance phase. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Land Tier A + Tier B fixture into v1-epic (#14 + #15) (#20) * Add Tier A: path validator + fix surfaced RESOURCES violations Adds tests/static/test_paths.py which scans every .smk file (and the root Snakefile) for ``RESOURCES + "{interconnect}/..."`` literals, the forbidden form after PR #12 reorganized resources/ into category-first subfolders (NETWORKS, HEATING_COP, POPULATION, CO2, ...). Routes every rule input/output through the right category constant so the new test passes: - workflow/Snakefile data_model rule: RESOURCES -> NETWORKS (clustered network output). - workflow/rules/build_sector.smk sector_input_files() + add_sectors: RESOURCES -> NETWORKS (elec_*_ec_l*_opts*.nc inputs and the add_sectors output), HEATING_COP (cop_soil/cop_air *.nc inputs), POPULATION (pop_layout_elec_*.csv input), CO2 (co2_storage_*.csv input). - workflow/rules/solve_electricity.smk solve_network rule: RESOURCES -> NETWORKS (network input). - workflow/rules/validate.smk solve_network_validation rule: RESOURCES -> NETWORKS (network input). These changes align with the producer rules, which already write to the correct category-prefixed paths; the bug was on the consumer side referencing stale RESOURCES + {interconnect}/ layouts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add Tier A: snakemake -n dry-run resolves on tutorial + default configs Adds tests/static/test_dag_dryrun.py which runs ``snakemake -n`` against config.tutorial.yaml (cluster_network + solve_network) and config.default.yaml (cluster_network) to catch typo'd rule names, wildcard mismatches, syntactically broken .smk files, and unresolved config keys before they reach a real build. A session-scoped autouse fixture mirrors init_pypsa_usa.sh, copying any missing config templates (and policy_constraints/) from workflow/repo_data/config/ into workflow/config/ so the test runs cleanly in CI without depending on the init script. Surfaced two real KeyErrors in the merged config that blocked the dry-run; fixed by extending workflow/config/config.common.yaml (and the canonical template at workflow/repo_data/config/config.common.yaml) with sensible defaults so every loaded config exposes the keys the rules read unconditionally: - ``co2.storage`` / ``co2.network.*`` — read in build_electricity.smk, build_sector.smk, add_sectors.py, add_extra_components.py without an ``enable`` guard. Defaults to ``false`` so behavior is unchanged. - ``renewable.EGS.drilling_cost`` — read in add_electricity.py to switch between the ``capex_usd_kw`` and ``advanced_capex_usd_kw`` columns of the EGS supply curve. Defaults to ``base`` (selects the non-advanced column). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add Tier A: config-key existence validator Adds tests/static/test_config_keys.py which AST-walks every workflow/scripts/*.py module, extracts subscript chains rooted at ``snakemake.config[...][...]``, and asserts each chained key path exists in the merged default config. Catches the class of bug fixed in PR #10 — scripts dereferencing config keys that no live YAML defines (typos, dropped flags, dead branches). The merged config is built in the same order the Snakefile loads configfiles (cluster -> common -> plotting -> api -> sector -> default), with each file preferred from workflow/config/ if present and falling back to the canonical workflow/repo_data/config/ template otherwise. The tutorial config is excluded — it's a minimal opt-in subset, not a key-vocabulary source of truth. All 57 parametrized scripts pass after the ``co2`` / ``renewable.EGS. drilling_cost`` defaults added in the preceding commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add minimal CA-only test config for tests/integration Adds workflow/repo_data/config/config.test.yaml — a minimal CA-only Western slice (simpl=20, clusters=4m, 1-day snapshots, 2030 horizon) sized for <5 min builds. Mirrors the seeding pattern used for other config templates in repo_data/config (the autouse fixture in tests/static/test_dag_dryrun.py copies missing files from there into workflow/config/ at test time). Includes explicit electricity, conventional, lines, links, costs, clustering, custom_files, and solving blocks needed for the DAG to resolve to cluster_network without pulling config.default.yaml inheritance. * Add Tier B fixture + smoke test (cluster_simpl bus count) - `tests/integration/conftest.py` provides a session-scoped `built` fixture that runs `snakemake --until cluster_network` once per session using config.test.yaml. Skips gracefully when `workflow/data`, `workflow/cutouts`, or `workflow/repo_data` are missing, letting developers run `pytest -m fast` without setting up the data deps. Uses `--scheduler greedy` to avoid snakemake's ILP-based scheduler (which can fail when CBC isn't available). Includes its own autouse `_seed_runtime_configs` fixture mirroring Tier A so it works regardless of collection order. - `tests/integration/test_artifacts.py` has one placeholder assertion (`test_post_cluster_simpl_bus_count`) that loads elec_s20.nc and asserts bus count == 20. PR 4 will flesh out the full per-stage assertion suite. * PR3 quality fixes: TimeoutExpired handling, xdist guard, solving cleanup - Catch subprocess.TimeoutExpired and surface stderr tail in the fail message (previously the hang case dropped helpful output). - Add pytest_collection_modifyitems hook that exits cleanly if pytest-xdist is active for integration tests (avoids snakemake lock races). - Strip the dead solving block from config.test.yaml — fixture stops at cluster_network, solver never invoked. - Inline comment for --scheduler greedy rationale. - BuiltArtifacts docstring cross-referencing config.test.yaml constants. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Tier B: per-stage artifact assertions (#16) * Tier B: per-stage shape assertions for the five build artifacts Replaces the single smoke test from PR 3 with a class-per-stage layout covering aggregate_to_substations, cluster_simpl, add_demand, add_electricity, and cluster_network. Asserts file existence, bus counts matching wildcards, no-NaN in load-bearing columns, netCDF roundtrip, busmap coverage, and filename wildcard propagation. * Replace broken ./test.sh with fast-tests + e2e-tests CI jobs (#17) Two parallel jobs gate every push/PR to master and v1-epic: - fast-tests: no data deps, ~1min wall-clock, runs pytest -m fast - e2e-tests: cached data+cutouts, ~10min wall-clock, runs pytest -m integration Plus a weekly Tuesday cron (upstream-regression) that pulls latest PyPSA/atlite/linopy from master and runs the full suite. The previous `./test.sh` step was a silent no-op (file didn't exist). Manual post-merge step: mark fast-tests and e2e-tests as required status checks in branch protection for master and v1-epic. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Revise equivalence protocol to config-only determinism (no pinned busmap) cluster_network kmeans is seeded (random_state=0) and unchanged on both branches, so identical inputs yield identical busmaps from basic config alone; custom_busmap repair drops off the critical path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix four post-refactor breakages found during validation runs - add_electricity: map buses to states via reeds_state->full name for capital-cost multipliers; guard match_plant_to_bus when plants lack a 'country' column; group renewable capacities by bus identity (bus == substation after aggregate_to_substations) - build_demand: state-zone lookup via reeds_state->full name - build_powerplants: fill missing EIA-860 summer/winter derates with 1.0 (multi-unit CC sub-units otherwise propagate NaN into p_max_pu) - cluster_network: preserve 'country' column through cluster_regions dissolve (consumed by match_plant_to_bus) Equivalence verdict pending (Phase 3 of the 2026-08-07 master spec): the derate fill plausibly changes results vs anchor and needs a ledger entry or a fix-to-parity. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix upstream-merge fallout caught by Tier A checks - postprocess.smk: repoint upstream's new rule block from the old {interconnect}/Geospatial/ layout to the GEOSPATIAL category constant - common.smk: materialize constants.py into the snakemake source cache so _helpers.py's module-level import resolves under a fresh cache (broken since upstream #764 on any machine without a warm cache) - change-log: record the upstream sync merge and its resolutions Tier A: 101 passed, 0 failed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add CA equivalence harness implementation plan (Phase 2-4) Grounded in a 5-reader research workflow over both branches: artifact path pairing table, simpl='' viability, shared-config key requirements, anchor worktree provisioning steps, loader/report tooling inventory. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * T1: add shared equivalence-harness config (both branches) CA-only Western slice, Jan 6-19 2019, clusters=10, seeded gurobi barrier. Carries the anchor-required keys (co2.storage, clustering feature keys) and non-null renewable_land_access for godeeep. Both prong targets dry-run-resolve on v1-epic. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * T2-T4: equivalence harness package + AEO scenario config fix Harness: paths.py (verified pairing map), build.py (dual-side driver, anchor worktree provisioning, documented infra patch), compare.py (D2/D7 tolerances, waivers), report.py (self-contained HTML), run.py CLI, tier-C pytest marker. Config fix (debug agent): build_cost_data crashed on both branches because the script's 'Reference' AEO fallback matches zero PUDL rows; costs.aeo.scenario: reference added to the shared harness config. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Change-log: PR #21 bugfixes merged during harness bringup Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Equivalence config: clusters 10 -> 4m (reeds transport-model constraint) cluster_network's reeds transport-model path requires clusters == the footprint's ReEDS zone count (4 for CA); discovered by the anchor build assert. Even better for prong 1: the zonal busmap is pure attribute membership - no kmeans anywhere in the chain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Equivalence config: add hard-required costs sub-keys (itc/ptc/max_growth/emission_prices/social_discount_rate/min_year) Direct-subscript reads in add_extra_components and prepare_network on both branches; values mirror config.default.yaml. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix simpl='' identity branch: normalize region bus names (harness catch) cluster_simpl's pass-through branch copied regions geojson verbatim, keeping float-formatted names ('35827.0') while the network/busmap use bare IDs ('35827'). The godeeep profile path's three-way bus intersection came up empty, silently writing 0-bus profiles that crashed add_electricity two rules later. Fix normalizes names in the identity branch (same as cluster_regions does for kmeans), adds a fail-loud disjoint-bus-spaces guard in build_renewable_profiles, and skips carriers with no profile buses in attach_wind_and_solar. Results effect: simpl='' runs regain silently-dropped RE generators; simpl=N unchanged. Found by the first candidate equivalence build. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Equivalence config: solve-stage policy-constraint keys (both branches) regional_Co2_limits (read by add_regional_co2limit, opts/policy.py:548, fired by the REM opt), plus sibling TCT/RPS keys. Dead keys (transmission_interface_limits, SAFE_*, agg_p_nom_limits) deliberately omitted - zero readers on either branch. Both prong-1 solves now complete optimally: candidate obj -2.56292e9, anchor -2.55060e9 (0.48% delta - goes to the harness comparison for adjudication). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix double-applied length_factor in transmission capital costs aggregate_to_substations folds lines.length_factor into length; the post-reorder add_electricity call multiplied it again (hav x 1.5625 instead of x 1.25 -> 25% line CAPEX inflation in line-preserving runs). Harness catch: assembled-stage comparison, ratio 1.2486 on 2810/2811 lines vs anchor. Clustered/solved outputs of reeds transport runs were unaffected (ITL costs recomputed identically on both branches). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Comparator: semantic normalization + fan-out suppression Float-label normalization, Load re-keyed on bus attr, demand pair corrected to system-total check (anchor CSV is nodal), per-frame finding cap with suppression marker, Generator[bus,carrier] p_nom aggregate view. Prong-1 findings: 16882 -> 379; profiles fully clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Deltas ledger + first waivers (DL-1..DL-4, provisional pending user sign-off) Stage-ordering capital-cost residues (clustered stage verified identical), plotting palette, slack-bus bookkeeping. Open findings (demand -6.3%, hydro attachment, storage sets, naming, objective 0.48%) remain unwaived pending investigation verdicts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Ledger: DL-5..DL-7 (Catalina placement, naming stage-ordering, aggregation composition) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix hydro attachment and demand conservation (harness catches #3 and #4) Hydro: attach_breakthrough_renewable_plants filtered raw plant.csv bus_ids against the substation-level network index - 12,848 MW of CA hydro silently dropped, 128.6 MW of Eastern hydro attached at colliding ids. Fix remaps through bus2sub + busmap_s{simpl} (patterned after aggregate_egs; busmap_s added to rule inputs). Verified: hydro p_nom 12,976.8 MW == anchor exactly; zero Eastern collisions. Demand: remove_transformers dropped 420 trafo-secondary buses with their Pd/LAF_state before demand existed; build_demand allocates by LAF_state without renormalizing -> -6.281% total demand and 95 zero- load substations. Fix transfers Pd/LAF_state through the trafo map and composes it into busmap_b.csv (full 4,248-bus coverage). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Prong 1 GREEN: finalize waivers DL-5..DL-8; ledger resolution note All fixes landed; solved objective rel diff 6.6e-05 (gate 1e-3); comparison: 0 live / 305 total findings. Every waived class carries a ledger row with quantified evidence; content-bearing aggregates (Generator[bus,carrier], Load_t.p_set, StorageUnit totals) remain unwaived as guards. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Spec: implementation-outcomes addendum (prong 1 green, benchmarks) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Prong 2 GREEN: DL-9 (existing-RE discretization) + prong-scoped waivers Investigation closed the MW accounting exactly: the +15%/+8% existing onwind/solar delta is the shared silent-drop of plants in profile-less groups (upstream issue #16) interacting with by-design different cluster geometries; candidate strictly closer to ground truth. Waivers now match on prong so prong-1's objective gate stays live. Both prongs PASS; pytest -m equivalence: 2 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Report: explicit white background / dark text (dark-mode viewers rendered dark-on-dark) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Harness: clusters 4m -> 4 (user choice: aggregate all carriers to zones) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Report: p_nom-by-attribute plots + harness-only benchmark summary table New section compares generator p_nom grouped by carrier / reeds_zone / build-year decade for two pairings: each side's add_electricity output and the simplify-stage networks. Benchmark table now filters to this harness's rules only (historic runs polluted the shared cluster_network benchmark path), shows wall time + max RSS per rule for both sides, and ends with a TOTAL row (wall sum, RSS peak) with cross-side caveats. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Revamp equivalence report into one explanatory artifact Grilling-session design: executive summary (verdict banner, provenance fingerprint, delta index from the ledger, stage timeline strip), unified diff-DAG built from both sides' rulegraphs with per-rule walltime annotations, stage-ordered narrative spine with born/inherited/masked delta-class tracking, 3-panel maps (V1-epic | anchor | diff) reusing plot_network_maps choropleths at the assembled + solved stages (identical quantities collapse to a sentence; the only capacity map rendered is oil - the 11.3 MW DL-5 Catalina placement), and a rule-grouped benchmark table with instance subrows and cross-side caveats. candidate -> V1-epic in all user-facing text. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Docs overhaul: approved design spec (grilling decisions D1-D14) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Docs P0: fix config-reference rendering, complete configtables, add guard tests The literalinclude marker convention silently mis-scoped most config sections (the 'costs' section rendered the imports block; the real costs: block never rendered). Every documented key now carries a '# docs : NAME' marker and every include uses start-after/end-before against markers; tests/docs asserts each rendered slice contains exactly its own key and that workflow/config stays in sync with repo_data/config. Configtables corrected against live config and scripts (real carrier names, StorageUnit/Link keys, dead PyPSA-Eur keys removed, EGS/policy/temporal rows added, offwind.csv created). config-wildcards rewritten for the post-refactor DAG (cluster_simpl, no {cutout} wildcard, full m/a/c suffixes). CCTS SVGs embedded as figures (prose untouched). Also: renewable.dataset + renewable_weather_years defaults in config.common.yaml and a defensive renewable_scenarios read unblock tutorial-config DAG parsing; dead simplify_network walltime keys replaced. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Docs P0: correct getting-started and power-sector data pages about-usage: real solved-network path (s{simpl}, c-prefix, {sector}, run-name dir) with a filename decoder; --configfile added to the troubleshooting command (Snakefile's default configfile is commented out); output-locations section. about-install: Python >=3.11,<3.12 stated, uv sync step, load-bearing pins, solver alignment. README: minimal quickstart. data-generators: godeeep is the default dataset, profiles are built at {simpl} resolution (removed the pre-refactor 41,564-zone story), heading-level fix. data-demand: eer profile source, corrected horizons, new demand-disaggregation section wiring the orphaned pop_layout figures. data-transmission: kmeans/modularity only. data-policies: provenance-first rewrite linking to model-constraints. config-spatial: simplify-early mechanism, verified topological_boundaries set (county|reeds_zone|state — balancing_area raises). renewables datatable: GODEEEP + NREL reV rows; orphaned sector_natural_gas.csv removed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Docs P1/P2: Model Description section, constraints reference, regenerated DAGs, generated figures New toctree section: model-workflow (stage-by-stage DAG narrative with rule/script/output tables and the simplify-early rationale), model-components (PyPSA component usage, spatial and temporal structure), model-constraints (every power-sector extra_functionality constraint with trigger, LaTeX formulation, and source link; core LP formulation delegated to upstream PyPSA), model-network-schema (publishes docs/network-schema.md via include wrapper), release-notes (user-facing v1 summary). about-introduction rewritten: model description up front, local DAG figure instead of the master-branch hotlink, real repository tree replacing the cookiecutter placeholder. Both DAG images regenerated from the current rule graph (dag.svg for docs, dag_sector.jpg via sector-config rulegraph). New docs_figures rule + plot_docs_figures.py renders network-aggregation and example-output figures from real workflow artifacts into _static/generated/. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Docs P1/P2: PDF-safe figures on sector pages, image diet, build hygiene Mechanical pass on the frozen sector pages: all 37 raw-HTML <img> figures converted to {figure} directives (raw HTML never reached the LaTeX writer, so the PDF build readthedocs requests was figure-less); every cross-reference label preserved; 74 single-word typos fixed; prose and structure untouched. Oversized images downsampled (worst was 4.8 MB). Build config: docs requirements pruned to actual docs deps (autodoc has been disabled for a long time, so pypsa/atlite/cartopy/dask/snakemake were dead weight), sphinx-copybutton added, vestigial conf.py extensions removed, readthedocs installs graphviz + imagemagick for diagram/SVG handling. contributing.md documents the config marker convention and the generated-figure rules. Sphinx build is warning-clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: correct factual errors found in review of #790 - data-demand: describe both `electricity: demand: bus_allocation:` modes (`population`, the default since #777, and `breakthrough`) and the `load_weight` column they populate, instead of claiming the weight is always Breakthrough `Pd`. - model-workflow: `add_sectors` runs `add_sectors.py` (not `build_sector.py`), and `export_statistics` runs `plot_statistics.py` in `export` mode (there is no `export_statistics.py`). - model-components: `model_topology.topological_boundaries` accepts `county`, `reeds_zone`, and `state`; `balancing_area` raises in `aggregate_to_substations`. - about-introduction: 2030/2040/2050 are the EFS horizons; EER covers 2021, 2025, 2030, 2035, 2040, 2045, 2050 (matches data-demand and `ReadEer.MODEL_YEARS`). - data-policies / release-notes: after #791 `workflow/config/` is untracked; the policy CSVs are tracked in `workflow/repo_data/config/policy_constraints/` and copied by `init_pypsa_usa.sh`. - about-usage: the default config uses 33 clusters, not 30. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: drop balancing_area from the topological_boundaries option set Three remaining occurrences of the same verified-false claim. Only 'county', 'reeds_zone' and 'state' are accepted: aggregate_to_substations.py:139-143 raises ValueError on anything else, and cluster_network.py's match has no balancing_area branch. - about-introduction.md:54 (clustering boundary list) - model-workflow.md:73 (cluster_resources stage description) - config.default.yaml:73 (inline comment, rendered into the docs by the MODEL_TOPOLOGY literalinclude — comment only, the file still parses identically) Other 'balancing' references are left alone: build_shapes really does emit balancing-authority shapes, buses really do carry a balancing_area attribute, and demand really is disaggregated from BA level. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

ktehranchi merged commit f629c8d into PyPSA:develop Aug 28, 2026
1 check passed

ktehranchi deleted the fix/untrack-workflow-config branch August 28, 2026 21:05

ktehranchi mentioned this pull request Aug 28, 2026

Restore renewable keys dropped by #791 (fixes fresh-clone parse crash) #793

Merged

5 tasks

ktehranchi mentioned this pull request Aug 29, 2026

Config reorganization: auto-loaded base config, tracked layers, schema validation #794

Merged

6 tasks

Navigation Menu

Uh oh!

Untrack workflow/config/config.common.yaml (unbreaks init_pypsa_usa.sh) - #791

Untrack workflow/config/config.common.yaml (unbreaks init_pypsa_usa.sh)#791
ktehranchi merged 1 commit into
PyPSA:developfrom
ktehranchi:fix/untrack-workflow-config

ktehranchi commented Aug 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

ktehranchi commented Aug 28, 2026

Changes proposed in this Pull Request

Why it is wrong to track it

What tracking it breaks

Second problem this fixes

Relationship to #790

Checklist

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant