PyPSA v1 migration: pypsa 1.3.0 / linopy 0.9.1 on a pandas 3 stack (redo of #762) - #778
Merged
ktehranchi merged 98 commits intoAug 28, 2026
Conversation
The LEFT JOINs in load_eia_operable_data and load_heat_rates_data joined the multi-year EIA-860 SCD tables (yearly_generators, scd_plants, scd_generators_energy_storage) without any date constraint. Each generator carries ~24 years of annual snapshots, so the intermediate join produced ~7,000 rows per generator (~200M rows for load_heat_rates_data) before the final aggregation collapsed them. Pre-aggregating each SCD table to one row per plant/generator using the same array_agg(... ORDER BY report_date DESC) FILTER (WHERE ... IS NOT NULL)[1] "latest non-null" pattern already used elsewhere in the file keeps the projection identical while reducing the join cardinality from ~24x24 to 1x1. Measured against s3://pudl.catalyst.coop/v2025.5.0: - peak RSS: 17.3 GB -> 2.2 GB - runtime: 503 s -> 24 s Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
These PNGs are referenced in the PR body to visualize the original-vs- refactored marginal_cost and heat_rate distributions. Safe to remove once the PR is reviewed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
build_powerplants: pre-aggregate SCD tables to fix memory blowup
Phase 1 of the simplify-early refactor. Splits the existing simplify_network
rule into two rules with no behavior change for the default kmeans algorithm:
- aggregate_to_substations: pure-topology aggregation to substations
(convert_to_voltage_level, remove_transformers, busmap-by-sub_id).
Writes resources/{interconnect}/elec_b.nc + busmap_b.csv.
- cluster_simpl: optional k-means/modularity reduction to {simpl} clusters.
Reads elec_b.nc, writes elec_s{simpl}.nc + simpl regions, matching the
former simplify_network output interface so downstream rules
(cluster_network etc.) are untouched.
HAC clustering is dropped from both rules (cluster_network now raises if
algorithm='hac') along with the unused 'to_substations' and 'feature'
config knobs. simplify_network.py is removed; its functions move into
aggregate_to_substations.py.
Snakemake dry-run on the default Western config produces the expected
chain: aggregate_to_substations -> cluster_simpl -> cluster_network.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Dead-code cleanup audit against v1-epic. No behavior change.
YAML config keys removed (no code reference):
- config.plotting.yaml: costs_max, costs_threshold, energy_max, energy_min,
energy_threshold (with matching docs/source/configtables/plotting.csv rows)
- config.tutorial.yaml: sector.natural_gas.allow_imports_exports
- config.common.yaml: renewable.hydro.{PHS_max_hours, resource.hydrobasins,
resource.flowspeed, hydro_max_hours, clip_min_inflow, normalization,
multiplier}; atlite.default_cutout; electricity.prm regional block
- config.default.yaml: model_topology.interface_transmission_limits;
solving.mem
Snakemake params: declarations removed (declared but never read by
the target script):
- build_bus_regions: focus_weights
- add_extra_components: ucap (script reads via snakemake.config instead)
- prepare_network: adjustments, co2base
- add_sectors: electricity, costs, plotting
- plot_network_maps: plotting, retirement
- plot_statistics: plotting
- plot_natural_gas: plotting
- solve_network: planning_horizons, transmission_network, sector_config
- solve_network_validation: planning_horizons, co2_sequestration_potential
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 1: split simplify_network into aggregate_to_substations + cluster_simpl
Outputs under resources/ previously sat at a single interconnect-rooted
level (e.g. resources/texas/elec_b.nc, resources/texas/busmap_s50.csv,
resources/texas/profile_solar_s50.nc). Files of unrelated types were
intermingled, making the tree hard to navigate.
This change introduces a category-first layout: the top level under
resources/ is what the file IS (networks/, busmaps/, profiles/, ...) and
{interconnect} becomes the next level down.
resources/
networks/{interconnect}/elec_*.nc
busmaps/{interconnect}/busmap_*.csv, bus2sub.csv, sub.csv
profiles/{interconnect}/profile_*.nc, nrel_mapping_cache
geospatial/{interconnect}/*.geojson, bus_gis.csv, lines_gis.csv
costs/costs_{year}.csv, sector_costs_{year}.csv
prices/{interconnect}/{state,ba}_*_prices.csv, pudl_fuel_costs.csv
demand/{interconnect}/{end_use}_*.csv|pkl
population/{interconnect}/pop_layout_*.nc|csv
temperature/{interconnect}/temp_{soil,air}_*.nc
heating_cop/{interconnect}/cop_{soil,air}_*.nc
co2/{interconnect}/co2_storage_*.csv
powerplants/powerplants.csv
Implementation:
- workflow/Snakefile defines twelve category constants (NETWORKS,
BUSMAPS, PROFILES, GEOSPATIAL, COSTS, PRICES, POWERPLANTS, DEMAND,
HEATING_COP, TEMPERATURE, POPULATION, CO2) composed from RESOURCES, so
the RDIR (run name) and shared_resources prefixes still work.
- Every RESOURCES + "{interconnect}/<file>" reference in the .smk files
swapped to the matching category constant. No filenames change.
- "Geospatial/" (capital G) renamed to "geospatial/" (lowercase) for
consistency with the new top-level subfolders.
- "resources/powerplants.csv" hardcoded literal moved to
"resources/powerplants/powerplants.csv" (still shared across runs).
- Two off-rule literal paths updated: a __main__ test path in
build_natural_gas.py and argparse defaults in plot_caps_summary.py.
- Doc reference in about-usage.md sharpened to resources/networks/.
Verified by snakemake -n: full DAG resolves end-to-end for
resources/Default/networks/texas/elec_base_network_l_pp.pkl, with each
intermediate input/output routed to the new category folder.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reorganize resources/ into category-first subfolders
…nd-params Remove unused config keys and unused snakemake rule params
Brings in Phase 2 (move simplify/cluster_simpl ahead of demand+RE) and Phase 3 (aggregate EGS supply curves through cluster_simpl busmap). Both PRs were originally merged into their stacked-base branches rather than v1-epic; this merge propagates their changes up. Resolved conflicts in build_electricity.smk by taking the post-#12 resources/ layout (category-first constants: NETWORKS, GEOSPATIAL, PROFILES, DEMAND, BUSMAPS) and applying Phase 2's path repointing (network refs to elec_s{simpl}.nc, _s{simpl} suffix on demand/profile outputs). Repointed aggregate_egs outputs and add_electricity EGS inputs to PROFILES/BUSMAPS per the new layout.
Two-part approach: a curated docs/network-schema.md catalog of custom columns plus a _helpers.log_network_schema helper that logs per-script column entry/exit diffs. Motivated by recurring consense aggregation crashes (most recently LAF_state in aggregate_to_substations). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Six tasks: add helper + tests, wire into topology / add-* / sectors chains, smoke test on test_small western, seed catalog from logs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Logs component row count and column set on entry; on exit emits row and column diffs vs. the entry snapshot. Logging only — no asserts. Wires into scripts in follow-up tasks; tested in isolation here. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds entry/exit schema logging to build_base_network, aggregate_to_substations, cluster_simpl, cluster_network. Logging only; no behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… add_demand Logging only; no behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…twork Logging only; no behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tiered test pyramid (static <30s + integration <5min) targeting the classes of breakage that the simplify-early refactor (PRs #7-#12) made visible: path/wiring drift, dead config keys, silent artifact-shape regressions. Schema-catalog assertions deferred to a follow-up PR after the schema-tracking spec lands. Brainstorming output. Five-PR migration plan included.
Initial catalog of custom columns on PyPSA components, populated from [schema ...] log output of build_base_network and aggregate_to_substations entry on the small_mh western config. Sector- and electricity-stage columns will be added as those rules get exercised under the schema logger. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop editorial framing ("the bug class this catalog was created to
address"). Keep the factual note that LAF_state is missing from
bus_strategies and the fix is tracked separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five-PR migration: scaffolding → Tier A static checks → Tier B fixture → Tier B full assertions → CI wiring. Plan includes exact code, file paths, verification commands per step. PR 6 (schema assertions) deferred pending the schema-tracking initiative.
Adds a repo-root conftest.py that auto-marks every test collected from workflow/scripts/test/ with the 'fast' marker, so pytest -m fast picks them up without per-file changes. Also skips 9 pre-existing test failures unrelated to this PR (RPS constraint helper signature drift and ERM API drift) so pytest -m fast exits 0. Each skip carries a reason pointing back to v1-epic.
Approved-design doc for a new simpl="county" wildcard value that bypasses k-means and uses the substation network's county FIPS (prefixed with reeds_zone) as a direct busmap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four-task TDD plan: extract resolve_simpl_mode and build_county_busmap helpers, wire them into __main__ dispatch, and document the new simpl="county" wildcard value. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
All PRs now target develop (never master), so fast-tests/e2e-tests must run there once v1-epic merges. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds the ADOPTED-FIX anchor-patch category to the equivalence harness (apply_adopted_fix_patches in tests/equivalence/build.py): mirrors the DL-11 empty-county-sweep fix onto the anchor worktree, marker-idempotent, plus a one-shot .eq-force-rerun marker that build_side turns into -R build_bus_regions (--rerun-triggers mtime neither reruns on code changes nor revisits missing intermediates when the final target looks current). Ledger gains the countersigned DL-11 row (with the known plants_must_add seam residual and forced-rerun mechanics); CHANGELOG and spec D10 document the two anchor-patch categories. Post-adoption CA prong-1 rerun: data stages clean (80/82 findings waived, matching baseline); the 2 live solved-stage findings trace to a pre-existing build_powerplants.py source-data divergence (diagnosed 2026-08-23, previously absorbed by DL-7's waiver), addressed separately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Step two of the v1 landing: the API port was proven on 1.2.4/pandas 2.2.2
(previous commit), now the pins move to pypsa==1.3.0 which requires
pandas>=3.0. Pin moves: pandas 2.2.2 -> 3.0.5; xarray 2024.9.0 -> 2026.7.0
(floor forced by pandas 3); geopandas 1.0.1 -> 1.1.4 (pandas-3 line).
Deliberately held: numpy 1.26.0 (rasterio/atlite numpy-1.x ABI),
dask/distributed 2024.12.0 (import-verified under pandas 3), linopy 0.9.1.
xarray 2026 fallout:
- StorageUnit RESERVES energy balance rewritten fully in model space,
mirroring pypsa 1.3's internal define_storage_unit_constraints
(n.optimize._window + c.da accessors). The old pypsa-0.30-era copy mixed
pandas-converted DataArrays with model coords and raised AlignmentError
under xarray 2026's strict alignment.
- ERM nodal-balance RHS columns axis pinned to "Bus": region_buses.index is
named "name" under v1, which broadcast the constraint over a spurious dim
and broke dual extraction in store_ERM_duals.
pandas 3 sweep (audited class-by-class; both suites green):
- groupby(axis=1) removed -> transpose-group-transpose (_helpers
aggregate_p_nom, plot_network_maps emissions map).
- Offset aliases: 'H' -> 'h' (add_sectors), deprecated 'd' -> 'D'
(plot_statistics, plot_statistics_sector, summary), and the config
time_resolution knob (documented as "int H") lowercased before resampling
in prepare_network ('4H' is a hard error in pandas 3).
- read_csv(squeeze=) removed -> .squeeze("columns") (cluster_network).
- geopandas: .unary_union -> .union_all() (build_renewable_profiles).
- pypsa.options.api.legacy_string_dtype = True pinned wherever networks are
loaded (_helpers, unit-test conftest, tests/integration conftest,
equivalence harness load_network) so component frames stay on object
dtype until the deliberate pypsa-2.0-era flip.
- Audited clean: chained-assignment/CoW hazards (63 candidate sites),
str-dtype checks (0 sites), 20+ removed pandas API families, all freq
literals + dynamic freq paths + 14 YAML configs.
Tests unchanged on the final stack: unit 45 passed / 1 skipped, static 72
passed. Tier-C equivalence not rerun (env moved twice; re-baseline needed).
CLAUDE.md env section, docs/pypsa-v1-migration.md, and the changelog entry
updated for the 1.3.0 landing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ktehranchi
changed the title
Second ADOPTED-FIX anchor patch (apply_powerplants_adoption): dynamic whole-file adoption of the candidate's build_powerplants.py onto the anchor, sentinel-gated (ges_latest CTE), interface-checked against the pristine e7f8bd7 file, idempotent by content comparison; .eq-force-rerun writing is now merge-safe (mark_force_rerun). Root cause of the 2.34% prong-1 solve divergence: upstream joins EIA-860 tables raw so ~24 years of report_date vintages reweight its means (Watson Cogen at an impossible 62% efficiency); v1-epic pre-aggregates. PUDL release and all tracked inputs identical — the query was the sole divergence. Harness after adoption (both sides rebuilt, candidate's stale powerplants.csv regenerated too): prong 1 PASS 0 live/72, objective rel 2.34e-2 -> 2.1e-6, per-carrier p_nom_opt equal to 0.01 MW; prong 2 PASS 0 live/3 (all DL-9-class, recalibrated: solar gap unchanged 3,586.6 MW, onwind gap 3,680.1 MW after DL-11 decontamination). DL-7 re-scoped: efficiency + marginal_cost members were DL-12 source data (waivers deleted as dead); only the non-composable unweighted fuel_cost mean survives, metadata-only. Countersigned (ktehranchi, 2026-08-23). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
filter_plants_by_region filters plants by sjoin against regions_onshore / regions_offshore. Since DL-11 (88bede4) those layers tile only the model footprint in scoped runs — a model_topology.include: {reeds_state: [CA]} run tiles 409.8k km2 instead of 2.93M km2. The plants_must_add fallback, however, stayed unconditional: it selects plants lying outside every ReEDS shape of the run's interconnect whose ReEDS-membership interconnect disagrees with their EIA `interconnection`, and concatenates them into plants_filt with no test against the (now footprint-sized) regions. match_plant_to_bus's second pass then assigns any plant without a zone match to the nearest network bus with no distance bound. Measured on the CA equivalence run (western, 2030, simpl=''), that fallback leaks 23 plants / 1,887.4 MW onto California buses — NM 1,112.1 MW of wind and solar, MT 370.3 MW (incl. Fort Peck hydro 162.4 MW across 4 units), Buffalo Ridge II SD 210.0 MW, and Hardy Hills Solar IN 195.0 MW from 2,508 km away. Every entry is at least 890 km from the CA footprint, so 50/100/200 km all drop the whole population. Fix: when the run is footprint-scoped, keep only must-add plants within SEAM_PLANT_MAX_KM = 100 km of the union of regions_onshore + regions_offshore, measured in EPSG:5070. Plants inside the footprint have distance 0 and are always kept, so genuine near-seam plants still attach — match_plant_to_bus is deliberately left alone, since its unbounded second pass is correct once the leak population is filtered upstream. Gated on model_topology.include being truthy (read from snakemake.config in main(), threaded as the filter_plants_by_region(footprint_scoped=...) parameter). With the gate off not a single statement changes, so unfiltered interconnect/usa runs are byte-identical BY CONSTRUCTION. That gate is load-bearing, not cosmetic: against a full-western footprint the same population is mostly legitimate, and an unconditional 100 km bound would delete 8 plants / 694.9 MW (incl. Hardy Hills and Buffalo Ridge II) from an unfiltered western run. Every dropped plant is logged at WARNING with name, carrier, state, MW and distance, plus a summary line with count and total MW. New unit tests in workflow/scripts/test/test_seam_plants.py build synthetic regions/ReEDS shapes/membership so an in-footprint plant, a ~50 km seam plant and a ~500 km far plant all land in plants_must_add, then pin gate-off keeping all three (legacy behavior) and gate-on keeping the first two while dropping and loudly logging the third. workflow/scripts/test/: 41 passed, 10 skipped (pre-existing skips), no regressions. Full quantification, method and caveats: docs/superpowers/specs/2026-08-23-seam-plant-quantification.md — including three findings adjacent to this fix and deliberately NOT changed here: the primary sjoin leaks zero out-of-state plants; the plants_nearshore sjoin_nearest path compares EPSG:4326 degrees against EPSG:3857 metres and so matches nothing (11 SD plants / 230.0 MW silently dropped today); and 46 plants / 3,326.3 MW (CA 41, WA 5) fall outside every national ReEDS shape before the fallback runs. Results-changing for scoped runs; pending ledger countersignature. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mirrors v1-epic d98cb93 onto the pinned anchor e7f8bd7 as the third ADOPTED-FIX patch, so the harness keeps comparing like-for-like instead of scoring the candidate against a known-leaking anchor. apply_seam_adoption does targeted string surgery rather than DL-12's whole-file adoption: v1-epic's add_electricity.py legitimately differs from the anchor's in the simplify-early bus2sub/sub_id removals, the length_factor=1.0 decision (DL-1/DL-2) and the schema-logging calls, so copying it wholesale would smuggle those unrelated deltas onto the anchor. The whole filter_plants_by_region body is byte-identical between e7f8bd7 and v1-epic, so the anchor takes the same footprint_scoped parameter plumbing as the candidate rather than an inlined-config variant. Verified AST-identical across both sides: the helper body (docstrings stripped), the SEAM_PLANT_MAX_KM value, the filter_plants_by_region signature, the gated call block and the main() wiring. The constant block and the helper body are sliced out of the LIVE candidate file so the numeric logic both sides run is the same text and any drift in v1-epic's helper re-triggers the forced rerun; only the four wiring edits, which must adapt to the anchor's own shape, are hardcoded. Safety rails match apply_powerplants_adoption: the candidate must carry the sentinel and yield both slices, all four needles are verified to occur exactly once against the PRISTINE anchor file fetched from git (not the possibly already-patched worktree), the pristine anchor must not already contain the sentinel, and the result is checked for end-to-end footprint_scoped wiring before anything is written. Idempotent on the sentinel; marks add_electricity for the one-shot forced rerun on newly-applied. Also wires EQ_UNTIL=assembled into run.py's build targets. paths.py has stopped the COMPARED pairs at the assembled stage since the harness was written, and exported assembled_target/anchor_assembled_target for the matching build targets, but run.py never used them — so an EQ_UNTIL=assembled run still drove the whole chain through a solve it then did not compare. At usa scope that is a national solve. Inert for the CA runs, which do not set the variable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rint
The equivalence harness's CA prong 2 (simpl=20) died in add_electricity with
GEOSException: TopologyException: side location conflict, raised by
_drop_distant_seam_plants' pd.concat(region_geoms).union_all().
Root cause is the reprojection, not the source data: regions_onshore_s20 and
regions_offshore_s20 are 100% valid as stored in EPSG:4326, but converting them
to EPSG:5070 for the distance measurement leaves 9 of 29 polygons
self-intersecting or degenerate ("Too few points in geometry component"), and
GEOS union_all refuses invalid input. Prong 1 never tripped it because the
simpl='' layer reprojects to 2,014 fine-grained polygons that all stay valid --
so the bug was latent behind the granularity of the first prong tested.
The union was only ever a means to a distance, and dist(p, union(R_i)) equals
min_i dist(p, R_i), so take the minimum over regions directly and skip the
union. Pairwise distance is robust to self-intersection where the union is not.
Verified equal, not merely similar: on the simpl='' layer where union_all does
succeed, the two methods agree to 0.000000000 m over 60 probe points; on the
simpl=20 layer the union raises and the per-region path returns clean
distances. Cost is 0.215 s at prong-1 scale (23 plants x 2,014 regions).
Also drops empty/NA geometries before measuring, and returns early if nothing
survives.
New regression test builds a self-intersecting bowtie plus an overlapping box --
the minimal shape that reproduces the GEOS failure -- placed far from every test
plant so the expected keep/drop set is unchanged and only the distance path is
under test. Confirmed to fail with the original union_all implementation
(identical TopologyException) and pass with this one. workflow/scripts/test/:
42 passed, 10 skipped.
Harness side: apply_seam_adoption's idempotence is now by CONTENT rather than by
the sentinel string, matching apply_powerplants_adoption. Sentinel-based
skipping would have left the anchor running the pre-fix helper forever, since
the sentinel is present either way; content comparison re-applies the patch and
re-arms the one-shot forced-rerun marker whenever the candidate's helper
changes. Verified: the anchor picked up this fix automatically on the next
provision, and both sides remain AST-identical in helper body, gated block,
signature and main() wiring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Records the countersigned seam-plant bound adoption (d98cb93 + 103f219 + anchor mirror 85cda59) with the corrected symmetric magnitude (1,725.0 MW leaves the assembled network on both sides, not the 27 MW name-count estimate), the harness verdicts (prong 1 PASS 0 live, objective rel 2.46e-06; prong 2 PASS with DL-9 absolute gaps exactly unchanged), and the GEOS per-region-distance robustness fix. Also gitignores the transient .eq-force-rerun marker. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fuel_cost residual is 8.1277 $/MWh (40.24%) on current artifacts (8.4498 when first measured post-DL-12); adds the three-fix invariance of the DL-9 absolute gaps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
usa data-stage harness PASS 0 live/113 - identical count and class structure to the pre-adoption baseline, confirming DL-11/12/13 left the national comparison untouched. Seam gate verified on all axes (source AST, config eval, 0 seam-drop lines in both sides' usa logs vs 24 at CA). usa report regenerated, superseding the stale 2026-08-22 artifacts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… MECS path pandas 3 raised its optional-dependency floors, which only error at use time (import_optional_dependency), so tests missed them: openpyxl 3.1.2 -> 3.1.5 (caught via build_demand's MECS read_excel path), matplotlib 3.8.0 -> 3.9.3, scipy 1.11.3 -> 1.14.1. All 36 floors in pandas.compat._optional.VERSIONS now audited clean against the lock. pyproject, environment.yaml and uv.lock updated in sync. build_demand.py: replace the vacuous `assert not (mecs == np.NaN)` (NaN never compares equal, so it always passed; np.NaN is also removed in numpy 2) with a real `mecs.isna()` check. Verified passing against the actual MECS workbook (table3_2.xlsx, 405x9, zero NaNs) -- safe by construction too, since dropna/replace run before astype(float). No other capitalized np.NaN usages repo-wide. Results effect: none. Unit tier 45 passed / 1 skipped, static tier 72 passed on the bumped matplotlib/scipy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ness
pandas 3 preserves missing values through .astype(str) from every source
dtype (object, float64, str, Int64, datetime64, category all verified)
instead of stringifying them to "nan" as pandas 2 did. Two load-bearing
sites relied on the old behavior:
- build_powerplants merge_ads_data: ADS "Long Name" match keys (343 NaN
in GeneratorList.csv) crashed the re.sub name normalization (loud).
- build_powerplants set_parameters: build_decade = build_year.astype(str)
feeds impute_missing_plant_data's groupby + inner merge as a key. Under
pandas 3 the 923 plants with no generator_operating_date (767 proposed
units / 74.7 GW: 411 solar, 173 batteries, 47 onwind, 19 CCGT) get a
NaN key, match no group, and are silently dropped from powerplants.csv;
under pandas 2 they landed in the "nan0s" bucket and survived to have
build_year backfilled from current_planned_generator_operating_date in
add_electricity (silent, results-changing).
Both pinned byte-identical to pandas-2 semantics with .fillna("nan").
Found during the Tier-C equivalence re-baseline (first full pipeline run
under the new stack) plus an 82-site audit of every pandas astype(str)/
map(str) call; all other sites verified SAFE empirically, two latent
config-gated sites flagged in the changelog.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The previous commit accidentally tracked three machine-local symlinks (workflow/data, workflow/cutouts, .worktrees/anchor-e7f8bd70) created to run the equivalence harness from this worktree; remove them from the index and gitignore them. Add the changelog entry for the two pandas-3 astype(str) NaN regressions (ADS Long Name crash; 74.7 GW silent proposed-generator drop via the build_decade imputation key). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…the re-baseline Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…gration-data-storage-26840f # Conflicts: # .gitignore # docs/CHANGELOG-v1-epic.md
…das 3 Clean from-scratch candidate rebuild of both CA prongs against the anchor (develop e7f8bd7, pypsa 0.30 env). Pre-adjudication: prong 1 16 live / 88 findings, prong 2 0 live / 3. All 16 live findings verified as zero-physics artifacts of the 0.30 -> 1.3 conventions, in 3 classes: - sub_network topology metadata v1 serializes and 0.30 left empty (11 findings; Bus/Line values + SubNetwork rows) -> waived (DL-15). - StorageUnit.cyclic_state_of_charge_per_period: the attribute is not stored in the anchor netCDFs at all (0.30 omitted default-valued attrs); the pypsa-1.3 reader backfills the NEW default False while the anchor's solve-time behavior was True, and the candidate pins True explicitly. Comparison manufactures the diff on read (4 findings) -> waived (DL-15) with the file-level proof in the justifications. - Network.objective: candidate folds objective_constant in (linopy 0.9), anchor split it out. Like-for-like (objective + objective_constant) agrees to rel 5.3e-07 -> fixed as a comparator NORMALIZATION in compare.py (_total_objective), not a waiver; the objective gate stays live and correctly measured. Prong-2's DL-9 objective waiver (a real physics delta) is unaffected. Post-adjudication: prong 1 PASS 0 live / 87 (one fewer total because the objective finding is normalized away, not waived); prong 2 PASS 0 live / 3. DL-15 drafted in the deltas ledger, PENDING COUNTERSIGNATURE (DL-14 is reserved by proto/nearshore-crs-fix). Known follow-up: extend the three classes' verification + waivers to the usa leg when it is rebuilt on the 1.3 stack. Changelog updated; its stale "harness not re-run" clause corrected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # .gitignore # CLAUDE.md # docs/CHANGELOG-v1-epic.md # docs/superpowers/specs/2026-08-07-deltas-ledger.md # tests/equivalence/compare.py # tests/equivalence/waivers.yaml # tests/integration/conftest.py # workflow/scripts/_helpers.py # workflow/scripts/aggregate_to_substations.py # workflow/scripts/cluster_network.py # workflow/scripts/cluster_simpl.py
ktehranchi
force-pushed
the
claude/pypsa-v1-migration-data-storage-26840f
branch
from
August 28, 2026 21:12
00768f8 to
62563bc
Compare
5 tasks
ktehranchi
added a commit
that referenced
this pull request
Aug 28, 2026
Two defects landed on develop with #777 and would make the next Tier-C harness run return an untrustworthy verdict. B1 - the usa leg compared two different demand pipelines. config.equivalence.yaml:77 pins `bus_allocation: breakthrough` so the western leg stays apples-to-apples with the anchor, which has no census-population method. config.equivalence-usa.yaml carried no such pin, and build.py copies the candidate's config into the anchor worktree, so BOTH sides ran unpinned at the new `population` default. Candidate demand was weighted by 2020 census county population, the anchor by Breakthrough Pd - diverging Load_t.p_set, Bus.LAF_state, the kmeans geometry (cluster_network.py:68,82 weights on load_weight) and the objective. Every resulting finding would read as a migration regression. DL-13's recorded usa leg stopped reproducing. B2 - prong 1 reported pass: false for a non-migration reason. load_weight is candidate-only and survives every compared stage, so compare.py:174-184 raised an unwaived column_set finding at five stages and compare.py:546-549 flipped the run to failing. The pin is the substantive fix; the waiver is sound only because of it. With breakthrough pinned the column carries the anchor's own legacy Pd weighting, so its presence is numerically inert. Demand content stays guarded by the UNWAIVED Load / Load_t.p_set comparisons, and load_weight VALUE findings are deliberately not waived - the waiver matches kind: column_set only, verified against compare.py's is_waived. Not yet exercised against a live harness run; the Tier-C re-baseline #778 requires is still outstanding.
ktehranchi
added a commit
to ktehranchi/pypsa-usa
that referenced
this pull request
Aug 28, 2026
Refresh the docs overhaul onto develop after the pypsa 1.3.0 / linopy 0.9.1 / pandas 3.0.5 migration (PyPSA#778), the population-based demand allocation (PyPSA#777), and the untracking of workflow/config/config.common.yaml (PyPSA#791). Conflicts: - docs/source/configtables/electricity.csv: kept the overhaul's restructured table and re-added develop's electricity.demand.bus_allocation row under the branch's '-- ' nesting convention. - workflow/config/config.common.yaml: resolved to develop's deletion; the file is user-generated and now gitignored, so the overhaul's edit is dropped (workflow/repo_data/config/config.common.yaml remains canonical). Also corrected docs/source/about-install.md, which still advertised the pre-migration pins and claimed PyPSA 1.x was unsupported.
ktehranchi
added a commit
to ktehranchi/pypsa-usa
that referenced
this pull request
Aug 28, 2026
Refreshes the cleanliness sweep onto develop after the pypsa 1.3.0 / linopy 0.9.1 / pandas 3 migration (PyPSA#778), the config.common.yaml untrack (PyPSA#791) and the equivalence-harness recalibration (PyPSA#792). Conflicts, all in favour of develop's migrated code: - _helpers.py: develop deleted load_network (pypsa.descriptors.Dict and override_components are gone in v1); the sweep's own deletions of the unreferenced pdbcast / aggregate_* / get_aggregation_strategies / load_network_for_plots / setup_custom_logger helpers still apply, so the file is a pure deletion relative to develop. - add_extra_components.py: attach_stores stays deleted (still dead on develop -- its only call site is a comment); the sweep's copy_timeseries_for_suffix helper is kept. - summary.py: keeps develop's migrated imports (pypsa.statistics.get_bus_and_carrier no longer exists) and drops the unreferenced _helpers.configure_logging import along with the __main__ block that used it; get_energy_total / get_demand_base / get_capacity_base / get_capacity_brownfield / get_capital_costs stay deleted. _iter_components remains in use by get_energy_timeseries. No pre-1.3 API is reintroduced and no cleanup hunk was applied on top of code the migration rewrote. Dependency files are untouched and workflow/config/ holds only .gitkeep.
6 tasks
ktehranchi
added a commit
that referenced
this pull request
Aug 31, 2026
) The v1 migration (#778) bumped pandas to 3.0.5 and geopandas to 1.1.4 in workflow/envs/environment.yaml but left the packages around them at their 2023 values. The file no longer solves on conda-forge, so micromamba dies at env-creation time -- this is what killed the e2e-tests job in CI run 33279801900, and it breaks every mamba-based user setup too. Four hard conflicts, all reproduced locally with micromamba 2.9.0: * pandas 3.0.5 run-constrains `pytables >=3.10.1`, so `pytables==3.9.1` was unsatisfiable. Bumped to 3.10.2, matching pyproject's `tables==3.10.2`. * pandas 3.0.5 run-constrains `lxml >=5.3.0`, so `lxml==4.9.3` knocked out the only py311 pandas build; that in turn made `python==3.11.9` unsatisfiable. lxml is now `>=5.3.0` and python mirrors pyproject's `requires-python` as `>=3.11,<3.12`. * conda-forge's rasterio 1.3.8 builds pin `proj 9.2.1`/`9.3.0` exactly and `libgdal 3.7.x`, whose `pcre2 <10.44` pins are mutually exclusive with the Qt6 stack the `matplotlib` metapackage pulls in. * the `pip:` section's unbounded `tsam` spec resolved to 4.0.0, which caps `pandas<=3.0.3` and so silently uninstalled the conda-built pandas 3.0.5 after the conda solve succeeded. numpy stays at 1.26.0. conda-forge's pandas 3.0.5 declares `numpy >=1.26.0,<3`, so the numpy-1 hold that rasterio/atlite need is compatible with pandas 3 and no numpy 2 bump is required. Two pins deliberately diverge from uv.lock, documented in a comment block at the top of the file: * `rasterio==1.3.10` (uv.lock: 1.3.8) -- nearest solvable conda-forge build in the same minor series. It resolves against libgdal 3.9.1 / pyogrio 0.9.0, far closer to the uv stack than the 2023-era libgdal 3.7.3 / pyogrio 0.7.2 that pinning 1.3.8 would force (and which is only reachable at all by dropping the `matplotlib` metapackage for `matplotlib-base`). * `tsam==2.3.6` exactly, rather than pyproject's `>=2.3.6`, for the pandas-clobbering reason above. 2.3.6 is what uv.lock resolves to. Other changes bring the file back in line with pyproject: `dask` pinned to 2024.12.0 (was `>=2023.7.0`), `pulp==2.7.0` added, `highspy>=1.9.0` and `dill>=0.3.9` bounded, and the 2023-era ipython/jupyter/notebook/ ipykernel/tqdm pins relaxed since pyproject does not pin them. Verified: micromamba 2.9.0 dry-run solve succeeds for linux-64 (486 pkgs) and osx-arm64 (430 pkgs); a full non-dry-run create on osx-64 succeeds and imports pypsa, pandas, geopandas, rasterio, tables, atlite and linopy, with pandas still at the conda-built 3.0.5 after the pip step. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upstream mirror of ktehranchi#25. Stacked on #775 (base
v1-epic): until #775 merges this diff also shows the v1-epic commits beyond the anchor; afterwards it collapses to the migration alone. Independent of #776/#777. Supersedes #762.What
Migrates the whole workflow from
pypsa==0.30.2/linopy==0.3.14/pandas==2.2.2topypsa==1.3.0/linopy==0.9.1on a pandas 3 stack (pandas==3.0.5,xarray==2026.7.0,geopandas==1.1.4). This takes on #762 as part of the v1-epic: the changes were redone against the v1-epic tree (simplify-early DAG,opts/constraint modules, equivalence harness) and verified against the real pypsa v1 API rather than cherry-picked — several of #762's mechanical choices don't hold on v1 (names=kwarg is removed,addreturnsNone, thepypsa.definitions.structuresimport home is wrong for everything exceptDict, and the deprecatedget_bounds_puwrapper is broken for static-only attrs).Two-step landing: the port was written and proven first against
pypsa==1.2.4on the existingpandas==2.2.2stack — 1.2.4 is the newest v1 that runs on pandas 2.x and its API behaves identically to 1.3.0 — so the 0.30 → v1 API changes could be validated with pandas held constant. The pandas-3 stack was then bumped in the same branch (section below). The 1.2.4 / pandas 2.2.2 / xarray 2024.9.0 / geopandas 1.0.1 pin set therefore remains a viable fallback if pandas-3 problems surface downstream or on HPC; the one piece to re-check on that path is the rewritten StorageUnit RESERVES builder, which now mirrors 1.3's internaldefine_storage_unit_constraintsand leans on then.optimize._windowhelper.Full migration map + follow-up candidates:
docs/pypsa-v1-migration.md. Changelog entry indocs/CHANGELOG-v1-epic.md.Highlights beyond the renames
"name"(notGenerator/Generator-ext): fixed selectors/groupers acrossopts/{policy,reserves,land,sector,interchange}.py; the RESERVES operational-constraint builders were rewritten to mirror pypsa v1's internal implementations (xarrayget_bounds_pu).DataArray(...)before linopy arithmetic so the flatsnapshotdim survives (linopy 0.9 otherwise unstacks toperiod × timestep). The StorageUnit RESERVES energy balance no longer needs this — it was later rewritten fully in model space, see below.e_cyclic_per_period/cyclic_state_of_charge_per_perioddefaults True→False — all 15 cyclic adds now pin*_per_period=True. UC runs may see small accepted deltas from upstream ramp-limit fixes."name":bus2sub.csvkeeps its legacyBusheader (index_label="Bus") so downstream readers/artifacts are untouched; nearest-bus matching is index-positional now (fixes two latent.Bus-attribute bugs inmatch_missing_buseswhen called withn.buses-derived frames).n.components["Generator"].defaults— Update PyPSA dependency >= V1 #762'sdata/unit_commitment.csv(all zeros) was deliberately not adopted; behavior is byte-identical without a new data file. The CSV remains a good hook if we ever want per-carrier non-zero UC defaults (results-affecting, separate PR).Network.copy()drops the hiddenname="snapshot"attr of MultiIndex snapshots, breakingc.daaccessors on the copy. netCDF round-trips are unaffected (production safe); test conftest heals copies. Worth reporting to PyPSA/PyPSA.tables==3.10.2pinned explicitly (the re-lock dropped it as a transitive;build_demand/EER tests need pytables).pandas 3 / xarray 2026
Step two of the landing, in the same branch.
Pins.
pypsa1.2.4 → 1.3.0;pandas2.2.2 → 3.0.5 (required by 1.3);xarray2024.9.0 → 2026.7.0 (floor forced by pandas 3, which needsxarray>=2024.10— took the current release rather than the bare minimum);geopandas1.0.1 → 1.1.4 (the 1.1 line is the pandas-3-compatible one);linopystays 0.9.1.numpyis deliberately held at 1.26.0 — the pinnedrasterio==1.3.8/atlite==0.3.0wheels are built against the numpy 1.x ABI, pandas 3 does not force a bump, and numpy 2 is its own migration with its own binary-compat blast radius.dask/distributedheld at 2024.12.0, import-verified under pandas 3 rather than bumped speculatively. One class of breakage only surfaces at use time: pandas 3 raised its optional-dependency floors —openpyxl3.1.2 → 3.1.5 (caught via the MECSread_excelpath, which no test covers),matplotlib3.8.0 → 3.9.3,scipy1.11.3 → 1.14.1; all 36 floors inpandas.compat._optional.VERSIONSwere then audited clean against the lock. A vacuousnp.NaNequality assert inbuild_demand.pywas made a realisna()check while in there (verified passing on the actual MECS workbook; also clears a numpy-2 landmine).AlignmentError: it mixed DataArrays converted from pandas (carrying their ownperiodindex) with model-space coords (whoseperiodcomes from the linopy model), and 2026 no longer tolerates the two conflictingperiodindexes.define_SU_reserve_constraintsinopts/reserves.pyis now a direct mirror of pypsa 1.3's internaldefine_storage_unit_constraints— the samen.optimize._windowmachinery (.subset(sns), snapshot weightings,roll_within_periodsfor the previous-SOC term,period_start_maskfor the period-start split) and the samec.da.*accessors, built in xarray end to end with no pandas round-trip. Only deliberate departures from upstream: the*_RESERVESvariable names, and no spill term (the shadow reserve system has no spillage variable)."Bus". Second v1 index-rename bug in the same file. The ERM nodal-balance RHS DataFrame took its columns fromregion_buses.index, whose name is"name"under v1, so the columns axis inherited it; linopy then broadcast the constraint over a spuriousnamedim (warningConstant RHS contains dimensions {'name'}) and the dual was no longer pivotable instore_ERM_duals. Fixed withrhs.columns.name = "Bus".pypsa.options.api.legacy_string_dtype = Truepinned inworkflow/scripts/_helpers.pyand the unit-test conftest, keeping pypsa component frames onobjectdtype under pandas 3's new defaultstrdtype. Deliberate holding position, not permanent — pypsa intends to drop the switch at 2.0, so flipping it off (and fixing whatever dtype assumptions that surfaces) is the follow-up PR, best done with the equivalence harness available.DataFrame.groupby(axis=1)and relatives), retired offset aliases in date ranges and resampling ('4H' → '4h', deprecated 'd' → 'D'), chained-assignment patterns that copy-on-write turns into silent no-ops, and dtype checks written againstobject-dtype strings. Mechanical, intended behavior-preserving.Tests
Both tiers re-run on the final pypsa 1.3.0 / pandas 3.0.5 / xarray 2026.7.0 stack; counts are unchanged from the 1.2.4 step.
test_e2e_solve_network_myopic) was bisected: its fixture model is infeasible on pristine v1-epic under pypsa 0.32 too (pre-existing, now documented in the skip reason).astype(str)regressions (incl. a 74.7 GW proposed-generator drop via an imputation key) and required merging the footprint-scoped-regions arc from v1-epic (the branch predated it — the anchor carried it as adopted fixes). The 16 residual live findings were all zero-physics 0.30→1.3 convention artifacts: sub_network metadata (waived), a default-backfill read artifact oncyclic_state_of_charge_per_period(waived, with file-level proof), and the objective-constant reporting convention — normalized in the comparator, where like-for-like totals agree to rel 5.3e-07. Ledger entry DL-15 drafted, pending countersignature. Outstanding: the usa leg re-verification on the 1.3 stack; full production-scale runs.PyPSA v1 data-storage features we can now use
docs/pypsa-v1-migration.mdhas the full survey with integration points. Adopted now: typed components layer (n.components,.static/.dynamic/.da), string-list statistics groupers, andpypsa.options. Top follow-ups: custom groupers registered on our columns (reeds_state,rec_trading_zone, …),n.shapesfor bus-region GeoDataFrames inside the netCDF,n.metaas the provenance/schema channel for the equivalence harness,NetworkCollectionfor myopic per-period and scenario comparison,n.cluster.temporalto replace hand-rolled resampling, andn.set_scenariosfor weather-year work. Landing on 1.3.0 also makes the v1.3 modeling additions — piecewise-linear costs, maintenance scheduling, phase-shifters — available; none is wired in here, each being a results-affecting modeling choice of its own.🤖 Generated with Claude Code