v1-epic: simplify-early pipeline with results-equivalence guarantee - #775
Merged
Conversation
The LEFT JOINs in load_eia_operable_data and load_heat_rates_data joined the multi-year EIA-860 SCD tables (yearly_generators, scd_plants, scd_generators_energy_storage) without any date constraint. Each generator carries ~24 years of annual snapshots, so the intermediate join produced ~7,000 rows per generator (~200M rows for load_heat_rates_data) before the final aggregation collapsed them. Pre-aggregating each SCD table to one row per plant/generator using the same array_agg(... ORDER BY report_date DESC) FILTER (WHERE ... IS NOT NULL)[1] "latest non-null" pattern already used elsewhere in the file keeps the projection identical while reducing the join cardinality from ~24x24 to 1x1. Measured against s3://pudl.catalyst.coop/v2025.5.0: - peak RSS: 17.3 GB -> 2.2 GB - runtime: 503 s -> 24 s Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
These PNGs are referenced in the PR body to visualize the original-vs- refactored marginal_cost and heat_rate distributions. Safe to remove once the PR is reviewed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
build_powerplants: pre-aggregate SCD tables to fix memory blowup
Phase 1 of the simplify-early refactor. Splits the existing simplify_network
rule into two rules with no behavior change for the default kmeans algorithm:
- aggregate_to_substations: pure-topology aggregation to substations
(convert_to_voltage_level, remove_transformers, busmap-by-sub_id).
Writes resources/{interconnect}/elec_b.nc + busmap_b.csv.
- cluster_simpl: optional k-means/modularity reduction to {simpl} clusters.
Reads elec_b.nc, writes elec_s{simpl}.nc + simpl regions, matching the
former simplify_network output interface so downstream rules
(cluster_network etc.) are untouched.
HAC clustering is dropped from both rules (cluster_network now raises if
algorithm='hac') along with the unused 'to_substations' and 'feature'
config knobs. simplify_network.py is removed; its functions move into
aggregate_to_substations.py.
Snakemake dry-run on the default Western config produces the expected
chain: aggregate_to_substations -> cluster_simpl -> cluster_network.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Dead-code cleanup audit against v1-epic. No behavior change.
YAML config keys removed (no code reference):
- config.plotting.yaml: costs_max, costs_threshold, energy_max, energy_min,
energy_threshold (with matching docs/source/configtables/plotting.csv rows)
- config.tutorial.yaml: sector.natural_gas.allow_imports_exports
- config.common.yaml: renewable.hydro.{PHS_max_hours, resource.hydrobasins,
resource.flowspeed, hydro_max_hours, clip_min_inflow, normalization,
multiplier}; atlite.default_cutout; electricity.prm regional block
- config.default.yaml: model_topology.interface_transmission_limits;
solving.mem
Snakemake params: declarations removed (declared but never read by
the target script):
- build_bus_regions: focus_weights
- add_extra_components: ucap (script reads via snakemake.config instead)
- prepare_network: adjustments, co2base
- add_sectors: electricity, costs, plotting
- plot_network_maps: plotting, retirement
- plot_statistics: plotting
- plot_natural_gas: plotting
- solve_network: planning_horizons, transmission_network, sector_config
- solve_network_validation: planning_horizons, co2_sequestration_potential
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 1: split simplify_network into aggregate_to_substations + cluster_simpl
Outputs under resources/ previously sat at a single interconnect-rooted
level (e.g. resources/texas/elec_b.nc, resources/texas/busmap_s50.csv,
resources/texas/profile_solar_s50.nc). Files of unrelated types were
intermingled, making the tree hard to navigate.
This change introduces a category-first layout: the top level under
resources/ is what the file IS (networks/, busmaps/, profiles/, ...) and
{interconnect} becomes the next level down.
resources/
networks/{interconnect}/elec_*.nc
busmaps/{interconnect}/busmap_*.csv, bus2sub.csv, sub.csv
profiles/{interconnect}/profile_*.nc, nrel_mapping_cache
geospatial/{interconnect}/*.geojson, bus_gis.csv, lines_gis.csv
costs/costs_{year}.csv, sector_costs_{year}.csv
prices/{interconnect}/{state,ba}_*_prices.csv, pudl_fuel_costs.csv
demand/{interconnect}/{end_use}_*.csv|pkl
population/{interconnect}/pop_layout_*.nc|csv
temperature/{interconnect}/temp_{soil,air}_*.nc
heating_cop/{interconnect}/cop_{soil,air}_*.nc
co2/{interconnect}/co2_storage_*.csv
powerplants/powerplants.csv
Implementation:
- workflow/Snakefile defines twelve category constants (NETWORKS,
BUSMAPS, PROFILES, GEOSPATIAL, COSTS, PRICES, POWERPLANTS, DEMAND,
HEATING_COP, TEMPERATURE, POPULATION, CO2) composed from RESOURCES, so
the RDIR (run name) and shared_resources prefixes still work.
- Every RESOURCES + "{interconnect}/<file>" reference in the .smk files
swapped to the matching category constant. No filenames change.
- "Geospatial/" (capital G) renamed to "geospatial/" (lowercase) for
consistency with the new top-level subfolders.
- "resources/powerplants.csv" hardcoded literal moved to
"resources/powerplants/powerplants.csv" (still shared across runs).
- Two off-rule literal paths updated: a __main__ test path in
build_natural_gas.py and argparse defaults in plot_caps_summary.py.
- Doc reference in about-usage.md sharpened to resources/networks/.
Verified by snakemake -n: full DAG resolves end-to-end for
resources/Default/networks/texas/elec_base_network_l_pp.pkl, with each
intermediate input/output routed to the new category folder.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reorganize resources/ into category-first subfolders
…nd-params Remove unused config keys and unused snakemake rule params
Brings in Phase 2 (move simplify/cluster_simpl ahead of demand+RE) and Phase 3 (aggregate EGS supply curves through cluster_simpl busmap). Both PRs were originally merged into their stacked-base branches rather than v1-epic; this merge propagates their changes up. Resolved conflicts in build_electricity.smk by taking the post-#12 resources/ layout (category-first constants: NETWORKS, GEOSPATIAL, PROFILES, DEMAND, BUSMAPS) and applying Phase 2's path repointing (network refs to elec_s{simpl}.nc, _s{simpl} suffix on demand/profile outputs). Repointed aggregate_egs outputs and add_electricity EGS inputs to PROFILES/BUSMAPS per the new layout.
Two-part approach: a curated docs/network-schema.md catalog of custom columns plus a _helpers.log_network_schema helper that logs per-script column entry/exit diffs. Motivated by recurring consense aggregation crashes (most recently LAF_state in aggregate_to_substations). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Six tasks: add helper + tests, wire into topology / add-* / sectors chains, smoke test on test_small western, seed catalog from logs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Logs component row count and column set on entry; on exit emits row and column diffs vs. the entry snapshot. Logging only — no asserts. Wires into scripts in follow-up tasks; tested in isolation here. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds entry/exit schema logging to build_base_network, aggregate_to_substations, cluster_simpl, cluster_network. Logging only; no behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… add_demand Logging only; no behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…twork Logging only; no behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tiered test pyramid (static <30s + integration <5min) targeting the classes of breakage that the simplify-early refactor (PRs #7-#12) made visible: path/wiring drift, dead config keys, silent artifact-shape regressions. Schema-catalog assertions deferred to a follow-up PR after the schema-tracking spec lands. Brainstorming output. Five-PR migration plan included.
Initial catalog of custom columns on PyPSA components, populated from [schema ...] log output of build_base_network and aggregate_to_substations entry on the small_mh western config. Sector- and electricity-stage columns will be added as those rules get exercised under the schema logger. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop editorial framing ("the bug class this catalog was created to
address"). Keep the factual note that LAF_state is missing from
bus_strategies and the fix is tracked separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five-PR migration: scaffolding → Tier A static checks → Tier B fixture → Tier B full assertions → CI wiring. Plan includes exact code, file paths, verification commands per step. PR 6 (schema assertions) deferred pending the schema-tracking initiative.
Adds a repo-root conftest.py that auto-marks every test collected from workflow/scripts/test/ with the 'fast' marker, so pytest -m fast picks them up without per-file changes. Also skips 9 pre-existing test failures unrelated to this PR (RPS constraint helper signature drift and ERM API drift) so pytest -m fast exits 0. Each skip carries a reason pointing back to v1-epic.
Approved-design doc for a new simpl="county" wildcard value that bypasses k-means and uses the substation network's county FIPS (prefixed with reeds_zone) as a direct busmap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four-task TDD plan: extract resolve_simpl_mode and build_county_busmap helpers, wire them into __main__ dispatch, and document the new simpl="county" wildcard value. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The interconnect parameterization left literal '{IC}' in the regex
pattern lists, so no usa benchmark path matched and only stale western
solve_network rows (whose pattern lacked the interconnect) leaked
through. Patterns now .format(IC=...) at match time and the solve
pattern carries the interconnect.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per user request for the 41k-region national run: 3-panel maps (V1-epic | anchor | difference) for EVERY carrier's existing capacity (identical-vs-differing summary line retained, no more collapse to text), maximum installable capacity (profile p_nom_max + finite extendable generator p_nom_max), capacity-weighted mean p_max_pu per carrier, and demand; profile-file CF maps retained. Zero-linewidth regions and capped dpi keep the 41k-polygon renders legible and the HTML at 3.6MB (29 map figures, 0 failures). Western report regression verified. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…signment
remap_caps_to_cluster now WARNS unconditionally with dropped entry
count / MW / % of national total (CA prong-1: 17,340/17,890 entries,
9.43 of 9.70 TW onwind p_nom_max, 97.3%). New default-off config
nrel_caps_reassign {enable, max_km:100} reassigns unmapped entries to
the nearest in-footprint bus within max_km (haversine, chunked);
published caps carry no x/y so enabling raises a clear config error
until the HPC rollup regenerates - build_nrel_bus_capacities.py now
writes capacity-weighted site-centroid x/y for that regeneration.
Flag-off output verified byte-identical; 7 new unit tests; fast gate
108 passed. Ledger: no new delta while default-off. Long-term option
(a) - re-roll caps per run geometry - stays HPC-side.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Self-contained brief for an HPC-side agent: verify generation geometry against the published 17,890-bus tessellation before building, back up and force-regenerate caps only (avail unchanged), per-file identity verification (x/y additions only), CA recovery quantification with max_km sensitivity, and an explicitly human-gated Zenodo publish step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 22, 2026
Merged
Collaborator
Author
|
Stack index: this PR is the base of the v1-epic stack. Stacked on it (both target 🤖 Generated with Claude Code |
Documents the interconnect-wide empty-county sweep root cause (regions 7x CA, 215.5 GW WECC fleet attached to a CA-demand-only model) and the quantified before/after from branch proto/footprint-scoped-regions (ccfe4b77). Docs only; no pipeline behavior change on v1-epic. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
|
Stack index update: #778 (PyPSA v1 migration — pypsa 1.2.4 / linopy 0.9.1, supersedes #762) joins the stack, base 🤖 Generated with Claude Code |
When a run is scoped with model_topology.include (e.g. reeds_state: [CA]), build_bus_regions previously swept every busless county in the FULL interconnect and glued its geometry onto the nearest retained bus: a CA-only run's onshore regions covered 2.93M km2 (86% outside CA, ~7x the state). Those inflated polygons then passed the whole WECC fleet through filter_plants_by_region's sjoin (215.5 GW existing capacity attached to a CA-demand-only model, incl. 22.6 GW coal and 5.4 GW out-of-state nuclear) and skewed border-bus godeeep CF aggregation. Now the sweep is restricted to the ReEDS zones present in the (already include-filtered) base network. Gated on include being set: unfiltered interconnect runs are byte-identical. After the fix a CA run tiles 409.8k km2 (0.2% out-of-state slivers) and attaches an 84.5 GW fleet that matches California's actual one carrier-by-carrier. Results-changing for scoped runs; pending ledger countersignature. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
6 tasks
All PRs now target develop (never master), so fast-tests/e2e-tests must run there once v1-epic merges. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds the ADOPTED-FIX anchor-patch category to the equivalence harness (apply_adopted_fix_patches in tests/equivalence/build.py): mirrors the DL-11 empty-county-sweep fix onto the anchor worktree, marker-idempotent, plus a one-shot .eq-force-rerun marker that build_side turns into -R build_bus_regions (--rerun-triggers mtime neither reruns on code changes nor revisits missing intermediates when the final target looks current). Ledger gains the countersigned DL-11 row (with the known plants_must_add seam residual and forced-rerun mechanics); CHANGELOG and spec D10 document the two anchor-patch categories. Post-adoption CA prong-1 rerun: data stages clean (80/82 findings waived, matching baseline); the 2 live solved-stage findings trace to a pre-existing build_powerplants.py source-data divergence (diagnosed 2026-08-23, previously absorbed by DL-7's waiver), addressed separately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second ADOPTED-FIX anchor patch (apply_powerplants_adoption): dynamic whole-file adoption of the candidate's build_powerplants.py onto the anchor, sentinel-gated (ges_latest CTE), interface-checked against the pristine e7f8bd7 file, idempotent by content comparison; .eq-force-rerun writing is now merge-safe (mark_force_rerun). Root cause of the 2.34% prong-1 solve divergence: upstream joins EIA-860 tables raw so ~24 years of report_date vintages reweight its means (Watson Cogen at an impossible 62% efficiency); v1-epic pre-aggregates. PUDL release and all tracked inputs identical — the query was the sole divergence. Harness after adoption (both sides rebuilt, candidate's stale powerplants.csv regenerated too): prong 1 PASS 0 live/72, objective rel 2.34e-2 -> 2.1e-6, per-carrier p_nom_opt equal to 0.01 MW; prong 2 PASS 0 live/3 (all DL-9-class, recalibrated: solar gap unchanged 3,586.6 MW, onwind gap 3,680.1 MW after DL-11 decontamination). DL-7 re-scoped: efficiency + marginal_cost members were DL-12 source data (waivers deleted as dead); only the non-composable unweighted fuel_cost mean survives, metadata-only. Countersigned (ktehranchi, 2026-08-23). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
filter_plants_by_region filters plants by sjoin against regions_onshore / regions_offshore. Since DL-11 (88bede4) those layers tile only the model footprint in scoped runs — a model_topology.include: {reeds_state: [CA]} run tiles 409.8k km2 instead of 2.93M km2. The plants_must_add fallback, however, stayed unconditional: it selects plants lying outside every ReEDS shape of the run's interconnect whose ReEDS-membership interconnect disagrees with their EIA `interconnection`, and concatenates them into plants_filt with no test against the (now footprint-sized) regions. match_plant_to_bus's second pass then assigns any plant without a zone match to the nearest network bus with no distance bound. Measured on the CA equivalence run (western, 2030, simpl=''), that fallback leaks 23 plants / 1,887.4 MW onto California buses — NM 1,112.1 MW of wind and solar, MT 370.3 MW (incl. Fort Peck hydro 162.4 MW across 4 units), Buffalo Ridge II SD 210.0 MW, and Hardy Hills Solar IN 195.0 MW from 2,508 km away. Every entry is at least 890 km from the CA footprint, so 50/100/200 km all drop the whole population. Fix: when the run is footprint-scoped, keep only must-add plants within SEAM_PLANT_MAX_KM = 100 km of the union of regions_onshore + regions_offshore, measured in EPSG:5070. Plants inside the footprint have distance 0 and are always kept, so genuine near-seam plants still attach — match_plant_to_bus is deliberately left alone, since its unbounded second pass is correct once the leak population is filtered upstream. Gated on model_topology.include being truthy (read from snakemake.config in main(), threaded as the filter_plants_by_region(footprint_scoped=...) parameter). With the gate off not a single statement changes, so unfiltered interconnect/usa runs are byte-identical BY CONSTRUCTION. That gate is load-bearing, not cosmetic: against a full-western footprint the same population is mostly legitimate, and an unconditional 100 km bound would delete 8 plants / 694.9 MW (incl. Hardy Hills and Buffalo Ridge II) from an unfiltered western run. Every dropped plant is logged at WARNING with name, carrier, state, MW and distance, plus a summary line with count and total MW. New unit tests in workflow/scripts/test/test_seam_plants.py build synthetic regions/ReEDS shapes/membership so an in-footprint plant, a ~50 km seam plant and a ~500 km far plant all land in plants_must_add, then pin gate-off keeping all three (legacy behavior) and gate-on keeping the first two while dropping and loudly logging the third. workflow/scripts/test/: 41 passed, 10 skipped (pre-existing skips), no regressions. Full quantification, method and caveats: docs/superpowers/specs/2026-08-23-seam-plant-quantification.md — including three findings adjacent to this fix and deliberately NOT changed here: the primary sjoin leaks zero out-of-state plants; the plants_nearshore sjoin_nearest path compares EPSG:4326 degrees against EPSG:3857 metres and so matches nothing (11 SD plants / 230.0 MW silently dropped today); and 46 plants / 3,326.3 MW (CA 41, WA 5) fall outside every national ReEDS shape before the fallback runs. Results-changing for scoped runs; pending ledger countersignature. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mirrors v1-epic d98cb93 onto the pinned anchor e7f8bd7 as the third ADOPTED-FIX patch, so the harness keeps comparing like-for-like instead of scoring the candidate against a known-leaking anchor. apply_seam_adoption does targeted string surgery rather than DL-12's whole-file adoption: v1-epic's add_electricity.py legitimately differs from the anchor's in the simplify-early bus2sub/sub_id removals, the length_factor=1.0 decision (DL-1/DL-2) and the schema-logging calls, so copying it wholesale would smuggle those unrelated deltas onto the anchor. The whole filter_plants_by_region body is byte-identical between e7f8bd7 and v1-epic, so the anchor takes the same footprint_scoped parameter plumbing as the candidate rather than an inlined-config variant. Verified AST-identical across both sides: the helper body (docstrings stripped), the SEAM_PLANT_MAX_KM value, the filter_plants_by_region signature, the gated call block and the main() wiring. The constant block and the helper body are sliced out of the LIVE candidate file so the numeric logic both sides run is the same text and any drift in v1-epic's helper re-triggers the forced rerun; only the four wiring edits, which must adapt to the anchor's own shape, are hardcoded. Safety rails match apply_powerplants_adoption: the candidate must carry the sentinel and yield both slices, all four needles are verified to occur exactly once against the PRISTINE anchor file fetched from git (not the possibly already-patched worktree), the pristine anchor must not already contain the sentinel, and the result is checked for end-to-end footprint_scoped wiring before anything is written. Idempotent on the sentinel; marks add_electricity for the one-shot forced rerun on newly-applied. Also wires EQ_UNTIL=assembled into run.py's build targets. paths.py has stopped the COMPARED pairs at the assembled stage since the harness was written, and exported assembled_target/anchor_assembled_target for the matching build targets, but run.py never used them — so an EQ_UNTIL=assembled run still drove the whole chain through a solve it then did not compare. At usa scope that is a national solve. Inert for the CA runs, which do not set the variable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rint
The equivalence harness's CA prong 2 (simpl=20) died in add_electricity with
GEOSException: TopologyException: side location conflict, raised by
_drop_distant_seam_plants' pd.concat(region_geoms).union_all().
Root cause is the reprojection, not the source data: regions_onshore_s20 and
regions_offshore_s20 are 100% valid as stored in EPSG:4326, but converting them
to EPSG:5070 for the distance measurement leaves 9 of 29 polygons
self-intersecting or degenerate ("Too few points in geometry component"), and
GEOS union_all refuses invalid input. Prong 1 never tripped it because the
simpl='' layer reprojects to 2,014 fine-grained polygons that all stay valid --
so the bug was latent behind the granularity of the first prong tested.
The union was only ever a means to a distance, and dist(p, union(R_i)) equals
min_i dist(p, R_i), so take the minimum over regions directly and skip the
union. Pairwise distance is robust to self-intersection where the union is not.
Verified equal, not merely similar: on the simpl='' layer where union_all does
succeed, the two methods agree to 0.000000000 m over 60 probe points; on the
simpl=20 layer the union raises and the per-region path returns clean
distances. Cost is 0.215 s at prong-1 scale (23 plants x 2,014 regions).
Also drops empty/NA geometries before measuring, and returns early if nothing
survives.
New regression test builds a self-intersecting bowtie plus an overlapping box --
the minimal shape that reproduces the GEOS failure -- placed far from every test
plant so the expected keep/drop set is unchanged and only the distance path is
under test. Confirmed to fail with the original union_all implementation
(identical TopologyException) and pass with this one. workflow/scripts/test/:
42 passed, 10 skipped.
Harness side: apply_seam_adoption's idempotence is now by CONTENT rather than by
the sentinel string, matching apply_powerplants_adoption. Sentinel-based
skipping would have left the anchor running the pre-fix helper forever, since
the sentinel is present either way; content comparison re-applies the patch and
re-arms the one-shot forced-rerun marker whenever the candidate's helper
changes. Verified: the anchor picked up this fix automatically on the next
provision, and both sides remain AST-identical in helper body, gated block,
signature and main() wiring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Records the countersigned seam-plant bound adoption (d98cb93 + 103f219 + anchor mirror 85cda59) with the corrected symmetric magnitude (1,725.0 MW leaves the assembled network on both sides, not the 27 MW name-count estimate), the harness verdicts (prong 1 PASS 0 live, objective rel 2.46e-06; prong 2 PASS with DL-9 absolute gaps exactly unchanged), and the GEOS per-region-distance robustness fix. Also gitignores the transient .eq-force-rerun marker. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fuel_cost residual is 8.1277 $/MWh (40.24%) on current artifacts (8.4498 when first measured post-DL-12); adds the three-fix invariance of the DL-9 absolute gaps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
usa data-stage harness PASS 0 live/113 - identical count and class structure to the pre-adoption baseline, confirming DL-11/12/13 left the national comparison untouched. Seam gate verified on all axes (source AST, config eval, 0 seam-drop lines in both sides' usa logs vs 24 at CA). usa report regenerated, superseding the stale 2026-08-22 artifacts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Merged
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Staging PR of the complete v1-epic line against a dedicated upstream
v1-epicbranch (created ate7f8bd70, the current tip ofdevelop, so this diff is exactly the v1-epic work). Mirrors ktehranchi#22, where the fork-side review runs.Contents: the simplify-early DAG restructuring — substation aggregation and
{simpl}clustering (cluster_resources) moved ahead of the per-bus heavy rules, so renewable profiles, demand, andadd_electricityrun at cluster granularity; the category-firstresources/layout; a tiered test suite (static checks, integration build, and a Tier C results-equivalence harness); and the fixes + signed accepted-deltas ledger that the harness produced. Full change record: CHANGELOG-v1-epic.md; master spec: pipeline-equivalence-and-perf-design.md.Why trust it: equivalence vs
develop(e7f8bd7) is machine-checkedTwo-prong, config-only protocol (
tests/equivalence/): prong 1 runs both branches withsimpl=''(identical clustering by construction) and requires every downstream artifact to match; prong 2 runs production-style clustering and checks clustering-invariant aggregates.Bugs found and fixed by the harness (all silent on develop-style validation)
length_factor(decision recorded: factor lives inlength; affects TAMU runs only — reeds transport rebuilds ITL costs).simpl=''from float-formatted region names.Two latent develop-side issues surfaced en route (documented, not fixed here): DC-link aggregation deflates the length-independent inverter-pair cost term by 1/length_factor (ledger DL-2 amendment), and
attach_renewable_capacities_to_atlitesilently drops existing-RE capacity in profile-less groups ('issue #16'; a prototype fix recovers 100%).Performance
CA-slice per-rule wall (V1-epic vs develop): demand 139s vs 217s, solar profiles 39s vs 61s, onwind 34s vs 42s, add_electricity 13s vs 15s; solve identical (same LP, as equivalence requires). Full-Western pre-harness measurements: onwind profiles 142→28s, add_electricity 387→116s, power_build_demand 352→171s. The whole-US data pipeline builds in ~16 minutes on a laptop. Memory instrumentation is spec'd for a Linux/HPC pass (macOS reports max_rss=0).
Review notes
tests/equivalence/) is interconnect-parameterized (EQ_INTERCONNECT) and regenerates its own baseline from this branch's merge-base — reviewers can rerun everything locally; see the master spec.{simpl}-stage rule is nowcluster_resources.🤖 Generated with Claude Code