Skip to content

Docs overhaul (power sector): correctness fixes, Model Description section, generated figures - #790

Merged
ktehranchi merged 75 commits into
PyPSA:developfrom
ktehranchi:docs-overhaul-develop
Aug 28, 2026
Merged

Docs overhaul (power sector): correctness fixes, Model Description section, generated figures#790
ktehranchi merged 75 commits into
PyPSA:developfrom
ktehranchi:docs-overhaul-develop

Conversation

@ktehranchi

Copy link
Copy Markdown
Collaborator

Re-lands #776. Closes nothing new.

Why this PR exists

PR #776 ("Docs overhaul (power sector)") was merged — but into the wrong branch. At
the time two branches shared the name v1-epic: the fork branch ktehranchi:v1-epic
(the real simplify-early epic) and an upstream branch PyPSA/pypsa-usa:v1-epic that
was a stale snapshot of develop carrying none of the epic content. #776 targeted the
stale upstream one, so its squash commit (9697fb44) landed on a branch nothing else
merges from.

#775 then merged the fork's epic branch into develop. The epic itself is on
develop and correct (git diff ktehranchi/v1-epic upstream/develop -- workflow/ tests/
is empty), but the docs overhaul never travelled with it and is currently stranded.

This PR re-lands exactly that content on top of current develop.

Changes proposed in this Pull Request

Identical in substance to what was reviewed in #776 — 67 files, 53 of them under
docs/source/, no dependency changes:

  • P0 correctness — fixed literalinclude scoping in the config reference (the old
    :start-at: <key>: convention matched by substring and sliced the wrong YAML block),
    completed the configtables/*.csv set (new offwind.csv, corrections across
    atlite, clustering, costs, electricity, emissions, load,
    nrel_exclusion, onwind, opts, solar, solving), and corrected the
    getting-started and power-sector data pages against the current pipeline.
  • New Model Description sectionmodel-workflow.md, model-components.md,
    model-constraints.md, model-network-schema.md, plus release-notes.md.
  • Figures — regenerated _static/dag.svg, added generated figures
    (_static/generated/network_aggregation.png, example_outputs.png) with a
    docs_figures Snakemake rule and workflow/scripts/plot_docs_figures.py that
    produces them; PDF-safe figures on the sector pages and an image diet across
    _static/.
  • Guard teststests/docs/test_docs_config.py asserts every literalinclude
    slice resolves to exactly the YAML keys it claims to document, and that
    workflow/config/ has not drifted from the canonical workflow/repo_data/config/.
  • Build hygiene.readthedocs.yaml, docs/requirements.txt, docs/source/conf.py.
  • Config annotations# docs : <NAME> anchors and inline comments in
    workflow/repo_data/config/config.{common,default,cluster}.yaml; workflow/config/config.common.yaml
    re-synced to the canonical template it had drifted from. One behavioural
    consequence to be aware of: that re-sync brings the live config.common.yaml in line
    with the template's EGS.drilling_cost: advanced, EGS.seismic_exclusion: true and
    the ucap: block, and drops the unused offshore_network.enable key. This was part
    of Docs overhaul (power sector): correctness fixes, Model Description section, generated figures #776 as reviewed.
  • One non-docs line: workflow/rules/build_electricity.smk now reads
    config.get("renewable_scenarios", ["historical"])[0] instead of indexing directly,
    so the GoDEEEP horizon guard does not raise when the key is absent.

Merge resolution notes

Conflicts were epic-vs-epic (both sides carry the same epic, one as 88 commits and one
as squash 01f742f1) and were resolved to develop's content. Where develop's epic
had moved on since docs-overhaul branched, the docs were updated to match rather than
reverted:

  • The rule cluster_simpl was renamed to cluster_resources on develop after this
    branch forked. Rule-name references in the user-facing pages
    (about-introduction, config-spatial, config-configuration, config-wildcards,
    model-workflow, release-notes, configtables/clustering.csv) and the walltime
    keys in config.default.yaml / config.cluster.yaml were updated accordingly. The
    script filename is still cluster_simpl.py on develop, so references to the file
    were left alone. docs/source/_static/dag.svg is a generated artifact and still
    shows the old rule label; it needs a snakemake dag re-render, which was not run here.
  • develop's nrel_caps_reassign key falls inside the documented # docs : NREL_EXCLUSION
    slice, so it was added to the expected key set in tests/docs/test_docs_config.py.
  • docs/source/publications.bib is byte-identical to develop — this branch never
    touched it, so the v0.9.0 entries (barnes2026b, ai2026, barnes_2026a) and the
    removal of the duplicate barnes_2026 key are preserved as-is.
  • pyproject.toml and uv.lock are byte-identical to develop.

Testing

  • pre-commit ran on the merge commit and passed with no file modifications.
  • pytest tests/docs/ — 38 passed, 1 failed. The failure is
    test_config_tree_sync, which compares workflow/config/ against
    workflow/repo_data/config/. It only passes in a checkout where
    init_pypsa_usa.sh has populated workflow/config/; in a bare clone that
    directory holds only the tracked config.common.yaml, so the test reports the
    other templates as missing instead of skipping. This is inherited from Docs overhaul (power sector): correctness fixes, Model Description section, generated figures #776 as
    reviewed, not introduced by this merge, and is worth a follow-up to tighten the
    skip condition.
  • Not run here: no Sphinx build, no snakemake invocation of any kind (so the
    docs_figures rule and the DAG re-render were not exercised), no Tier A/Tier B
    pytest run, no equivalence harness. The Read the Docs PR build on this PR is the
    first real check that the pages render.

Checklist

  • I tested my contribution locally and it seems to work fine. — not ticked: the
    docs were not built locally in this pass; see Testing above.
  • pre-commit passes locally. pytest -m fast was not run; pytest tests/docs/
    results are above.
  • Code and workflow changes are sufficiently documented.
  • Changed dependencies are added to workflow/envs/environment.yaml. — no
    dependency changes; pyproject.toml and uv.lock match develop exactly.
  • Changes in configuration options are added in workflow/repo_data/config/config.default.yaml.
  • Changes in configuration options are also documented in docs/source/configtables/*.csv.

ktehranchi and others added 30 commits May 13, 2026 15:39
The LEFT JOINs in load_eia_operable_data and load_heat_rates_data joined
the multi-year EIA-860 SCD tables (yearly_generators, scd_plants,
scd_generators_energy_storage) without any date constraint. Each
generator carries ~24 years of annual snapshots, so the intermediate
join produced ~7,000 rows per generator (~200M rows for
load_heat_rates_data) before the final aggregation collapsed them.

Pre-aggregating each SCD table to one row per plant/generator using the
same array_agg(... ORDER BY report_date DESC) FILTER (WHERE ... IS NOT
NULL)[1] "latest non-null" pattern already used elsewhere in the file
keeps the projection identical while reducing the join cardinality from
~24x24 to 1x1.

Measured against s3://pudl.catalyst.coop/v2025.5.0:
- peak RSS:  17.3 GB -> 2.2 GB
- runtime:    503 s   -> 24 s

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
These PNGs are referenced in the PR body to visualize the original-vs-
refactored marginal_cost and heat_rate distributions. Safe to remove
once the PR is reviewed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
build_powerplants: pre-aggregate SCD tables to fix memory blowup
Phase 1 of the simplify-early refactor. Splits the existing simplify_network
rule into two rules with no behavior change for the default kmeans algorithm:

- aggregate_to_substations: pure-topology aggregation to substations
  (convert_to_voltage_level, remove_transformers, busmap-by-sub_id).
  Writes resources/{interconnect}/elec_b.nc + busmap_b.csv.
- cluster_simpl: optional k-means/modularity reduction to {simpl} clusters.
  Reads elec_b.nc, writes elec_s{simpl}.nc + simpl regions, matching the
  former simplify_network output interface so downstream rules
  (cluster_network etc.) are untouched.

HAC clustering is dropped from both rules (cluster_network now raises if
algorithm='hac') along with the unused 'to_substations' and 'feature'
config knobs. simplify_network.py is removed; its functions move into
aggregate_to_substations.py.

Snakemake dry-run on the default Western config produces the expected
chain: aggregate_to_substations -> cluster_simpl -> cluster_network.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Dead-code cleanup audit against v1-epic. No behavior change.

YAML config keys removed (no code reference):
- config.plotting.yaml: costs_max, costs_threshold, energy_max, energy_min,
  energy_threshold (with matching docs/source/configtables/plotting.csv rows)
- config.tutorial.yaml: sector.natural_gas.allow_imports_exports
- config.common.yaml: renewable.hydro.{PHS_max_hours, resource.hydrobasins,
  resource.flowspeed, hydro_max_hours, clip_min_inflow, normalization,
  multiplier}; atlite.default_cutout; electricity.prm regional block
- config.default.yaml: model_topology.interface_transmission_limits;
  solving.mem

Snakemake params: declarations removed (declared but never read by
the target script):
- build_bus_regions: focus_weights
- add_extra_components: ucap (script reads via snakemake.config instead)
- prepare_network: adjustments, co2base
- add_sectors: electricity, costs, plotting
- plot_network_maps: plotting, retirement
- plot_statistics: plotting
- plot_natural_gas: plotting
- solve_network: planning_horizons, transmission_network, sector_config
- solve_network_validation: planning_horizons, co2_sequestration_potential

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 1: split simplify_network into aggregate_to_substations + cluster_simpl
Outputs under resources/ previously sat at a single interconnect-rooted
level (e.g. resources/texas/elec_b.nc, resources/texas/busmap_s50.csv,
resources/texas/profile_solar_s50.nc). Files of unrelated types were
intermingled, making the tree hard to navigate.

This change introduces a category-first layout: the top level under
resources/ is what the file IS (networks/, busmaps/, profiles/, ...) and
{interconnect} becomes the next level down.

  resources/
    networks/{interconnect}/elec_*.nc
    busmaps/{interconnect}/busmap_*.csv, bus2sub.csv, sub.csv
    profiles/{interconnect}/profile_*.nc, nrel_mapping_cache
    geospatial/{interconnect}/*.geojson, bus_gis.csv, lines_gis.csv
    costs/costs_{year}.csv, sector_costs_{year}.csv
    prices/{interconnect}/{state,ba}_*_prices.csv, pudl_fuel_costs.csv
    demand/{interconnect}/{end_use}_*.csv|pkl
    population/{interconnect}/pop_layout_*.nc|csv
    temperature/{interconnect}/temp_{soil,air}_*.nc
    heating_cop/{interconnect}/cop_{soil,air}_*.nc
    co2/{interconnect}/co2_storage_*.csv
    powerplants/powerplants.csv

Implementation:
- workflow/Snakefile defines twelve category constants (NETWORKS,
  BUSMAPS, PROFILES, GEOSPATIAL, COSTS, PRICES, POWERPLANTS, DEMAND,
  HEATING_COP, TEMPERATURE, POPULATION, CO2) composed from RESOURCES, so
  the RDIR (run name) and shared_resources prefixes still work.
- Every RESOURCES + "{interconnect}/<file>" reference in the .smk files
  swapped to the matching category constant. No filenames change.
- "Geospatial/" (capital G) renamed to "geospatial/" (lowercase) for
  consistency with the new top-level subfolders.
- "resources/powerplants.csv" hardcoded literal moved to
  "resources/powerplants/powerplants.csv" (still shared across runs).
- Two off-rule literal paths updated: a __main__ test path in
  build_natural_gas.py and argparse defaults in plot_caps_summary.py.
- Doc reference in about-usage.md sharpened to resources/networks/.

Verified by snakemake -n: full DAG resolves end-to-end for
resources/Default/networks/texas/elec_base_network_l_pp.pkl, with each
intermediate input/output routed to the new category folder.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reorganize resources/ into category-first subfolders
…nd-params

Remove unused config keys and unused snakemake rule params
Brings in Phase 2 (move simplify/cluster_simpl ahead of demand+RE) and
Phase 3 (aggregate EGS supply curves through cluster_simpl busmap).
Both PRs were originally merged into their stacked-base branches rather
than v1-epic; this merge propagates their changes up.

Resolved conflicts in build_electricity.smk by taking the post-#12
resources/ layout (category-first constants: NETWORKS, GEOSPATIAL,
PROFILES, DEMAND, BUSMAPS) and applying Phase 2's path repointing
(network refs to elec_s{simpl}.nc, _s{simpl} suffix on demand/profile
outputs). Repointed aggregate_egs outputs and add_electricity EGS
inputs to PROFILES/BUSMAPS per the new layout.
Two-part approach: a curated docs/network-schema.md catalog of custom
columns plus a _helpers.log_network_schema helper that logs per-script
column entry/exit diffs. Motivated by recurring consense aggregation
crashes (most recently LAF_state in aggregate_to_substations).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Six tasks: add helper + tests, wire into topology / add-* / sectors
chains, smoke test on test_small western, seed catalog from logs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Logs component row count and column set on entry; on exit emits row
and column diffs vs. the entry snapshot. Logging only — no asserts.
Wires into scripts in follow-up tasks; tested in isolation here.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds entry/exit schema logging to build_base_network,
aggregate_to_substations, cluster_simpl, cluster_network. Logging
only; no behavior change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… add_demand

Logging only; no behavior change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…twork

Logging only; no behavior change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tiered test pyramid (static <30s + integration <5min) targeting the
classes of breakage that the simplify-early refactor (PRs #7-#12) made
visible: path/wiring drift, dead config keys, silent artifact-shape
regressions. Schema-catalog assertions deferred to a follow-up PR after
the schema-tracking spec lands.

Brainstorming output. Five-PR migration plan included.
Initial catalog of custom columns on PyPSA components, populated from
[schema ...] log output of build_base_network and
aggregate_to_substations entry on the small_mh western config.
Sector- and electricity-stage columns will be added as those rules
get exercised under the schema logger.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop editorial framing ("the bug class this catalog was created to
address"). Keep the factual note that LAF_state is missing from
bus_strategies and the fix is tracked separately.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five-PR migration: scaffolding → Tier A static checks → Tier B fixture
→ Tier B full assertions → CI wiring. Plan includes exact code, file
paths, verification commands per step. PR 6 (schema assertions) deferred
pending the schema-tracking initiative.
Adds a repo-root conftest.py that auto-marks every test collected from
workflow/scripts/test/ with the 'fast' marker, so pytest -m fast picks
them up without per-file changes.

Also skips 9 pre-existing test failures unrelated to this PR (RPS
constraint helper signature drift and ERM API drift) so pytest -m fast
exits 0. Each skip carries a reason pointing back to v1-epic.
Approved-design doc for a new simpl="county" wildcard value that
bypasses k-means and uses the substation network's county FIPS
(prefixed with reeds_zone) as a direct busmap.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four-task TDD plan: extract resolve_simpl_mode and build_county_busmap
helpers, wire them into __main__ dispatch, and document the new
simpl="county" wildcard value.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ktehranchi and others added 23 commits August 7, 2026 17:42
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cluster_network's reeds transport-model path requires clusters == the
footprint's ReEDS zone count (4 for CA); discovered by the anchor build
assert. Even better for prong 1: the zonal busmap is pure attribute
membership - no kmeans anywhere in the chain.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…wth/emission_prices/social_discount_rate/min_year)

Direct-subscript reads in add_extra_components and prepare_network on
both branches; values mirror config.default.yaml.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cluster_simpl's pass-through branch copied regions geojson verbatim,
keeping float-formatted names ('35827.0') while the network/busmap use
bare IDs ('35827'). The godeeep profile path's three-way bus
intersection came up empty, silently writing 0-bus profiles that
crashed add_electricity two rules later. Fix normalizes names in the
identity branch (same as cluster_regions does for kmeans), adds a
fail-loud disjoint-bus-spaces guard in build_renewable_profiles, and
skips carriers with no profile buses in attach_wind_and_solar.

Results effect: simpl='' runs regain silently-dropped RE generators;
simpl=N unchanged. Found by the first candidate equivalence build.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
regional_Co2_limits (read by add_regional_co2limit, opts/policy.py:548,
fired by the REM opt), plus sibling TCT/RPS keys. Dead keys
(transmission_interface_limits, SAFE_*, agg_p_nom_limits) deliberately
omitted - zero readers on either branch. Both prong-1 solves now
complete optimally: candidate obj -2.56292e9, anchor -2.55060e9
(0.48% delta - goes to the harness comparison for adjudication).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
aggregate_to_substations folds lines.length_factor into length; the
post-reorder add_electricity call multiplied it again (hav x 1.5625
instead of x 1.25 -> 25% line CAPEX inflation in line-preserving runs).
Harness catch: assembled-stage comparison, ratio 1.2486 on 2810/2811
lines vs anchor. Clustered/solved outputs of reeds transport runs were
unaffected (ITL costs recomputed identically on both branches).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Float-label normalization, Load re-keyed on bus attr, demand pair
corrected to system-total check (anchor CSV is nodal), per-frame
finding cap with suppression marker, Generator[bus,carrier] p_nom
aggregate view. Prong-1 findings: 16882 -> 379; profiles fully clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ign-off)

Stage-ordering capital-cost residues (clustered stage verified
identical), plotting palette, slack-bus bookkeeping. Open findings
(demand -6.3%, hydro attachment, storage sets, naming, objective
0.48%) remain unwaived pending investigation verdicts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ation composition)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Hydro: attach_breakthrough_renewable_plants filtered raw plant.csv
bus_ids against the substation-level network index - 12,848 MW of CA
hydro silently dropped, 128.6 MW of Eastern hydro attached at
colliding ids. Fix remaps through bus2sub + busmap_s{simpl} (patterned
after aggregate_egs; busmap_s added to rule inputs). Verified: hydro
p_nom 12,976.8 MW == anchor exactly; zero Eastern collisions.

Demand: remove_transformers dropped 420 trafo-secondary buses with
their Pd/LAF_state before demand existed; build_demand allocates by
LAF_state without renormalizing -> -6.281% total demand and 95 zero-
load substations. Fix transfers Pd/LAF_state through the trafo map and
composes it into busmap_b.csv (full 4,248-bus coverage).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All fixes landed; solved objective rel diff 6.6e-05 (gate 1e-3);
comparison: 0 live / 305 total findings. Every waived class carries a
ledger row with quantified evidence; content-bearing aggregates
(Generator[bus,carrier], Load_t.p_set, StorageUnit totals) remain
unwaived as guards.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Investigation closed the MW accounting exactly: the +15%/+8% existing
onwind/solar delta is the shared silent-drop of plants in profile-less
groups (upstream issue #16) interacting with by-design different
cluster geometries; candidate strictly closer to ground truth.
Waivers now match on prong so prong-1's objective gate stays live.
Both prongs PASS; pytest -m equivalence: 2 passed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ered dark-on-dark)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New section compares generator p_nom grouped by carrier / reeds_zone /
build-year decade for two pairings: each side's add_electricity output
and the simplify-stage networks. Benchmark table now filters to this
harness's rules only (historic runs polluted the shared cluster_network
benchmark path), shows wall time + max RSS per rule for both sides, and
ends with a TOTAL row (wall sum, RSS peak) with cross-side caveats.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Grilling-session design: executive summary (verdict banner, provenance
fingerprint, delta index from the ledger, stage timeline strip),
unified diff-DAG built from both sides' rulegraphs with per-rule
walltime annotations, stage-ordered narrative spine with
born/inherited/masked delta-class tracking, 3-panel maps
(V1-epic | anchor | diff) reusing plot_network_maps choropleths at the
assembled + solved stages (identical quantities collapse to a sentence;
the only capacity map rendered is oil - the 11.3 MW DL-5 Catalina
placement), and a rule-grouped benchmark table with instance subrows
and cross-side caveats. candidate -> V1-epic in all user-facing text.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…uard tests

The literalinclude marker convention silently mis-scoped most config sections
(the 'costs' section rendered the imports block; the real costs: block never
rendered). Every documented key now carries a '# docs : NAME' marker and every
include uses start-after/end-before against markers; tests/docs asserts each
rendered slice contains exactly its own key and that workflow/config stays in
sync with repo_data/config. Configtables corrected against live config and
scripts (real carrier names, StorageUnit/Link keys, dead PyPSA-Eur keys
removed, EGS/policy/temporal rows added, offwind.csv created). config-wildcards
rewritten for the post-refactor DAG (cluster_simpl, no {cutout} wildcard,
full m/a/c suffixes). CCTS SVGs embedded as figures (prose untouched).
Also: renewable.dataset + renewable_weather_years defaults in config.common.yaml
and a defensive renewable_scenarios read unblock tutorial-config DAG parsing;
dead simplify_network walltime keys replaced.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
about-usage: real solved-network path (s{simpl}, c-prefix, {sector}, run-name
dir) with a filename decoder; --configfile added to the troubleshooting command
(Snakefile's default configfile is commented out); output-locations section.
about-install: Python >=3.11,<3.12 stated, uv sync step, load-bearing pins,
solver alignment. README: minimal quickstart. data-generators: godeeep is the
default dataset, profiles are built at {simpl} resolution (removed the
pre-refactor 41,564-zone story), heading-level fix. data-demand: eer profile
source, corrected horizons, new demand-disaggregation section wiring the
orphaned pop_layout figures. data-transmission: kmeans/modularity only.
data-policies: provenance-first rewrite linking to model-constraints.
config-spatial: simplify-early mechanism, verified topological_boundaries set
(county|reeds_zone|state — balancing_area raises). renewables datatable: GODEEEP
+ NREL reV rows; orphaned sector_natural_gas.csv removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ated DAGs, generated figures

New toctree section: model-workflow (stage-by-stage DAG narrative with
rule/script/output tables and the simplify-early rationale), model-components
(PyPSA component usage, spatial and temporal structure), model-constraints
(every power-sector extra_functionality constraint with trigger, LaTeX
formulation, and source link; core LP formulation delegated to upstream PyPSA),
model-network-schema (publishes docs/network-schema.md via include wrapper),
release-notes (user-facing v1 summary). about-introduction rewritten: model
description up front, local DAG figure instead of the master-branch hotlink,
real repository tree replacing the cookiecutter placeholder. Both DAG images
regenerated from the current rule graph (dag.svg for docs, dag_sector.jpg via
sector-config rulegraph). New docs_figures rule + plot_docs_figures.py renders
network-aggregation and example-output figures from real workflow artifacts
into _static/generated/.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mechanical pass on the frozen sector pages: all 37 raw-HTML <img> figures
converted to {figure} directives (raw HTML never reached the LaTeX writer, so
the PDF build readthedocs requests was figure-less); every cross-reference
label preserved; 74 single-word typos fixed; prose and structure untouched.
Oversized images downsampled (worst was 4.8 MB). Build config: docs
requirements pruned to actual docs deps (autodoc has been disabled for a long
time, so pypsa/atlite/cartopy/dask/snakemake were dead weight), sphinx-copybutton
added, vestigial conf.py extensions removed, readthedocs installs graphviz +
imagemagick for diagram/SVG handling. contributing.md documents the config
marker convention and the generated-figure rules. Sphinx build is warning-clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…velop

# Conflicts:
#	.github/workflows/main.yml
#	CLAUDE.md
#	docs/CHANGELOG-v1-epic.md
#	docs/source/config-configuration.md
#	docs/source/configtables/clustering.csv
#	docs/superpowers/specs/2026-08-07-deltas-ledger.md
#	docs/superpowers/specs/2026-08-07-pipeline-equivalence-and-perf-design.md
#	tests/equivalence/build.py
#	tests/equivalence/compare.py
#	tests/equivalence/paths.py
#	tests/equivalence/report.py
#	tests/equivalence/report_sections/benchmarks.py
#	tests/equivalence/report_sections/legacy_plots.py
#	tests/equivalence/report_sections/maps.py
#	tests/equivalence/report_sections/stages.py
#	tests/equivalence/run.py
#	tests/equivalence/waivers.yaml
#	workflow/config/config.common.yaml
#	workflow/repo_data/config/config.common.yaml
#	workflow/repo_data/config/config.default.yaml
#	workflow/repo_data/config/config.equivalence.yaml
#	workflow/rules/build_electricity.smk
#	workflow/scripts/add_electricity.py
#	workflow/scripts/build_renewable_profiles.py
Refresh the docs overhaul onto develop after the pypsa 1.3.0 / linopy 0.9.1 /
pandas 3.0.5 migration (PyPSA#778), the population-based demand allocation (PyPSA#777),
and the untracking of workflow/config/config.common.yaml (PyPSA#791).

Conflicts:
- docs/source/configtables/electricity.csv: kept the overhaul's restructured
  table and re-added develop's electricity.demand.bus_allocation row under the
  branch's '-- ' nesting convention.
- workflow/config/config.common.yaml: resolved to develop's deletion; the file
  is user-generated and now gitignored, so the overhaul's edit is dropped
  (workflow/repo_data/config/config.common.yaml remains canonical).

Also corrected docs/source/about-install.md, which still advertised the
pre-migration pins and claimed PyPSA 1.x was unsupported.
ktehranchi and others added 3 commits August 28, 2026 15:29
# Conflicts:
#	workflow/repo_data/config/config.common.yaml
- data-demand: describe both `electricity: demand: bus_allocation:` modes
  (`population`, the default since PyPSA#777, and `breakthrough`) and the
  `load_weight` column they populate, instead of claiming the weight is
  always Breakthrough `Pd`.
- model-workflow: `add_sectors` runs `add_sectors.py` (not `build_sector.py`),
  and `export_statistics` runs `plot_statistics.py` in `export` mode (there is
  no `export_statistics.py`).
- model-components: `model_topology.topological_boundaries` accepts `county`,
  `reeds_zone`, and `state`; `balancing_area` raises in
  `aggregate_to_substations`.
- about-introduction: 2030/2040/2050 are the EFS horizons; EER covers 2021,
  2025, 2030, 2035, 2040, 2045, 2050 (matches data-demand and
  `ReadEer.MODEL_YEARS`).
- data-policies / release-notes: after PyPSA#791 `workflow/config/` is untracked;
  the policy CSVs are tracked in `workflow/repo_data/config/policy_constraints/`
  and copied by `init_pypsa_usa.sh`.
- about-usage: the default config uses 33 clusters, not 30.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three remaining occurrences of the same verified-false claim. Only
'county', 'reeds_zone' and 'state' are accepted:
aggregate_to_substations.py:139-143 raises ValueError on anything else,
and cluster_network.py's match has no balancing_area branch.

- about-introduction.md:54  (clustering boundary list)
- model-workflow.md:73      (cluster_resources stage description)
- config.default.yaml:73    (inline comment, rendered into the docs by
  the MODEL_TOPOLOGY literalinclude — comment only, the file still
  parses identically)

Other 'balancing' references are left alone: build_shapes really does
emit balancing-authority shapes, buses really do carry a balancing_area
attribute, and demand really is disaggregated from BA level.
@ktehranchi
ktehranchi merged commit 515bf54 into PyPSA:develop Aug 28, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant