Config reorganization: auto-loaded base config, tracked layers, schema validation - #794
Merged
Conversation
config.default.yaml documented every user-facing option but was never loaded, so every scenario config (tutorial, test, equivalence) had to be a full copy of it and drifted structurally. config.test.yaml had drifted far enough that it no longer carried `solving:` or `costs.atb`, which cluster_simpl and build_cost_data dereference directly. Snakefile now loads repo_data/config/config.default.yaml as the last layered configfile, beneath whatever the user passes with --configfile, and config.tutorial.yaml / config.test.yaml become sparse overlays carrying only the keys that differ from the merged base (plus scenario.interconnect, which is pinned deliberately so the footprint cannot follow a change to the default scenario). Lists are replaced wholesale by snakemake's merge, so any differing list is restated in full. The two equivalence configs stay self-contained on purpose: the harness copies them into an anchor worktree (upstream e7f8bd7) that does not load config.default.yaml, so a sparse overlay there would feed the two sides different configs. GURO_PAR_BARDENSETHRESH is added to their gurobi-default block for the same reason - the candidate now inherits it from the default layer and the anchor never would. Verified by replaying snakemake's update_config merge for all five configs against the origin/develop baseline: no key changes value; the only diffs are additions from the default layer, plus model_topology.include/aggregate going from {} to None (update_config maps an empty dict onto a null parent as null). Every consumer of those two keys is a falsy check or an `is not None` guard that skips the same work either way. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d files init_pypsa_usa.sh copied the whole template tree into the gitignored workflow/config/ once and refused to refresh, so every user's layered base (common, plotting, sector, cluster) silently froze at whatever the checkout looked like on the day they ran it. Only API keys and HPC account settings are genuinely per-user. The Snakefile now reads its entire layered base out of the tracked repo_data/config/ templates, and treats config/config.api.yaml and config/config.cluster.yaml as OPTIONAL overlays on top of tracked placeholders - a fresh clone resolves without running init at all. The policy_constraints/ CSVs move the same way: configs, rules and the opts registry now reference repo_data/config/policy_constraints/ so nothing has to be copied. init_pypsa_usa.sh now seeds three files (config.default.yaml as a scenario starting point, config.api.yaml, config.cluster.yaml), is idempotent per-file, and says what it is not copying and why. .gitignore's bare `config/` pattern also matched workflow/repo_data/config, forcing every template edit through `git add -f`; it is now scoped to /workflow/config/*. test_config_tree_sync asserted a copy relationship that no longer exists. It is replaced by two tests that guard the new invariant instead: init may only seed known per-user files, and no `configfile:` may point back into config/. Tier A/B and the Tier C harness are updated for the new paths; the harness keeps feeding the anchor worktree from config/, since that pinned upstream checkout still reads its configs from there. Merged-config diff vs origin/develop: only the seven policy_constraints paths, all repointed from config/ to repo_data/config/ where the tracked CSVs actually live. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
config.common.yaml and config.default.yaml both defined renewable_weather_years,
renewable_scenarios and co2: with identical values, and split renewable: and
lines: across the two files - so answering "where is this configured?" meant
reading both and knowing the load order.
Each top-level key now lives in exactly one file, following where the bulk of
each block already was:
common owns renewable: (per-tech resource/land screens, plus `dataset`,
which was the only thing default contributed)
default owns lines: (the line-type catalog moves over to join
s_max_pu / s_nom_max / max_extension / length_factor),
plus co2:, renewable_weather_years and renewable_scenarios, whose
duplicates in common are deleted.
Both file headers now state the rule and the ownership split. Nothing moves
out of the loaded set, so the merged config is byte-identical: the merge
replay reports zero new differences against the origin/develop baseline, and
a cross-layer scan finds no remaining duplicate top-level keys.
No docs directive referenced the deleted # docs : RENEWABLE_DATASET or
# docs : LINES_TYPES markers; the literalinclude slices that neighbour them
still resolve to the same key sets (verified by tests/docs).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lltimes
The file has always been about the job scheduler, but its name collided with
the `{clusters}` wildcard and the `clustering:` config section, so "the
cluster config" was ambiguous every time it came up.
Per-rule wall times were specified in three places. config.default.yaml's
top-level `walltime:` block is the one the rules actually read
(`config_provider("walltime", "<rule>")` -> `resources: walltime=` ->
`sbatch --time {resources.walltime}`). config.cluster.yaml ALSO carried
per-rule `{rule}: {walltime: ...}` blocks with different values, which
nothing read: run_slurm.sh only dereferences `{cluster.account/email/
partition/output/error}` from `__default__`, and takes the time limit from
resources. Those dead blocks were still injected into the merged config as
top-level keys named after rules.
config.slurm.yaml now holds `__default__` (for --cluster-config) plus the one
`walltime:` block, and config.default.yaml's copy is deleted with a comment
distinguishing it from `solving.walltime`, which caps the solver rather than
the scheduler job. Live values are preserved exactly (config.default.yaml's
won); the two rules that only existed in the dead blocks, aggregate_egs and
solve_network_validation, are folded in - solve_network_validation is read
with no fallback and had been resolving to None.
Merged-config diff: the nine dead `<rule>.walltime` top-level keys are gone,
`walltime.aggregate_egs` and `walltime.solve_network_validation` are added,
and every previously-live walltime value is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The key was reachable only by pasting it into a yaml file inside the checkout. It is now read once in workflow/Snakefile, where the environment wins over config/config.api.yaml, so the secret can stay out of the repo entirely. A dozen rules dereference config["api"]["eia"] at parse time, so the override has to land before the include: statements - it does. The tracked repo_data/config/config.api.yaml keeps the key defined (as null) and documents both routes; the per-user copy under config/ is now an optional overlay. The two __main__ debug blocks in eia.py and build_natural_gas.py follow the same precedence, and no longer hard-require a config/ copy that a fresh clone does not have. Verified: with EIA_API_KEY set, the merged config carries the environment value; unset, it carries the yaml value (null on a bare checkout). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Config typos and out-of-range values had no early failure mode: an unknown
key just sat in the merged dict while the rule that wanted it fell back to
its `config_provider(..., default=...)` value, so the run completed with
quietly wrong numbers.
workflow/schemas/config.schema.yaml is checked by snakemake.utils.validate
right after the configfile chain merges. Scope is deliberately partial:
* the TOP LEVEL stays open - snakemake, the scenarios feature and user
configs all inject keys there, and closing it would make legitimate
extensions fatal;
* well-bounded subtrees are closed (electricity, model_topology, run,
scenario, clustering, solving, costs, conventional, lines, links, co2,
dac, custom_files, snapshots, renewable_snapshots), which is where typos
actually land;
* enum knobs are enumerated from the inline comments in the configs;
* set_default=False, and the schema declares no defaults, so it cannot
become a second invisible source of configuration.
The godeeep conditional requires renewable_land_access to be DECLARED and, if
set, to be one of reference|limited|open. It cannot also require non-null:
config.common.yaml ships it null while selecting godeeep, which
build_renewable_profiles rejects at runtime - a pre-existing defect that is
out of scope here and deliberately left for its own change.
Verified: all five shipped configs validate; `electricity: retirment:` and
each bad enum value fail with a clear ValidationError and exit 1. Tier A
gains tests/static/test_config_schema.py, which replays the merge and the
validation without spawning snakemake.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ty map
Bare nulls in the config read as "unfinished" but several of them are
meaningful, and blanket-converting them to {} would change behaviour. Each
was checked against its consumer:
model_topology.include STAYS NULL. build_base_network reads "not None" as
"a footprint filter was requested" and then asserts topological_boundaries
!= 'state'; {} would trip that assertion on every state-resolution run.
focus_weights STAYS NULL. cluster_network branches on `is not None`,
so {} would take the custom-weights path, re-normalise every region's
factor and log a misleading warning.
costs.max_growth stays null; {} is equivalent (the consumer
short-circuits on None and would otherwise iterate nothing), so there is
nothing to gain by changing it.
model_topology.aggregate BECOMES {}. Its only consumer is a truthiness check
in cluster_network (the parameter is not even referenced inside
convert_to_transport), so {} and None behave identically - and {} is what
config.test.yaml and both equivalence configs merged to before
config.default.yaml sat underneath them, because snakemake's
update_config maps an empty mapping onto a null parent as null.
Net effect on the merged config: config.test.yaml and both equivalence configs
go back to their pre-refactor value ({}), and config.default.yaml/tutorial move
from None to {} with identical behaviour.
Also documents, without changing, that config.common.yaml ships
renewable_land_access null while selecting the godeeep dataset -
build_renewable_profiles rejects that combination, so the shipped templates
cannot build profiles until the user picks a value. Choosing one here would add
the NREL exclusion artifacts to every DAG and move the numbers, so it is left
for its own change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Formatting only, plus @pytest.mark.fast on test_init_copies_only_user_owned_files and test_snakefile_reads_layered_base_from_repo_data so CI's `pytest -m fast` job actually runs them - without the marker they would never execute in CI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The working tree often sits on master, which lags develop badly; an agent analyzing "current state" from it recently reported config issues develop had already fixed. Make the branch policy explicit: all model work — changes, reviews, audits, analysis — must reference origin/develop, and feature branches/PRs base on develop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reorganization commits updated the prose pages but left three statements behind that the new loading order makes false: - The configtables still pointed the five `electricity:` policy-CSV paths (and the `REM` opts row) at `config/policy_constraints/`. The rules now read those CSVs straight out of `repo_data/config/policy_constraints/`; there is no copy step. - about-usage.md's troubleshooting note claimed every invocation must pass `--configfile` and that omitting it dies with `KeyError: 'scenario'`. `config.default.yaml` is now a loaded layer, so a bare `snakemake -n` resolves and builds the shipped defaults (verified by dry-run). Replaced with what actually happens. - The same page told users the `--configfile` in run_slurm.sh defaults to `config/config.default.yaml`; the script has always shipped a personal path. Stated it as the example it is, and named the `--cluster-config config/config.slurm.yaml` argument next to it. Also completes two things the sweep asked for and the pages only half said: about-usage.md now spells out the two ways to author a run (copy the seeded default, or write a bare overlay) and the wholesale-list replacement rule; config-configuration.md notes that $EIA_API_KEY wins over config/config.api.yaml. release-notes.md gains the four user-visible migration items that were missing from a section that promises to list them: the config.slurm.yaml rename, sparse overlays, schema validation at parse time, and the API-key environment variable. Nothing else in docs/source needed changing: the per-section YAML is pulled in by literalinclude, so the in-file comments documenting the load-bearing nulls render on their own, and no configtable attributes a key to a file that the one-file-per-key split moved. pytest tests/docs: 40 passed (unchanged). Sphinx -n build: no warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The shipped templates carried renewable.dataset: godeeep with renewable_land_access null, a pair build_renewable_profiles hard-rejects - so no shipped config could build renewable profiles from a clean checkout. Same drift class as #793: the maintainer's untracked config was papering over a broken tracked default. Set 'reference', NREL's central land-access assumption and the value the equivalence harness validates against (both equivalence configs already pin it, so their merged configs are unchanged). Adds retrieve_nrel_exclusion_artifact jobs to every godeeep DAG (6 for the tutorial). With a working default in place, tighten the schema's godeeep conditional to require non-null (null stays valid for dataset: atlite). This went unnoticed because CI's e2e job caches data/cutouts at the repo root while tests/integration/conftest.py checks workflow/data and workflow/cutouts, so Tier B silently skipped every run. Point the cache at the workflow/ paths and bump DATA_CACHE_NUMBER so stale root-level archives are not restored. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Update the nrel_exclusion configtable (default column, null now atlite-only, correct the restrictiveness ordering: open > reference > limited) and add the new default to the release-notes migration list. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The repo-root conftest auto-marks every workflow/scripts/test/ test as 'fast', and those tests solve with GLPK - a system binary the pip-only fast-tests job never had. All 13 Tier A CI failures on re-enabling the workflow were 'Solver glpk not installed' (or its optimize-status variant); they pass locally where glpsol exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 29, 2026
…organization Conflicts were two comment blocks in config.default.yaml where the reorg's one-owning-file pointer notes met #806's new registry prose; resolved by combining both texts, keeping renewable.dataset owned by config.common.yaml. The registry block itself and the _125m wind-height change auto-merged into common.yaml alongside the reference land-access default. Also declares the new top-level godeeep_cf_registry key in the config schema (permissive - the registry module does its own strict validation). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
for more information, see https://pre-commit.ci
ktehranchi
added a commit
that referenced
this pull request
Aug 31, 2026
Adapts the California/SERVM feature set to the auto-loaded layered base and schema validation from #794: - schema: admit our config surface in the closed sections — run.benchmark_cpuc / benchmark_cpuc_horizons; electricity.honor_planned_retirements, remote_contracted_resources, demand.profile enum += servm, demand.scenario.servm_weather_years; conventional.ambient_derate. - config.california.yaml and config.test.california.yaml rewritten as SPARSE OVERLAYS (programmatic deep-diff against the merged base); the test overlay drops its now-redundant solving/costs/land-access blocks and its stale pre-fix clustering strategies, inheriting the corrected defaults (start_up_cost: sum, capacity-weighted ramps). - policy_constraints paths follow the tracked repo_data/config/ layout (solve rules' interface_limits input, configtables, conftest, docs). - test_dag_dryrun: their repo_data paths + our California cases (incl. county-mode override) as a union. - configtables: their reference-default land-access row (ordering now consistent); our un-deadified transmission_interface_limits description kept, path-updated. Verified: both CA overlays parse against the schema (35-job / 26-job data_model DAGs); fast tier 350 passed, docs 41 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes # (n/a — follow-up to the config-organization review; drift class of #793).
Changes proposed in this Pull Request
Reorganizes how configuration is loaded and maintained, and fixes the shipped-default breakage that the reorganization surfaced.
Config reorganization (results-neutral by construction — see Testing):
config.default.yamlis auto-loaded as the base layer; configs passed via--configfileare now sparse overlays.config.tutorial.yamlshrank 368→93 lines,config.test.yaml109→43 (the latter had drifted badly — it was missing the entiresolving:section thatcluster_simpldereferences). The two equivalence configs deliberately remain full copies: the equivalence anchor worktree doesn't load the default layer, so sparse overlays would feed the two sides different configs.repo_data/config/— the copy-into-workflow/config/drift mechanism (root cause of Restore renewable keys dropped by #791 (fixes fresh-clone parse crash) #793) is gone. Only user-owned files live in git-ignoredworkflow/config/: your scenario config,config.api.yaml, andconfig.slurm.yaml(both existence-guarded, so a bare clone runs withoutinit_pypsa_usa.sh).policy_constraints/CSVs are referenced fromrepo_data/config/directly.renewable:/lines:no longer split across common and default;renewable_weather_years/renewable_scenariosdeduplicated.config.cluster.yaml→config.slurm.yaml(the old name collided with theclusterswildcard /clustering:section) and per-rule walltimes unified into the single consumed block; the dead copies removed. Two rules (aggregate_egs,solve_network_validation) whose walltimes silently resolved toNonenow have values.$EIA_API_KEY, yaml fallback.workflow/schemas/config.schema.yaml): typos likeretirment:orforesight: perfetcnow fail loudly at load instead of silently falling back to code defaults.focus_weights,model_topology.include) documented in-file;model_topology.aggregatedefaults to{}.config-configuration.md,about-usage.md,configtables,release-notes.mdmigration list. CLAUDE.md gains a branch-policy section (work againstdevelop, notmaster).Shipped-default fix (results-affecting for godeeep DAGs):
renewable_land_accessnow defaults toreference. The templates shippedrenewable.dataset: godeeepwith the key null — a pairbuild_renewable_profileshard-rejects, so no shipped config could build renewable profiles from a clean checkout (same drift class as Restore renewable keys dropped by #791 (fixes fresh-clone parse crash) #793: an untracked user config was papering over a broken tracked default).referenceis NREL's central land-access assumption and the value both equivalence configs already pin. Addsretrieve_nrel_exclusion_artifactjobs to every godeeep DAG (6 for the tutorial). With a working default in place, the schema's godeeep conditional requires non-null (null stays valid fordataset: atlite).data/cutoutsat the repo root whiletests/integration/conftest.pychecksworkflow/data/workflow/cutouts, so Tier B has been silently skipping — which is how the broken default went unnoticed. Cache paths fixed,DATA_CACHE_NUMBERbumped so stale root-level archives aren't restored. Note the CI workflow is currently disabled at the repo level; once re-enabled, the first run will rebuild the cache and exercise Tier B for real.Testing
snakemake.utils.update_config(snakemake 7.32.4) over the full layer order asserts deep equality of the merged dict for all five shipped configs vsorigin/develop. Intentional differences only: 7policy_constraintspath repoints, 9 dead per-rule walltime keys removed, 2 missing walltimes filled,model_topology.aggregate: None→{}(behavior-identical; verified against all consumers), the additions the default layer contributes to previously-drifted overlays (each audited; one solver-affecting key patched explicitly into the equivalence configs so the anchor side stays identical), and — from the fix —renewable_land_access: null→referencefor default/tutorial/test (the equivalence configs already pinnedreference, so their merged dicts are unchanged).pytest -m fast: 90 passed, 57 deselected.tests/docs: 40 passed.workflow/scripts/test: 57 passed, 1 pre-existing skip.rule all, defaultdata_model, test, and a bareworkflow/config/clone-state) all resolve; the tutorial DAG picks up exactly the 6 expectedretrieve_nrel_exclusion_artifactjobs.renewable_land_accessunder godeeep fail with clearValidationErrors.pre-commiton all changed files: pass. Sphinx docs build (-n, nitpicky): zero warnings.Checklist
pre-commitandpytest -m fastpass locally.workflow/envs/environment.yaml. (n/a — no dependency changes)workflow/repo_data/config/config.default.yaml.docs/source/configtables/*.csv.🤖 Generated with Claude Code