Re-pin conda environment.yaml for the pandas-3 stack - #802
Merged
Conversation
The v1 migration (#778) bumped pandas to 3.0.5 and geopandas to 1.1.4 in workflow/envs/environment.yaml but left the packages around them at their 2023 values. The file no longer solves on conda-forge, so micromamba dies at env-creation time -- this is what killed the e2e-tests job in CI run 33279801900, and it breaks every mamba-based user setup too. Four hard conflicts, all reproduced locally with micromamba 2.9.0: * pandas 3.0.5 run-constrains `pytables >=3.10.1`, so `pytables==3.9.1` was unsatisfiable. Bumped to 3.10.2, matching pyproject's `tables==3.10.2`. * pandas 3.0.5 run-constrains `lxml >=5.3.0`, so `lxml==4.9.3` knocked out the only py311 pandas build; that in turn made `python==3.11.9` unsatisfiable. lxml is now `>=5.3.0` and python mirrors pyproject's `requires-python` as `>=3.11,<3.12`. * conda-forge's rasterio 1.3.8 builds pin `proj 9.2.1`/`9.3.0` exactly and `libgdal 3.7.x`, whose `pcre2 <10.44` pins are mutually exclusive with the Qt6 stack the `matplotlib` metapackage pulls in. * the `pip:` section's unbounded `tsam` spec resolved to 4.0.0, which caps `pandas<=3.0.3` and so silently uninstalled the conda-built pandas 3.0.5 after the conda solve succeeded. numpy stays at 1.26.0. conda-forge's pandas 3.0.5 declares `numpy >=1.26.0,<3`, so the numpy-1 hold that rasterio/atlite need is compatible with pandas 3 and no numpy 2 bump is required. Two pins deliberately diverge from uv.lock, documented in a comment block at the top of the file: * `rasterio==1.3.10` (uv.lock: 1.3.8) -- nearest solvable conda-forge build in the same minor series. It resolves against libgdal 3.9.1 / pyogrio 0.9.0, far closer to the uv stack than the 2023-era libgdal 3.7.3 / pyogrio 0.7.2 that pinning 1.3.8 would force (and which is only reachable at all by dropping the `matplotlib` metapackage for `matplotlib-base`). * `tsam==2.3.6` exactly, rather than pyproject's `>=2.3.6`, for the pandas-clobbering reason above. 2.3.6 is what uv.lock resolves to. Other changes bring the file back in line with pyproject: `dask` pinned to 2024.12.0 (was `>=2023.7.0`), `pulp==2.7.0` added, `highspy>=1.9.0` and `dill>=0.3.9` bounded, and the 2023-era ipython/jupyter/notebook/ ipykernel/tqdm pins relaxed since pyproject does not pin them. Verified: micromamba 2.9.0 dry-run solve succeeds for linux-64 (486 pkgs) and osx-arm64 (430 pkgs); a full non-dry-run create on osx-64 succeeds and imports pypsa, pandas, geopandas, rasterio, tables, atlite and linopy, with pandas still at the conda-built 3.0.5 after the pip step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ktehranchi
added a commit
that referenced
this pull request
Aug 31, 2026
Adapts the California/SERVM feature set to the auto-loaded layered base and schema validation from #794: - schema: admit our config surface in the closed sections — run.benchmark_cpuc / benchmark_cpuc_horizons; electricity.honor_planned_retirements, remote_contracted_resources, demand.profile enum += servm, demand.scenario.servm_weather_years; conventional.ambient_derate. - config.california.yaml and config.test.california.yaml rewritten as SPARSE OVERLAYS (programmatic deep-diff against the merged base); the test overlay drops its now-redundant solving/costs/land-access blocks and its stale pre-fix clustering strategies, inheriting the corrected defaults (start_up_cost: sum, capacity-weighted ramps). - policy_constraints paths follow the tracked repo_data/config/ layout (solve rules' interface_limits input, configtables, conftest, docs). - test_dag_dryrun: their repo_data paths + our California cases (incl. county-mode override) as a union. - configtables: their reference-default land-access row (ordering now consistent); our un-deadified transmission_interface_limits description kept, path-updated. Verified: both CA overlays parse against the schema (35-job / 26-job data_model DAGs); fast tier 350 passed, docs 41 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes # (n/a — unblocks e2e-tests, surfaced by re-enabling the CI workflow; see #794).
Changes proposed in this Pull Request
workflow/envs/environment.yamlhas been unsolvable on conda-forge since the PyPSA v1 migration (#778) bumpedpandas==3.0.5/geopandas==1.1.4but left the rest of the pin set at 2023 vintages — CI's e2e job and every mamba-based setup die at env creation. This was four separate conflicts, not one:pytables==3.9.1andlxml==4.9.3violate pandas 3.0.5's run-constraints (pytables >=3.10.1,lxml >=5.3.0). lxml is what actually eliminated the only py311 pandas build — the solver'spython==3.11.9complaint was a symptom, not a cause.rasterio==1.3.8's exactproj/libgdal 3.7.xpins are mutually exclusive with the Qt6 stack thematplotlibmetapackage pulls in.tsamin thepip:block now resolves to 4.0.0, which capspandas<=3.0.3— so pip has been uninstalling the conda-built pandas 3.0.5 and installing 3.0.3 in every successfully created env since tsam 4.0.0 shipped. Now pinnedtsam==2.3.6(uv.lock's resolution). If your env predates this, recreate it — it likely carries pandas 3.0.3.Re-pins:
pytables==3.10.2(matches pyproject),lxml>=5.3.0,rasterio==1.3.10,python>=3.11,<3.12(mirrorsrequires-python),dask==2024.12.0(converges on pyproject),pulp==2.7.0added (pyproject-pinned, was missing), jupyter tooling unpinned (not pyproject-governed),highspy/dillmirror pyproject bounds. The scientific stack is otherwise untouched: pypsa 1.3.0, linopy 0.9.1, pandas 3.0.5, xarray 2026.7.0, geopandas 1.1.4, atlite 0.3.0, numpy 1.26.0 all verified still solvable — the feared numpy tension doesn't exist (conda-forge pandas 3.0.5 acceptsnumpy >=1.26,<3).Two divergences from uv.lock, documented in the file's header comment:
rasterio==1.3.10vs uv.lock's 1.3.8. Keeping 1.3.8 is possible viamatplotlib-base, but drags in a 2023-era GIS stack (libgdal 3.7.3, pyogrio 0.7.2) under a 2026 pandas; the one-patch bump lands on libgdal 3.9.1/pyogrio 0.9.0 — smaller net drift from the uv stack.tsam==2.3.6exact vs pyproject's>=2.3.6— mirroring the operator is precisely what triggers the silent pandas downgrade (a conda/pip-specific hazard; uv would error instead).Testing
CONDA_OVERRIDE_GLIBC=2.28, micromamba 2.9.0 matching CI's setup-micromamba): pass (486 pkgs).pre-commiton the file: pass.Checklist
pre-commitandpytest -m fastpass locally. (fast tier unaffected — single yaml file change)workflow/envs/environment.yaml.workflow/repo_data/config/config.default.yaml. (n/a)docs/source/configtables/*.csv. (n/a)🤖 Generated with Claude Code