feat: Add support for SOURCE packages by toprinse · Pull Request #244 · Quantco/pixi-pack

Thibaut OPRINSEN (toprinse) · 2025-11-24T14:27:24Z

Motivation

This PR adds support for building packages from source, a feature that is currently missing. See issue #80 for background and motivation.

When a local package is found in the pixi.lock of the main package, start a pixi-build command to build the local dependencies and add them as injected packages in the environment.tar

Changes

Introduces a pixi-build command to automatically build local packages from the main pixi.lock
Adding integration test to demonstrate the the added code is working as expected.

Closes #80

feat: try to add intergation tests

fix: remove unused fixture fix: fix typo fix: fix typo fix: fix typo fix: fix test on arm-64, osx and win builds fix: fix test on arm-64, osx and win builds fix: add valid checksum for other architectures fix: fix test using correct checksum fix: fix typo error fix: fix typo

fix: fix typo fix: fix typo

Pavel Zwerschke (pavelzw)

Looks very solid already, thanks!
I have a couple of comments

fix: fix typo for ci precommit fix: fix typo for ci precommit fix: fix typo for ci precommit

Thibaut OPRINSEN (toprinse) · 2025-12-01T12:06:15Z

Thanks Pavel Zwerschke (@pavelzw) ! So I made the requested changes, could you review the new version and tell me if there are some changes that are not relevant or things I could have done better ?

Pavel Zwerschke (pavelzw) · 2025-12-01T17:29:36Z

+        .arg(output_dir)
+        .arg("--build-platform")
+        .arg(platform)
+        .current_dir(manifest_dir);


with things like PIXI_PROJECT_ROOT (set by pixi shell) i think it's safer to explicitly use pixi build --path

also, another thing i noticed:

❯ cd examples/local-build/ ❯ cargo run --bin pixi-pack -- Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.99s Running `/Users/pavel/projects/pixi-pack/target/debug/pixi-pack` ⠁ Building local-build-main-pkg-0.1.0-hbf21a9e_0.conda ❌ Build failed with exit code: Some(1) Command: pixi build --output-dir /var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/.tmpZQnitN --build-platform osx-arm64 (in /Users/pavel/projects/pixi-pack/examples) 📤 stderr: Error: × the source directory '/Users/pavel/projects/pixi-pack/examples', does not contain a supported manifest help: Ensure that the source directory contains a valid pixi.toml, pyproject.toml, recipe.yaml, package.xml or mojoproject.toml file. Error: Failed to build package in /Users/pavel/projects/pixi-pack/examples ❯ cargo run --bin pixi-pack -- pixi.toml Compiling pixi-pack v0.7.5 (/Users/pavel/projects/pixi-pack) Finished `dev` profile [unoptimized + debuginfo] target(s) in 8.78s Running `/Users/pavel/projects/pixi-pack/target/debug/pixi-pack pixi.toml` ✅ Built local-build-main-pkg-0.1.0-hbf21a9e_0.conda ✅ Built local-build-local-pkg-0.1.0-hbf21a9e_0.conda ✅ Built local-build-local-pkg-2-0.1.0-hbf21a9e_0.conda ⏳ Downloading 14 packages... 📦 Created pack at /Users/pavel/projects/pixi-pack/examples/local-build/environment.tar with size 7.70 MiB.

something is still wrong with the manifest path...

I updated the base_dir logic to correctly handle manifest paths when building source packages. Previously, when options.manifest_path was a directory (not a file), the code incorrectly used its parent directory as the base, causing build failures with paths like /examples/./pixi.toml. Now it properly uses the manifest directory itself as the base when it's a directory. Also added canonicalize() to resolve relative paths (like ./) and ensure the manifest file exists before attempting to build.

oprinsen@lappc-p1138 local-build ⌥ (add_source_pkg_handling) cargo run --bin pixi-pack -- Compiling pixi-pack v0.7.5 (/home/oprinsen/pixi-pack) Finished `dev` profile [unoptimized + debuginfo] target(s) in 3.62s Running `/home/oprinsen/pixi-pack/target/debug/pixi-pack` DEBUG: options.manifest_path = /home/oprinsen/pixi-pack/examples/local-build DEBUG: source_data.location = Path(Unix(Utf8PathBuf { _encoding: "unix", inner: "." })) DEBUG: base_dir = /home/oprinsen/pixi-pack/examples/local-build ✅ Built local-build-main-pkg-0.1.0-hbf21a9e_0.conda DEBUG: options.manifest_path = /home/oprinsen/pixi-pack/examples/local-build DEBUG: source_data.location = Path(Unix(Utf8PathBuf { _encoding: "unix", inner: "local-pkg" })) DEBUG: base_dir = /home/oprinsen/pixi-pack/examples/local-build ✅ Built local-build-local-pkg-0.1.0-hbf21a9e_0.conda DEBUG: options.manifest_path = /home/oprinsen/pixi-pack/examples/local-build DEBUG: source_data.location = Path(Unix(Utf8PathBuf { _encoding: "unix", inner: "local-pkg2" })) DEBUG: base_dir = /home/oprinsen/pixi-pack/examples/local-build ✅ Built local-build-local-pkg-2-0.1.0-hbf21a9e_0.conda ⏳ Downloading 21 packages... 📦 Created pack at /home/oprinsen/pixi-pack/examples/local-build/environment.tar with size 15.28 MiB. oprinsen@lappc-p1138 local-build ⌥ (add_source_pkg_handling)

Pavel Zwerschke (pavelzw) · 2025-12-01T17:50:08Z

pushed some adjustments to your branch; after the manifest dir fixes, this is good to merge 👍🏻

fix: fix typo

Thibaut OPRINSEN (toprinse) · 2025-12-08T16:05:53Z

Hello Pavel Zwerschke (@pavelzw), I saw that the tests have failed after merging main into my branch. Is the issue linked to the hash or the build_version of the generated packages which has changed after merging the two branches ? I don't really know how to make the test reliable, as we hardcoded the hash and version in the tests, and I don't really know how to do it in another way. If I can do anything, just let me know !

Pavel Zwerschke (pavelzw) · 2025-12-08T18:46:20Z

yeah this looks really weird to me. sounds like a bug in pixi tbh.
i will create a minimal reproducer and see what the pixi people think of this one

Thibaut OPRINSEN (toprinse) · 2025-12-18T08:19:14Z

Hello Pavel Zwerschke (@pavelzw).

So I looked into the issue of the regenerated hashes and it seems that recently the cmake build backend used in the tests has been updated and the hash generation might have been impacted. I'm not sure about how it impacted the tests and it's still unclear to me but for now I added the new generated hashes using the cmake build backend 0.3.5 (maybe it could be intersting to pin the build backend version in the test to avoid new strange behaviors).

Another idea could be to use match patterns instead of the exact version of the package containing the hashes. Instead of looking for conda-meta/local-build-main-pkg-0.1.0-hb0f4dca_0.json we could also use something like starts_with("local-build-main-pkg-0.1.0-") inside the conda-meta directory.

I also updated the pixi version to use 0.61.0 so the API of pixi build command has changed a bit in pack.rs

Pavel Zwerschke (pavelzw) · 2025-12-18T08:34:39Z

Sorry for not coming back to you, lost track of this pr. One thing I'm wondering is why the build string from the pixi.lock didn't match the build string of the built package 🤔 theoretically the lockfile should always match the package exactly and thus represent the correct build string already? Maybe I'm missing something, will investigate later

In general, i think pinning the build backend makes sense in this case 👍🏻

I think it would be nice if we even persisted the build backends and host+build environments used in pixi.lock (optionally): prefix-dev/pixi#5161

Thibaut OPRINSEN (toprinse) · 2025-12-18T09:20:00Z

Sorry for not coming back to you, lost track of this pr. One thing I'm wondering is why the build string from the pixi.lock didn't match the build string of the built package 🤔 theoretically the lockfile should always match the package exactly and thus represent the correct build string already? Maybe I'm missing something, will investigate later

In general, i think pinning the build backend makes sense in this case 👍🏻

I think it would be nice if we even persisted the build backends and host+build environments used in pixi.lock (optionally): prefix-dev/pixi#5161

No problem, I was also really busy!

I can pin the build backend version in the tests to the latest version (0.3.5).

Regarding the build string mismatch issue, I'm not sure what approach you'd prefer: should we wait for pixi to provide a clear fix for this issue, or would you prefer to merge the PR as-is and address the build string inconsistency once pixi implements the changes mentioned in the issue you linked?

vincent-pollet · 2026-02-05T13:48:07Z

Hello! May I kindly ask what is the status about this PR? Anything that we could help with?

vincent-pollet · 2026-03-16T17:31:16Z

Pavel Zwerschke (@pavelzw) Sorry to ping you, is there anything we can do to help this get merged?

Pavel Zwerschke (pavelzw) · 2026-03-28T12:22:27Z

we now cleaned up and merged all other PRs and this one is next, sorry to keep you waiting for so long.

pixi lockfile v7 are coming soon-ish (prefix-dev/pixi#5607, conda/rattler#2026) and will lead to the version field not being fully required anymore.
we need to

merge main into this branch and resolve conflicts
see how the new lockfile format affects this PR.

Thibaut OPRINSEN (toprinse) · 2026-03-30T11:56:12Z

Hello Pavel Zwerschke (@pavelzw) .
So I merged the main branch into this one, fixed the conflicts and update tests.
I made a few changes :

I removed the win64 test (see comments in integrations_tests.rs because they were failing because the generated path from pixi build is too long. There is currently no clean solutions to avoid reaching these windows limitations.
Then, I'm not sure why but the test_pixi_pack_source sometimes fail, even I don't add major changes. I then get :

---- test_pixi_pack_source stdout ----
⏳ Downloading 25 packages...

thread 'test_pixi_pack_source' (13156) panicked at tests/integration_test.rs:858:5:
Err(could not download package: error decoding response body)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Maybe you have a better idea about what's happening ?

Pavel Zwerschke (pavelzw) · 2026-04-07T16:09:52Z

Then, I'm not sure why but the test_pixi_pack_source sometimes fail, even I don't add major changes

this looks like a networking issue and the package got dropped. maybe we need to add retries to the Client if we haven't done so already

Pavel Zwerschke (pavelzw) · 2026-05-04T06:23:24Z

Lockfile v7 is now on main (of pixi). This means we should be able to test against it :)

Pavel Zwerschke (pavelzw) · 2026-05-15T07:00:05Z

Added support for lockfile v7 in #306. hopefully this fixes our reproducibility issues

Thibaut OPRINSEN (toprinse) · 2026-05-27T09:23:56Z

Hello Pavel Zwerschke (@pavelzw). Sorry for the late answer. It's nice to have lockfiles v7 available !
Is there still things I can do for this PR ?

Pavel Zwerschke (pavelzw) · 2026-05-27T11:16:53Z

I think we would need to merge main and then check again whether the tests would now be working reliably

…ling

Thibaut OPRINSEN (toprinse) · 2026-05-27T14:43:41Z

Hello Pavel Zwerschke (@pavelzw). So, I merged main into this branch and fixed the breaking changes introduced by the rattler_lock refactor (specifically, CondaSourceData.package_record was removed — replaced by record() returning an Option<&PackageRecord>). The tests appear to be passing now. Still I'm not sure they are working reliably. Seems that the merge did not affect packages names and hashes in the test.

…kg_handling

Pavel Zwerschke (pavelzw) · 2026-07-03T14:50:03Z

+        if !status.success() {
+            anyhow::bail!("Failed to publish package in {}", manifest_dir.display());
+        }


in a follow-up we could stream the logs like pixi install

Pavel Zwerschke (pavelzw) · 2026-07-03T14:52:37Z

+                    UrlOrPath::Url(u) => {
+                        anyhow::bail!("Unexpected URL for local package source: {}", u)
+                    }


this could be useful as well in a follow-up to support things like this:

[workspace] preview = ["pixi-build"] channels = ["conda-forge"] platforms = ["linux-64", "linux-aarch64", "win-64", "osx-64", "osx-arm64"] [dependencies] agent-skill-conda-forge = { git = "https://github.com/pavelzw/skill-forge", rev = "main", subdirectory = "recipes/conda-forge" }

Daniel Elsner (delsner)

Thanks for your work! Sorry that it took us a while to get this through.

Thibaut OPRINSEN (toprinse) requested review from Daniel Elsner (delsner) and Pavel Zwerschke (pavelzw) as code owners November 24, 2025 14:27

Thibaut OPRINSEN (toprinse) changed the title ~~feat : Add support for SOURCE packages~~ feat: Add support for SOURCE packages Nov 24, 2025

github-actions Bot added the enhancement New feature or request label Nov 24, 2025

feat: update pixi-pack to be able to pack packages from source

a532262

feat: try to add intergation tests

Thibaut OPRINSEN (toprinse) force-pushed the add_source_pkg_handling branch from e214536 to faf6c1d Compare November 24, 2025 16:01

Thibaut OPRINSEN (toprinse) force-pushed the add_source_pkg_handling branch from 6abfcac to a9b0110 Compare November 25, 2025 07:49

Daniel Elsner (delsner) requested changes Nov 28, 2025

View reviewed changes

Comment thread src/pack.rs Outdated

Comment thread src/pack.rs Outdated

Comment thread src/pack.rs Outdated

fix: remove recursive search and use packages from lock file

0847e0d

fix: fix typo fix: fix typo

Thibaut OPRINSEN (toprinse) force-pushed the add_source_pkg_handling branch from abf424a to 0847e0d Compare November 29, 2025 13:22

Thibaut OPRINSEN (toprinse) requested a review from Daniel Elsner (delsner) November 29, 2025 13:22

Pavel Zwerschke (pavelzw) reviewed Nov 30, 2025

View reviewed changes

Comment thread src/pack.rs Outdated

fix: fix after review

1377324

fix: fix typo for ci precommit fix: fix typo for ci precommit fix: fix typo for ci precommit

Thibaut OPRINSEN (toprinse) force-pushed the add_source_pkg_handling branch from a8e2dbc to 1377324 Compare December 1, 2025 12:04

Pavel Zwerschke (pavelzw) added 3 commits December 1, 2025 18:42

refactor local-build structure

482bd9e

explicitly drop

8d7d436

reword

4f94186

Pavel Zwerschke (pavelzw) reviewed Dec 1, 2025

View reviewed changes

fix: fix manifest_dir

177e276

fix: fix typo

Thibaut OPRINSEN (toprinse) force-pushed the add_source_pkg_handling branch from 6a65aed to 177e276 Compare December 2, 2025 11:04

Pavel Zwerschke (pavelzw) mentioned this pull request Dec 18, 2025

pixi-build used dependencies (i.e. backend + host and build envs) should be persisted in pixi.lock prefix-dev/pixi#5161

Closed

Thibaut OPRINSEN (toprinse) and others added 2 commits December 18, 2025 11:33

Merge branch 'main' into add_source_pkg_handling

b6b2895

fix: fix cargo lock

0384d5c

merge main into add_source_pkg_handling

e7eac88

feat: add source package handling with cross-platform fixes

eaf607c

Thibaut OPRINSEN (toprinse) force-pushed the add_source_pkg_handling branch from c17b233 to eaf607c Compare March 30, 2026 12:11

Thibaut Oprinsen added 2 commits May 27, 2026 15:26

Merge remote-tracking branch 'upstream/main' into add_source_pkg_hand…

5540214

…ling

fix: fix typo

f26f503

Daniel Elsner (delsner) added 6 commits July 3, 2026 15:48

Merge branch 'main' of github.com:Quantco/pixi-pack into add_source_p…

61c64d1

…kg_handling

Bump pixi

6728477

Adjust implementation for most recent pixi publish

420b29f

Fix

30a922b

nit

e212351

fix?

fb84a3d

Pavel Zwerschke (pavelzw) approved these changes Jul 3, 2026

View reviewed changes

Daniel Elsner (delsner) approved these changes Jul 3, 2026

View reviewed changes

Daniel Elsner (delsner) and others added 3 commits July 3, 2026 17:04

Disable test again

579fd5c

Merge branch 'main' into add_source_pkg_handling

4d703ac

bump

d2ab537

Pavel Zwerschke (pavelzw) merged commit 62cb4f6 into Quantco:main Jul 3, 2026
15 of 19 checks passed

BrewTestBot mentioned this pull request Jul 3, 2026

pixi-pack 0.7.10 Homebrew/homebrew-core#291225

Merged

Uh oh!

Conversation

Thibaut OPRINSEN (toprinse) commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Pavel Zwerschke (pavelzw) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Thibaut OPRINSEN (toprinse) commented Dec 1, 2025

Uh oh!

Uh oh!

Pavel Zwerschke (pavelzw) Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Pavel Zwerschke (pavelzw) Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Thibaut OPRINSEN (toprinse) Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Pavel Zwerschke (pavelzw) commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Thibaut OPRINSEN (toprinse) commented Dec 8, 2025

Uh oh!

Pavel Zwerschke (pavelzw) commented Dec 8, 2025

Uh oh!

Thibaut OPRINSEN (toprinse) commented Dec 18, 2025

Uh oh!

Pavel Zwerschke (pavelzw) commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Thibaut OPRINSEN (toprinse) commented Dec 18, 2025

Uh oh!

vincent-pollet commented Feb 5, 2026

Uh oh!

vincent-pollet commented Mar 16, 2026

Uh oh!

Pavel Zwerschke (pavelzw) commented Mar 28, 2026

Uh oh!

Thibaut OPRINSEN (toprinse) commented Mar 30, 2026

Uh oh!

Pavel Zwerschke (pavelzw) commented Apr 7, 2026

Uh oh!

Pavel Zwerschke (pavelzw) commented May 4, 2026

Uh oh!

Pavel Zwerschke (pavelzw) commented May 15, 2026

Uh oh!

Thibaut OPRINSEN (toprinse) commented May 27, 2026

Uh oh!

Pavel Zwerschke (pavelzw) commented May 27, 2026

Uh oh!

Thibaut OPRINSEN (toprinse) commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Pavel Zwerschke (pavelzw) Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

Pavel Zwerschke (pavelzw) Jul 4, 2026

Choose a reason for hiding this comment

Uh oh!

Thibaut OPRINSEN (toprinse) commented Nov 24, 2025 •

edited

Loading

Pavel Zwerschke (pavelzw) Dec 1, 2025 •

edited

Loading

Thibaut OPRINSEN (toprinse) Dec 2, 2025 •

edited

Loading

Pavel Zwerschke (pavelzw) commented Dec 1, 2025 •

edited

Loading

Pavel Zwerschke (pavelzw) commented Dec 18, 2025 •

edited

Loading

Thibaut OPRINSEN (toprinse) commented May 27, 2026 •

edited

Loading