Skip to content

PEP 825: Various clarifications #4969

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
konstin wants to merge 18 commits into
base: main
Choose a base branch
from
+171 −79
Open

PEP 825: Various clarifications #4969

Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
510c525
PEP 825: reword ordering to avoid introducing new terms
mgorny Apr 23, 2026
1fd013b
PEP 825: add a note for installing from a source with no index-level …
mgorny Apr 30, 2026
58778b0
PEP 825: clarify that the feature and value lists will be ordered
mgorny May 1, 2026
db4fe20
PEP 825: extend the rationale further
mgorny May 4, 2026
f5a98eb
PEP 825: Extend the rationale for index support scenarios
mgorny May 8, 2026
7a4f9a9
PEP 825: add handling multiple sources to "Open Issues"
mgorny May 8, 2026
68f49fc
PEP 825: clarify index-level metadata in simple repository API
mgorny May 9, 2026
c4ccf6c
PEP 825: mention replacing tags entirely in "Rejected Ideas"
mgorny May 10, 2026
19acda7
Fix rebase
konstin May 11, 2026
bea8e82
Fix dead reference
konstin May 11, 2026
93f883f
Merge pull request #56 from wheelnext/konsti/open-issues-out-of-scope
konstin May 11, 2026
9d3bcf7
Update Change History (11-May-2026)
konstin May 11, 2026
51187f6
Merge pull request #57 from wheelnext/konsti/11-may-2026
konstin May 11, 2026
d9f0f44
Update peps/pep-0825.rst
konstin May 11, 2026
2dbbf45
PEP 825: clarify that we're talking about optional metadata attrs
mgorny May 12, 2026
69bf1b2
PEP 825: remove stray "that"
mgorny May 12, 2026
51e47c9
PEP 825: clarify providing namespace ordering
mgorny May 12, 2026
fce4d46
PEP 825: undefined → implementation-defined
mgorny May 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
250 changes: 171 additions & 79 deletions peps/pep-0825.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -308,11 +308,17 @@ there MUST exist a corresponding ``{name}-{version}-variants.json``
file. The ``{name}`` and ``{version}`` placeholders correspond to the
package name and version, normalized according to the same rules as
wheel files, as found in the :ref:`packaging:wheel-file-name-spec` of
the Binary Distribution Format specification. The exact URL where the
file is hosted is insignificant, but a link to it MUST be present on all
index pages where the variant wheels are linked. It is presented in the
same simple repository format as source distribution and wheel links in
the index, including an (OPTIONAL) hash.
the Binary Distribution Format specification.

The exact URL where the file is hosted is insignificant, but it MUST
be provided in all the responses where the variant wheels are included.
It should follow the rules for files in the
:ref:`packaging:simple-repository-api`, except that the optional
metadata attributes served by the index (such as ``core-metadata``,
``dist-info-metadata``, ``requires-python`` or ``yanked``) are not
meaningful for that file. Indexes MAY publish or skip these attributes,
as long as the values do not prevent correct operation. Tools MAY either
use or ignore these values.

This file uses the same structure as `variant metadata`_, except that
the ``variants`` object MUST list all variants available on the package
Expand Down Expand Up @@ -379,17 +385,20 @@ like:
Variant ordering
----------------

To determine which variant wheel to install when multiple wheels are
compatible, variants MUST be totally ordered by their variant
properties.
This specification defines an ordering between different wheels based on
the presence of variant metadata.

For the purpose of ordering, variant properties are grouped into
features, and features into namespaces. For every namespace, the tool
MUST obtain an ordered list of compatible features, and for every
feature, an ordered list of compatible values. The method of obtaining
these lists will be defined in a subsequent PEP.

The default ordering MUST be performed equivalent to the following
MUST obtain a list of compatible features, and for every feature, a list
of compatible values. The method of obtaining these lists will be
defined in a subsequent PEP. The items in these lists will be provided
Copy link
Copy Markdown
Member

@pfmoore pfmoore May 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"The method of obtaining these lists is not defined (and hence is tool-specific)."

We should avoid making this PEP explicitly dependent on "future PEPs", as if we do so it's impossible to approve this PEP on its own merits.

Copy link
Copy Markdown
Contributor Author

@konstin konstin May 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really see that contradiction, I think we have to live with some of that tension if we want to avoid having one massive PEP 817. There's other PEPs too which specifically leave gaps for future specifications, and I don't see that as fundamentally different. If we say it's tool-specific, that would be rather misleading, as we don't want to open up the namespace to everyone beyond experimentation, we have the expectation that in the end we'll have a compatibility standard where everyone speak the same language.

Copy link
Copy Markdown
Member

@pfmoore pfmoore May 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we have to live with some of that tension if we want to avoid having one massive PEP 817.

Agreed (although we have to accept that if the tension is too great, a single PEP 817 may be a better approach). The key here is that we have a way to implement PEP 825 in the absence of those other PEPs. If we can't start implementation work once PEP 825 is (provisionally) accepted, there's no point in accepting it independently. So regardless of what you might prefer, anything not specified in PEP 825 will be tool-defined. When the follow-up PEP XXX comes along, people who correctly guessed what it would say will be ready, and everyone else will have to change their implementation, but we can't know what the answer will be in advance. I'd rather we made it clear that's the situation, rather than having people think they can't start implementation work because parts of the behaviour need a further PEP to specify them.

There's other PEPs too which specifically leave gaps for future specifications

Maybe I'm getting too concerned about this. Can you give me some examples and I'll check how they handle the situation?

Copy link
Copy Markdown
Contributor

@mgorny mgorny May 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The whole point is that you can start the implementation (provided that you're ready to take the risk that the details will change). We say "subsequent PEP" in very specific places, telling you to leave specific gaps. Like "you can implement most of the sort algorithm, just leave the gap for functions that will provide these lists"; even the example below literally provides an implementation with gaps for these functions.

I don't think "guessing" really works here. I suppose you can infer what the implementation will be from PEP 817, or you can do your own thing. Whether this makes you PEP 825 compliant is unclear to me, since the PEP by design is part of the larger series, and therefore full compliance implies implementing all of them. A tool that implements PEP 825 but then diverges from subsequent PEPs does not really implement "variant wheels"; it implements a custom solution that is partially based on "variant wheels" but it is definitely not compliant with the design as a whole, and therefore it is not guaranteed to be interoperable.

in specific order that will impact variant wheel ordering.

The compatible wheels corresponding to a particular combination of
package name, version and build number MUST be grouped by their variant
label, and a separate group of non-variant wheels MUST be formed. The
groups of variant wheels MUST then be ordered according to the following
algorithm:

1. Construct the ordered list of namespaces by copying the value of the
Expand All @@ -402,9 +411,10 @@ algorithm:
value of the respective ``default-priorities.feature.{namespace}``
key.
Comment thread
konstin marked this conversation as resolved.

ii. Obtain the compatible feature names, in order. For every feature
name that is not present in the constructed list, append it to
the end.
ii. Take the ordered list of compatible feature names obtained
previously and iterate over it, in order. For every feature name
that is not present in the constructed list, append it to the
end.

After this step, a list of ordered feature names is available for
every namespace. This is ``feature_order`` in the example.
Expand All @@ -415,48 +425,49 @@ algorithm:
of the respective
``default-priorities.property.{namespace}.{feature_name}`` key.
Comment thread
konstin marked this conversation as resolved.

ii. Obtain the compatible feature values, in order. For every value
that is not present in the constructed list, append it to the
end.
ii. Take the ordered list of compatible feature values obtained
previously and iterate over it, in order. For every value that is
not present in the constructed list, append it to the end.

After this step, a list of ordered property values is available for
every feature. This is ``value_order`` in the example.

4. For every compatible variant, determine the most preferred value
corresponding to every feature in that variant. This is done by
finding among the values present in the variant properties the one
that has the lowest position in the ordered property value list.
After this step, a list of features along with their best values
is available for every variant. This is done in the
``Variant.best_value_properties()`` method in the example.
4. For every group, determine the most preferred value corresponding to
Copy link
Copy Markdown
Member

@pfmoore pfmoore May 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand this step. A group, as I understand it, is a set of wheels with the same variant label. So each wheel in a group has the same value for every property. So how can there be a "most preferred value" when there's only one value?

I suspect I'm misunderstanding here because the terminology still isn't clear to me. But I'm more concerned with making sure that the reader can follow what's being described than I am arguing about terms.

Or is this actually about trying to order the different groups, so you're trying to identify the most preferred group based on the values the labels denote?

Copy link
Copy Markdown
Contributor

@mgorny mgorny May 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See "variant properties". A single wheel can have multiple values corresponding to a single feature. You select the most preferred one from them for ordering.

If the wheel has:

nvidia :: sm_arch :: 120_real
nvidia :: sm_arch :: 110_real

you would sort only on:

nvidia :: sm_arch :: 120_real

every variant feature present in the variant properties corresponding
to the group. This is done by finding among the values the one that
has the lowest position in the ordered property value list. After
this step, a list of features along with their best values is
available for every variant. This is done in the
``VariantWheel.best_value_properties()`` method in the example.

5. For every item in the list constructed in the previous step,
construct a sort key that is a 3-tuple consisting of
its namespace, feature name and best feature value indices in the
respective ordered lists. This is done by the ``property_key()``
function in the example.

6. For every compatible variant, sort the list constructed in step 4
using the sort keys constructed in step 5, in ascending order. This
is done by the ``Variant.sorted_properties()`` method in the example.
6. For every group, sort the list constructed in step 4 using the sort
keys constructed in step 5, in ascending order. This is done by the
``VariantWheel.sorted_properties()`` method in the example.

7. To order variants, compare their sorted lists from step 6. If the
sort keys at the first position are different, the variant with the
7. To order groups, compare their sorted lists from step 6. If the
sort keys at the first position are different, the group with the
lower key is sorted earlier. If they are the same, compare the keys
at the second position, and so on, until either a tie-breaker is
found or the list in one of the variants is exhausted. In the latter
case, the variant with more keys is sorted earlier. As a fallback,
if both variants have the same number of keys, they are ordered
lexically by their variant label, ascending. This is done by the
found or the list in one of the groups is exhausted. In the latter
case, the group with more keys is sorted earlier. As a fallback,
if both groups have the same number of keys, they are ordered
lexically by the variant label, ascending. This is done by the
ultimate step of the example algorithm, with the comparison function
being implemented as ``Variant.__lt__()``.
being implemented as ``VariantWheel.__lt__()``.

After this process, the variant wheels are sorted from the most
preferred to the least preferred. The algorithm sorts the null variant
after all the other variants. The non-variant wheel MUST be ordered
after the null variant. Multiple wheels with the same variant property
set (and multiple non-variant wheels) MUST then be ordered according to
their platform compatibility tags.
The algorithm sorts the group of null variant wheels last, as they
feature no variant properties. The group of non-variant wheels MUST be
placed after all the other groups.

Within every group, the wheels MUST then be ordered according to their
platform compatibility tags. After this process, the variant wheels are
sorted from the most preferred to the least preferred.

The tools MAY provide options to override the default ordering, for
example by specifying a preference for specific namespaces, features
Expand All @@ -465,7 +476,8 @@ variants, or to select a particular variant.

Alternatively, the sort algorithm for variant wheels could be described
using the following pseudocode. For simplicity, this code does not
account for non-variant wheels or tags.
account for non-variant wheels or the subsequent ordering by platform
compatibility tags.

.. code:: python

Expand Down Expand Up @@ -530,7 +542,7 @@ account for non-variant wheels or tags.
)


class Variant:
class VariantWheel:
"""Example class exposing properties of a variant wheel"""

label: str
Expand Down Expand Up @@ -571,13 +583,13 @@ account for non-variant wheels or tags.
return self.label < other.label


# A list of variants to sort.
variants: list[Variant] = [...]
# A list of variant wheels to sort.
variant_wheels: list[VariantWheel] = [...]


# 7. Order variants by comparing their sorted properties
# (see Variant.__lt__())
variants.sort()
# 7. Order variant wheels by comparing their sorted properties
# (see VariantWheel.__lt__())
variant_wheels.sort()


Environment markers
Expand Down Expand Up @@ -737,6 +749,10 @@ Note that steps 4. through 8. are introduced specifically for variant
wheels. The remaining steps correspond to the current installer
behavior.

When installing from a source that does not provide an `index-level
metadata`_, the same algorithm can be used, except that the variant
metadata needs to be read directly from the wheels.
Copy link
Copy Markdown
Member

@pfmoore pfmoore May 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't feel like it's true. If you have 2 wheels served from a local directory, with different variant.json files in the wheels, doesn't that put you in the same situation as if you had multiple indexes (which, as I note, the algorithm doesn't cover)?

Copy link
Copy Markdown
Contributor

@mgorny mgorny May 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a single source, so the same rules of consistency as in "index-level metadata" apply, and the same merging algorithm described there.



Installing a local wheel
''''''''''''''''''''''''
Expand Down Expand Up @@ -786,28 +802,68 @@ To generate the ``{name}-{version}-variants.json`` file:
Rationale
=========

This PEP is part of a larger variant wheel design that was originally
proposed as :pep:`817`. However, due to its complexity, we decided to
split it into smaller parts that build one upon another. This PEP is the
first in the series, providing foundations including the file format
along with necessary metadata, index support and basic tool algorithms.
Aspects such as providing actual variant properties or building wheels
are deferred into subsequent PEPs.

Variant wheels use structured `variant properties`_ to express
multidimensional wheel compatibility matrices. For example, it permits
expressing that a single variant requires certain CPU and GPU features
independently. It can express both AND-style dependencies (such as
different CPU instruction sets) and OR-style dependencies (such as
different GPUs supported by a single package).
multidimensional wheel compatibility matrices. Properties are organized
in namespaces that can be defined and governed independently. The
key-value structure makes the properties more flexible: adding a new
compatibility axis can be done by adding a new key. It can support both
AND-style dependencies (for example, a CPU plugin could define multiple
keys corresponding to different instructions sets, all of which are used
in the package and therefore must be supported) and OR-style
dependencies (for example, a GPU plugin can define a single key listing
multiple GPU types, indicating that all of them are supported by the
package, and therefore the users needs to own only one of them).

The specification does not impose any formal limits on the number of
properties expressed, and specifically accounts for the possibility of
property sets being very long (for example, a long list of GPUs or CPU
extension sets). To avoid wheel filenames becoming very long, the
property lists are stored inside the wheel and mapped to a short label
that is intended to be human-readable.

To facilitate variant selection while installing from remote index,
the variant metadata is mirrored in a JSON file published on the index.
This enables installers to obtain variant property mapping without
having to fetch individual wheels.

Since JSON format does not feature a set type, sets are represented as
sorted lists. Sorting ensures that tools can safely use equality
comparison over dictionaries.
extension sets). To avoid wheel filenames becoming hard to comprehend
because of excess of information and potentially causing technical
issues because of their length, the property lists are stored inside
the wheel and mapped to a short label that is chosen by the package
maintainer and intended to be human-readable.

Wheel filenames alone do not provide sufficient metadata to drive
variant wheel selection. To avoid tools having to fetch the variant
metadata straight from multiple wheel files, the metadata from wheels
for every package version is combined and republished. This metadata is
scoped to a single package version to permit variants changing in the
future version.

The index support aims to account for three scenarios:

1. An index implementation that cannot embed additional metadata as part
of file list responses. For example, this covers installing straight
from a directory listing created by a webserver. To account for this
scenario, index-level metadata is published as a plain JSON file that
can be generated by the package maintainer and placed alongside
wheels.

2. An index implementation that has more complete wheel support but does
not wish to implement full variant wheel support immediately. The
index needs only to permit the user to upload said JSON file. To
account for minimalistic implementation, the specification permits
the index to treat said file similarly to a wheel, including
publishing attributes such as ``yanked``, as long as their values do
not prevent clients from working.

3. An index implementation that implements complete wheel variant
support. Such an index will parse uploaded variant wheels, and
dynamically create the index-level metadata. The JSON file path would
then be treated as an API endpoint rather than an actual file.

Since JSON format does not feature a set type, sets in the metadata are
represented as sorted lists. Sorting ensures reproducibility and makes
it possible to use equality comparison over whole dictionaries without
having to convert specific fields back to sets after deserialization.

The variant ordering algorithm has been proposed with the assumption
that variant properties take precedence over Platform compatibility
Expand All @@ -817,15 +873,25 @@ variant may require a different minimal libc version, in which case the
selection should be driven by the desired CUDA preference rather than
incidental platform tag difference.

While the provision of variant properties is deferred to a future PEP to
keep the specification easier to comprehend, a baseline assumption is
made that the compatible properties will be provided in specific order
corresponding to their preference, much like Platform compatibility tags
conventionally are. The variant metadata provides the ability to
override this order at package level. However, namespaces are unordered
by design (e.g. we will not decide upfront which GPU vendors take
precedence) and therefore they always need to be ordered by the package
maintainer.
While a future PEP will define how variant properties are provided, a
baseline assumption is made that the compatible properties will be
Copy link
Copy Markdown
Member

@pfmoore pfmoore May 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't be an assumption, it should be explicitly required in this PEP. Then this sentence can be reworded something along these lines:

This PEP simply requires that compatible properties are provided in a specific order corresponding to their preference. We do not state how tools will provide this ordered list, but a future PEP is planned which will standardise the mechanism.

Copy link
Copy Markdown
Contributor

@mgorny mgorny May 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't the rationale section be describing why were the specific decisions made in the specification part, rather than stating what is required (I.e. effectively repeating specification)? I dare say the "context" is the whole point of having a rationale in the first place.

Copy link
Copy Markdown
Member

@pfmoore pfmoore May 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, if you want to move this "baseline assumption" into the specification section as a requirement, that works for me.

Copy link
Copy Markdown
Contributor

@mgorny mgorny May 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But it literally does that, in the fragment you've commented on above:

The items in these lists will be provided in specific order that will impact variant wheel ordering.

We're merely trying to provide a bit more context of why this requirement is baked in there.

provided in specific order corresponding to their preference. This makes
it possible to use a generic sorting algorithm, and later define
properties as data without having to change the algorithm.

A future PEP will define how the ordering for features and values is
provided. However, namespaces are governed independently and considered
Copy link
Copy Markdown
Member

@pfmoore pfmoore May 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here. You need to say what this PEP requires, so that tools can (in principle) implement their own mechanism for providing an ordering. The fact that we'll standardise that mechanism in a future PEP isn't important here, except as context.

on equal footing, and therefore there will be no standard ordering for
them. Instead, the ordering of namespaces will be explicitly stated in
the variants metadata, which in turn will be provided by the package
maintainer as part of the build process. For completeness, it will also
be possible to provide overrides for the ordering of features and values
via the same mechanism.

In the vast majority of real use cases, ordering based on properties
will suffice. However, in a pathological case two different variant
wheels may end up with equal sort keys. To provide reproducible results
in this case, fallback sorting on variant label is performed.

A concept of null variant is introduced that is distinct from
non-variant wheels to facilitate a transition period. This variant is
Expand Down Expand Up @@ -938,7 +1004,7 @@ Reference Implementation
The `variantlib <https://github.com/wheelnext/variantlib>`__ project
contains a reference implementation of a complete variant wheel
solution. It is compliant with this PEP, but also goes beyond it,
providing example solutions to `open issues`_.
providing example solutions to some of the deferred items.

A client for installing variant wheels is implemented in a
`uv branch <https://github.com/astral-sh/uv/pull/12203>`__.
Expand Down Expand Up @@ -982,15 +1048,35 @@ would be incorrectly deemed compatible because of the
``manylinux_2_27_x86_64`` part.


Open Issues
===========
Replacing Platform compatibility tags entirely
----------------------------------------------

Technically, it would be entirely possible to convey the information
currently passed via the Platform compatibility tags via variant
properties, and remove these explicit tags from the filename. However,
we decided not to pursue this and instead preserve the existing
filenames for wheels that do not need additional variants, as we do not
believe that the effort required to update all the existing workflows
justifies the benefit of more compact, and slightly more consistent
naming.


Out of scope
------------

The following problems are deferred to subsequent PEPs:
The following problems are deferred to subsequent PEPs in the series:

- governance of variant namespaces
- determining which variant properties are compatible with the system
- building variant wheels

In addition to that, the following matters are left
implementation-defined:

- Selecting variant wheels from multiple sources. Currently, there is no
standard defined behavior for regular wheels, nor consensus across
different packaging tools on how to handle that.


Acknowledgements
================
Expand Down Expand Up @@ -1036,6 +1122,12 @@ Change History
- Changed ``pylock.toml`` integration to inline variant metadata
rather than storing a URL and a hash.

- 11-May-2026

- Added replacing platform compatibility tags entirely to rejected
ideas.
- Clarified interpretation of sorting algorithm and index support.


Appendices
==========
Expand Down
Loading