Skip to content

cpufeatures: fix x86 CPUID and XSAVE state checks - #1528

Open
eslerm wants to merge 5 commits into
RustCrypto:masterfrom
eslerm:cpufeatures-vaes-xsave-state
Open

cpufeatures: fix x86 CPUID and XSAVE state checks#1528
eslerm wants to merge 5 commits into
RustCrypto:masterfrom
eslerm:cpufeatures-vaes-xsave-state

Conversation

@eslerm

@eslerm eslerm commented Sep 2, 2026

Copy link
Copy Markdown

Closes #1510

1. Guard CPUID queries by max basic leaf. CPUID(1) and CPUID(7, N) were
queried unconditionally. Per the SDM a leaf above CPUID(0).EAX returns the
highest supported leaf's data, which check! then reads as feature bits. The
leaf 7 rows carrying no XSAVE requirement are where this bites, sha and adx
among them, since nothing masks a stale bit once it is read.

Leaf 1 and both leaf 7 subleaves are zeroed when CPUID(0).EAX does not reach
them. Subleaf 1, which the sha512/sm3/sm4 rows read, is additionally gated
on subleaf 0 reporting a max subleaf of at least 1. std_detect gates both the
same way:

https://docs.rs/crate/std_detect/latest/source/src/detect/os/x86.rs

2. Match XSAVE tags to the encodings they gate. None of the
three is an AVX-512 feature. ymm for the VEX-256 forms, none for the
legacy-SSE form of gfni, as with pclmulqdq. gfni needs "" rather than
"ymm": Tremont Atom has GFNI and no AVX at all. sm3 had the opposite error,
"xmm" where its VEX.128 encoding needs ymm; fixed the same way the avx row
was in #1511, leaving no "xmm" rows. Requires the max-basic-leaf
guard above: without it a part whose max basic leaf is below 7 aliases the leaf
7 query, and the relaxed tags no longer mask the aliased bits.

3. Make the guard and the tags unit-testable. Leaf collection moves into
__collect_leaves, parameterized over the CPUID sources. No CI runner reports a
max basic leaf below 7, so the guard is otherwise unreachable in tests.
mod x86 becomes pub mod x86, doc-hidden, since the macro now reaches that
function through $crate::x86:: when expanded downstream; aarch64 and
loongarch64 are already public on the same terms.

4. Require the AVX bit for vaes/vpclmulqdq. The ymm tag checks that the
OS enabled YMM state, not that the CPU reports AVX; fma and avx2 already
conjoin both. sha512 and sm4 are also VEX-encoded and still do not conjoin
it; left alone here, since they were added as a set and are worth their
own change.

5. Make an unknown XSAVE tag a compile error. The tag column was matched at
runtime with a _ => true arm, so a mistyped tag did not fail to build. It
silently removed the state check for that feature. Feature names were already
protected this way; the tag column was not.

vaes/vpclmulqdq no longer imply AVX-512 state. A consumer dispatching an
EVEX-512 form must gate on avx512f separately.

Downstream

aes gates its VAES-256 backend on the bare vaes token, so this changes which
backend it selects on non-AVX-512 parts. That backend batches 30 blocks and does
not override encrypt_tail_blocks, so shorter buffers fall through to cipher's
one-block-at-a-time default.

Aes128::encrypt_blocks, cycles/byte: below 480 bytes VAES-256 runs up to 34%
slower than AES-NI; at and above it, 4 to 5 times faster. The crossover is
exactly the batch width, and the tail cost recurs at every N mod 30 (a
45-block buffer is one full batch plus a 15-block tail, and lands at 1.8x rather
than 5x).

Two limits on those numbers. They were taken on Zen 5, which is not one of
the parts this change affects; it was the x86 hardware I had, and a Gracemont or
Zen 3 result may differ, particularly the size of the win. And they force backend
selection rather than masking CPUID, so they measure the two backends, not the
detection.

The tail path is a pre-existing aes gap. An encrypt_tail_blocks override
there would bound it at parity.

Testing

cargo test -p cpufeatures passes on x86_64-unknown-linux-gnu; on non-x86
targets the module and its tests are cfg'd out. Three of the seven tests fail
against the pre-fix code.

Detection was checked under masked CPUID; no test in this PR produces these.
Rows 1, 3 and 4 are qemu-x86_64 (qemu-user, TCG) with the named -cpu mask;
row 2 is a KVM guest.

configuration before after
-cpu Icelake-Server,-avx512f vaes=false vaes=true
KVM -cpu Snowridge (GFNI, no AVX) gfni=false gfni=true
-cpu Icelake-Server,level=1 vaes=false vaes=false
-cpu Skylake-Client vaes=false vaes=false

The level=1 row is why the guard comes first. Both shipped states are correct
there; a fourth state this PR never produces, relaxation applied with the guard
absent, reports vaes=true on a configuration with no leaf 7 at all.

vpclmulqdq was not measured on any configuration, because QEMU's TCG does not
expose its CPUID bit, and no test covers its tag. Its change rests on the same
encoding argument as vaes.

@eslerm eslerm closed this Sep 2, 2026
Closes RustCrypto#1510

CPUID(1) and CPUID(7, N) were queried unconditionally. Per the Intel
SDM a leaf above CPUID(0).EAX does not read as zero: the CPU returns
the highest supported basic leaf's data, which `check!` then reads as
feature bits. The leaf 7 rows carrying no XSAVE requirement are where
this bites, `sha` and `adx` among them, since nothing masks a stale
bit once it is read.

Leaf 7 subleaf 1 is gated a second time on subleaf 0's EAX, which
reports the highest valid subleaf.

Follows `std_detect`:
https://docs.rs/crate/std_detect/latest/source/src/detect/os/x86.rs
Four rows named a state their instructions do not use.

`gfni`, `vaes` and `vpclmulqdq` required `zmm`, so detection failed on
parts that have them without AVX-512. None of the three is an AVX-512
feature; Tremont Atom cores, for instance, have `gfni` and no AVX at
all. They now require `ymm` for the VEX-256 forms of `vaes` and
`vpclmulqdq`, and no state for the legacy-SSE form of `gfni`, as with
`pclmulqdq`.

`sm3` required only `xmm`, which is the opposite error. Its
instructions are VEX.128-encoded, and any VEX encoding faults unless
XCR0 bits 1 and 2 are both set, while `"xmm"` checks only bit 1. It
now requires `ymm`, the same fix the `avx` row took in RustCrypto#1511. No
`"xmm"` rows remain.

Each row now certifies its own CPUID bit and the XSAVE state the
non-EVEX-512 encodings use, and nothing else. `zmm` previously acted
as a de facto proxy for a wider set, since no part enables that state
without AVX-512 and therefore without AVX2. It no longer does, so a
consumer must gate separately on whatever else the code it dispatches
needs: `avx512f` for an EVEX-512 form, and for `vaes` the `avx2` and
`aes` that rustc's target feature closure enables inside a `vaes`
region. `vpclmulqdq` likewise implies `avx` and `pclmulqdq`.

Correct behaviour also requires the max-basic-leaf guard on the CPUID
queries: without it, a part whose max basic leaf is below 7 aliases
the leaf 7 query, and the relaxed tags no longer mask the aliased
bits.
Extract leaf collection into `__collect_leaves`, parameterized over
the CPUID sources so the max basic leaf guard can be driven with
synthetic data. No CI runner reports a max basic leaf below 7, so the
guard is otherwise unreachable under `cargo test` and a later
refactor could invert it unnoticed.

The XSAVE tag test expands `check!` inside the crate for the first
time, which brings `__xgetbv!`'s `unsafe` block under
`clippy::undocumented_unsafe_blocks`; hence the SAFETY comment.

`mod x86` becomes `pub mod x86`, doc-hidden, matching `aarch64` and
`loongarch64`, since the macro now reaches `__collect_leaves` through
`$crate::x86::` when expanded downstream.
The `ymm` tag checks that the OS enabled YMM state, not that the CPU
reports AVX. `fma` and `avx2` conjoin CPUID.1:ECX[28] alongside their
`ymm` tag; do the same here, so a CPUID configuration advertising
VAES while masking AVX cannot satisfy the token.

No shipping part is known to report VAES without AVX, so this is a
no-op on real hardware. It matters under a hypervisor or emulator
that masks the two independently.

`sha512` and `sm4` are VEX-encoded and still do not conjoin the AVX
bit. Left alone here, since they were added as a set and are worth
their own change.
The tag column was matched at runtime with a `_ => true` arm, so a
mistyped tag did not fail to build. It silently dropped the extended
state check for that feature, and `check!` would then report the
feature present on a machine whose OS had not enabled the state its
instructions need, faulting at the first one.

`__xsave_state!` has no catch-all arm, and a `const _` per row runs
every tag past the same list where the table is defined. The second
is what puts the error here, rather than in whichever downstream
crate first happens to check that feature.
@eslerm eslerm reopened this Sep 2, 2026
@eslerm
eslerm force-pushed the cpufeatures-vaes-xsave-state branch from 355fd8e to c11948b Compare September 2, 2026 07:42
@eslerm eslerm changed the title cpufeatures: relax vaes/vpclmulqdq/gfni XSAVE checks; guard leaf 7 by max basic leaf cpufeatures: fix x86 CPUID and XSAVE state checks Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

x86: guard CPUID leaf 7 by the maximum basic leaf; AVX detection may need XMM+YMM XCR0 state

1 participant