uclibc targets (and others?) cannot perform libc-test macro expansion due to lack of core lib

[skrap]

During the development of #5046 , the request was made that the bitrotten CI script for armv7-unknown-linux-uclibceabihf be resuscitated and made to work. However, I encountered difficulties with the somewhat-new macro expansion functions within the libc testing infrastructure.

• The current macro expansion code wants to call the following:

RUSTC_BOOTSTRAP=1 rustc -Zunpretty=expanded --edition 2021 /checkout/src/lib.rs --target armv7-unknown-linux-uclibceabihf

but this results in the following error:

macro expansion failed with exit status: 1: error[E0463]: can't find crate for `core`
  |
  = note: the `armv7-unknown-linux-uclibceabihf` target may not be installed
  = help: consider building the standard library from source with `cargo build -Zbuild-std`

error: aborting due to 1 previous error

For more information about this error, try `rustc --explain E0463`

@tgross35 suggested using -Zbuild-std but this is a cargo-ism, not a switch which rustc understands. I also was asked to try running cargo rustc in order to invoke the -Zbuild-std switch, but this isn't really the same as invoking rustc directly, as it wants to build the whole crate. (More precisely, it wants to build exactly one target within the crate.) The macro expansion code wants to build exactly one .rs file. This capability is not available from cargo rustc. If you try to make it work you end up with macro expansion trying to call:

RUSTC_BOOTSTRAP=1 cargo rustc --profile check --target armv7-unknown-linux-uclibceabihf -- -Zunpretty=expanded/checkout/src/lib.rs

which fails with:

macro expansion failed with exit status: 101: error: extra arguments to `rustc` can only be passed to one target, consider filtering the package by passing, e.g., `--lib` or `--bin NAME` to specify a single target

To reproduce:

• Dockerfile needs some amendments, use attached Dockerfile at ci/docker/armv7-unknown-linux-uclibceabihf/Dockerfile
• from libc top level, run RUSTUP_TOOLCHAIN=nightly LIBC_CI_ZBUILD_STD=1 ./ci/run-docker.sh armv7-unknown-linux-uclibceabihf

[skrap]

Something about posting the Dockerfile isn't working, so I'll include it inline here:

FROM ubuntu:25.04

RUN apt-get update && apt-get install -y --no-install-recommends \
        ca-certificates \
        curl \
        file \
        gcc \
        libc6-dev \
        patch \
        qemu-system-arm \
        qemu-user \
        bzip2 \
        xz-utils

ARG TEST_UCLIBC_TIME64
COPY install-uclibc.sh /
RUN /install-uclibc.sh "$TEST_UCLIBC_TIME64"

ENV PATH=$PATH:/rust/bin:/toolchain/bin \
    STAGING_DIR=/toolchain/armv7-buildroot-linux-uclibceabihf/sysroot \
    CC_armv7_unknown_linux_uclibceabihf=arm-buildroot-linux-uclibcgnueabihf-gcc \
    CARGO_TARGET_ARMV7_UNKNOWN_LINUX_UCLIBCEABIHF_LINKER=arm-buildroot-linux-uclibcgnueabihf-gcc \
    CARGO_TARGET_ARMV7_UNKNOWN_LINUX_UCLIBCEABIHF_RUNNER="qemu-arm -L /toolchain/arm-buildroot-linux-uclibcgnueabihf/sysroot/"

[tgross35]

I believe @semarie is running frequent downstream tests on a T3 target - Sebastien have you hit this issue and worked around somehow?

If there isn't an easy solution then I suppose we could have ctest do something like cargo build -Zbuild-std=core, then pass the resulting rlib as a --extern. That seems messy though, and wouldn't be great for crates other than libc that need the deps of full std.

@mbyx might have some ideas too as the ctest expert.

[tgross35]

Actually @skrap does cargo-expand work on that target? If so, there might be some clues there since that also invokes rustc directly.

[mbyx]

I do remember when I was working on the rewrite that cargo rustc couldn't easily handle single files, and that the possible workaround was passing the rlib as @tgross35 said. But actually getting the path to the rlib is difficult I think.

As for the error, I'm not sure.

[skrap]

cargo expand does work but it's not much more than a wrapper around cargo rustc --profile=check -- -Zunpretty=expanded; I used it for inspiration for my attempts at getting cargo rust to work. It runs into the same problem, where cargo rustc cannot take a path, as it's already giving a src/lib.rs to rustc internally.

It seems possible that cargo rustc could be used when we are just trying to macro-expand the libc crate, but I don't understand all of the other uses of it. I am afraid of silently breaking libc's test infrastructure by doing so.

[tgross35]

Maybe we could create a dummy manifest in a temp directory and set the path, then build that?

[lib]
path = "/path/to/source/lib.rs"

[skrap]

Maybe we could create a dummy manifest in a temp directory and set the path, then build that?

[lib]
path = "/path/to/source/lib.rs"

Won't work, as the new manifest doesn't know about all the features, configs, crate name, etc.

You can test this easily by just creating such a dummy manifest and running:
RUSTUP_TOOLCHAIN=nightly cargo rustc --manifest-path /tmp/fake/Cargo.toml

(with the path pointed to your libc/src/lib.rs checkout)

[semarie]

I believe @semarie is running frequent downstream tests on a T3 target - Sebastien have you hit this issue and worked around somehow?

I don't hit it because I am running the CI natively on OpenBSD and I am targeting OpenBSD. so the core is present.

[tgross35]

Won't work, as the new manifest doesn't know about all the features, configs, crate name, etc.

Does any of that matter? It would have dummy values for the required manifest fields, but the existing rustc invocation isn't getting any of that extra meta either.

[skrap]

Does any of that matter? It would have dummy values for the required manifest fields, but the existing rustc invocation isn't getting any of that extra meta either.

I don't know. It does output source code, so the expansion might have worked, but also

warning: `fake` (lib) generated 457 warnings (222 duplicates)
error: could not compile `fake` (lib) due to 27 previous errors; 457 warnings emitted

And the process exits with status 101. Is that good enough?

[tgross35]

Just tested and it works with:

bash-3.2$ cat Cargo.toml
[package]
name = "libc"
edition = "2021"

[lib]
# Path to my libc
path = "/home/tmgross/projects/rust-libc/src/lib.rs"

[lints.rust]
# We don't pass allowed cfg so ignore hundreds of warnings
unexpected_cfgs = "allow"

bash-3.2$ cargo rustc -Zbuild-std=core --target armv7-unknown-linux-uclibceabihf -- -Zunpretty=expanded > /dev/null
   Compiling libc v0.0.0 (/private/tmp/libc-build)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.37s

The edition needs to be set otherwise there will be import errors, but we have that anyway. Also if the crate has a #![crate_name = "libc"] then the name has to match - libc has that (iirc since you shouldn't need Cargo to build it) but it won't be a problem for most crates, we can get this info from a new .crate_name(...) function.

[skrap]

Ah, setting the edition was the part I was missing. Thank you!