diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index d134f4b2a..50a1e90c5 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -52,53 +52,30 @@ parallel, then to the lint / versioning / SDK jobs. **Validation (E2E) test infrastructure.** Fully documented in [`docs/ci-validation-infrastructure.md`](../docs/ci-validation-infrastructure.md) -(matrix contents, job names, per-backend coverage and status, and the runbook -for adding/removing an OS, backend, or plan). Backend E2E tests run from those -same build artifacts — never from a fresh build — so artifact production and -consumption stay in one workflow run: - -- `.github/workflows/Validation.Tests.Scheduled.yml` — scheduled entry point. - The `nightly` plan runs Mon–Sat; Sunday runs `nightly` *and* `weekly`. - `workflow_dispatch` takes a `plan` input to run one on demand. +— read it before changing any of the pieces below. Backend E2E tests run from +the build artifacts, never from a fresh build, so an entry point must call the +three `Build.*.Job.yml` workflows before calling the matrix job. + +- `.github/workflows/Validation.Tests.Scheduled.yml` — scheduled entry point + (`nightly` Mon–Sat, `nightly` + `weekly` on Sunday); `workflow_dispatch` + takes a `plan` input. - `.github/workflows/Validation.Tests.Matrix.Job.yml` — workflow-call-only, - takes a `plan` input. Its `resolve` job expands the plan into per-family - matrices, then the `windows` / `linux` / `macos` jobs each download the - artifact, prepare the host, and run the backend suite. - -An entry point must build the artifacts (call the three `Build.*.Job.yml` -workflows) before calling the matrix job. - -**The matrix is declarative:** - -- `scripts/ci/validation-test-matrix.json` is the catalog: `platforms` (each - with per-architecture target/artifact/1ES pool and the backends that platform - supports), `triggers` (which OS/backend pairs each plan runs), and the - optional `backendDelayedStart` (per-backend job-start stagger, in seconds). - The `triggers` keys *are* the plan list — the resolver reads them at run time, - so adding a plan needs no script change. -- `scripts/ci/resolve-validation-test-matrix.mjs` validates that catalog and - expands a plan (currently `pr`, `nightly`, `weekly`, `enabled`) into GitHub - Actions matrices. It rejects an invalid catalog before any specialized test - runner is allocated, so add a backend to a trigger only where the platform - declares it. -- A non-macOS platform architecture with an empty `pool` is never scheduled, - which is how a catalog entry stays declared but dormant. macOS entries use a - GitHub-hosted `runner` instead of a 1ES `pool`. - -**Host preparation** happens in the matrix job before the tests, keyed by the -matrix `backend` id: `scripts/ci/prepare-windows-host.ps1` and -`scripts/ci/prepare-linux-host.sh`. A backend with no prerequisites is an -explicit no-op, so the step runs unconditionally for every entry. - -**Test dispatch** goes through `tests/scripts/run_ci_backend_tests.ps1` -(Windows) and `tests/scripts/run_ci_backend_tests.sh` (Linux/macOS), which map -the matrix `backend` id to the repository's existing backend suite. Ids that -share a suite get their own case (`process-t1` and `process-t3` both run -`WinProcessContainer-Tests.ps1`, which derives the tier it expects from the -host's own `--probe`). A backend with no wired suite fails loudly rather than -reporting a false success. The Windows dispatcher points `TEMP` at -`$RUNNER_TEMP` before running a suite, so anything a test writes to the temp -directory is picked up by the job's log upload without per-file CI wiring. + takes a `plan`. Its `resolve` job expands the plan into per-family matrices; + the `windows` / `linux` / `macos` jobs then download the artifact, prepare + the host, and run the backend suite. +- `scripts/ci/validation-test-matrix.json` — the declarative catalog + (`platforms`, `triggers`, `backendDelayedStart`). Its `triggers` keys *are* + the plan list. `scripts/ci/resolve-validation-test-matrix.mjs` validates the + catalog and expands a plan, so a backend may only be triggered where its + platform declares it. +- `scripts/ci/prepare-{windows,linux,macos}-host.{ps1,sh}` — per-backend host + prep, plus an inventory of the workload interpreters. Backend prerequisites + are installed per job; most workload interpreters come from image provisioning + scripts outside this repository, while Windows prep installs packaged `winapp` + and OpenSSL per job. +- `scripts/ci/run_backend_validation_tests.{ps1,sh}` — map a matrix `backend` + id to the repository's existing backend suite. An unwired id fails loudly + rather than reporting a false success. ### Individual components @@ -151,19 +128,20 @@ tests\scripts\run_windows_sandbox_one_shot_tests.ps1 # Windows Sandbox one tests\scripts\run_windows_sandbox_state_aware_tests.ps1 # Windows Sandbox state-aware lifecycle E2E (provision/start/exec*/stop/deprovision; requires the Windows Sandbox optional feature; skips if absent) tests\scripts\run_lxc_all_tests.sh # All LXC tests (Linux) tests\scripts\run_bwrap_all_tests.sh # All Bubblewrap tests (Linux, requires bwrap). Must NOT run as root — several tests assert the sandbox drops capabilities, which cannot hold under a root launcher; the script refuses root explicitly. -sudo tests\scripts\run_bwrap_inbound_deny_test.sh # Bubblewrap inbound default-deny E2E (root-only: needs host CAP_NET_ADMIN to read the sandbox netns and inject a peer). Reported as skipped by the suite above; CI runs it separately from run_ci_backend_tests.sh. +sudo tests\scripts\run_bwrap_inbound_deny_test.sh # Bubblewrap inbound default-deny E2E (root-only: needs host CAP_NET_ADMIN to read the sandbox netns and inject a peer). Reported as skipped by the suite above; CI runs it separately from run_backend_validation_tests.sh. + # E2E test crate — Rust executor integration tests (from src/) cargo test -p wxc_e2e_tests # Invokes MXC binaries directly cargo test -p wxc_e2e_tests -- --ignored # Include stress tests (run_on_repeat) # WSLC has no cargo E2E suite — it is covered by tests\scripts\run_wslc_all_tests.ps1, -# which the validation matrix runs via tests\scripts\run_ci_backend_tests.ps1. +# which the validation matrix runs via scripts\ci\run_backend_validation_tests.ps1. # CI validation entry points — run a backend suite against a downloaded artifact # the way the validation matrix does. Take the matrix backend id exactly as it # appears in scripts/ci/validation-test-matrix.json. -tests\scripts\run_ci_backend_tests.ps1 -Backend process-t1 -BinaryDirectory -Architecture x64 -tests\scripts\run_ci_backend_tests.sh +scripts\ci\run_backend_validation_tests.ps1 -Backend process-t1 -BinaryDirectory -Architecture x64 +scripts\ci\run_backend_validation_tests.sh # Resolve a plan locally to see exactly what CI would schedule node scripts/ci/resolve-validation-test-matrix.mjs --plan nightly diff --git a/.github/workflows/Validation.Tests.Matrix.Job.yml b/.github/workflows/Validation.Tests.Matrix.Job.yml index 7b3f7b9d0..403ca7ecd 100644 --- a/.github/workflows/Validation.Tests.Matrix.Job.yml +++ b/.github/workflows/Validation.Tests.Matrix.Job.yml @@ -73,7 +73,7 @@ jobs: timeout-minutes: 45 shell: pwsh run: | - & ./tests/scripts/run_ci_backend_tests.ps1 ` + & ./scripts/ci/run_backend_validation_tests.ps1 ` -Backend '${{ matrix.backend }}' ` -BinaryDirectory (Join-Path $env:GITHUB_WORKSPACE 'artifacts\bin') ` -Architecture '${{ matrix.architecture }}' *>&1 | @@ -147,11 +147,11 @@ jobs: run: | set -euo pipefail if [[ '${{ matrix.backend }}' == 'lxc' ]]; then - sudo --preserve-env=RUNNER_TEMP bash tests/scripts/run_ci_backend_tests.sh \ + sudo --preserve-env=RUNNER_TEMP bash scripts/ci/run_backend_validation_tests.sh \ '${{ matrix.backend }}' "$GITHUB_WORKSPACE/artifacts/bin" 2>&1 | tee -a "$RUNNER_TEMP/mxc-ci.log" else - bash tests/scripts/run_ci_backend_tests.sh \ + bash scripts/ci/run_backend_validation_tests.sh \ '${{ matrix.backend }}' "$GITHUB_WORKSPACE/artifacts/bin" 2>&1 | tee -a "$RUNNER_TEMP/mxc-ci.log" fi @@ -195,15 +195,24 @@ jobs: echo "Waiting $seconds second(s) before starting tests." sleep "$seconds" + - name: Prepare backend prerequisites + timeout-minutes: 15 + shell: bash + run: | + set -euo pipefail + bash scripts/ci/prepare-macos-host.sh \ + '${{ matrix.backend }}' "$GITHUB_WORKSPACE/artifacts/bin" 2>&1 | + tee "$RUNNER_TEMP/mxc-ci.log" + - name: Run backend tests timeout-minutes: 60 shell: bash run: | set -euo pipefail chmod +x artifacts/bin/mxc-exec-mac artifacts/bin/unix-test-proxy - bash tests/scripts/run_ci_backend_tests.sh \ + bash scripts/ci/run_backend_validation_tests.sh \ '${{ matrix.backend }}' "$GITHUB_WORKSPACE/artifacts/bin" 2>&1 | - tee "$RUNNER_TEMP/mxc-ci.log" + tee -a "$RUNNER_TEMP/mxc-ci.log" - name: Upload logs if: always() diff --git a/docs/ci-validation-infrastructure.md b/docs/ci-validation-infrastructure.md index 2df9da68e..9687fadc6 100644 --- a/docs/ci-validation-infrastructure.md +++ b/docs/ci-validation-infrastructure.md @@ -29,10 +29,11 @@ the individual local test scripts are documented in | `.github/workflows/Validation.Tests.Matrix.Job.yml` | `workflow_call`-only. Resolves the plan and runs the per-family test jobs. | | `scripts/ci/validation-test-matrix.json` | The matrix: OS versions, backends, triggers, job staggering. | | `scripts/ci/resolve-validation-test-matrix.mjs` | Matrix validator + plan expander. Emits the GitHub Actions matrices. | -| `scripts/ci/prepare-windows-host.ps1` | Per-backend Windows host preparation / prerequisite assertions. | -| `scripts/ci/prepare-linux-host.sh` | Per-backend Linux package install and service startup (distro-aware). | -| `tests/scripts/run_ci_backend_tests.ps1` | Windows dispatcher: backend id → existing backend suite. Also points `TEMP` at `$RUNNER_TEMP` so logs get collected. | -| `tests/scripts/run_ci_backend_tests.sh` | Linux/macOS dispatcher: backend id → existing backend suite. | +| `scripts/ci/prepare-windows-host.ps1` | Per-backend Windows host preparation / prerequisite assertions, plus the `winget` repair and the packaged-tooling install. | +| `scripts/ci/prepare-linux-host.sh` | Per-backend Linux package install and service startup (distro-aware), plus the workload-interpreter inventory. | +| `scripts/ci/prepare-macos-host.sh` | Per-backend macOS host preparation / prerequisite assertions. | +| `scripts/ci/run_backend_validation_tests.ps1` | Windows dispatcher: backend id → existing backend suite. Also points `TEMP` at `$RUNNER_TEMP` so logs get collected. | +| `scripts/ci/run_backend_validation_tests.sh` | Linux/macOS dispatcher: backend id → existing backend suite. | ### Flow @@ -42,9 +43,9 @@ Validation.Tests.Scheduled.yml ├─ windows / linux / macos → Build.*.Job.yml (upload artifacts) └─ test-nightly / test-weekly → Validation.Tests.Matrix.Job.yml └─ resolve → resolve-validation-test-matrix.mjs --plan - ├─ windows job (matrix) → download artifact → prepare-windows-host.ps1 → run_ci_backend_tests.ps1 - ├─ linux job (matrix) → download artifact → prepare-linux-host.sh → run_ci_backend_tests.sh - └─ macos job (matrix) → download artifact → run_ci_backend_tests.sh + ├─ windows job (matrix) → download artifact → prepare-windows-host.ps1 → run_backend_validation_tests.ps1 + ├─ linux job (matrix) → download artifact → prepare-linux-host.sh → run_backend_validation_tests.sh + └─ macos job (matrix) → download artifact → prepare-macos-host.sh → run_backend_validation_tests.sh ``` An entry point **must** build the artifacts before calling the matrix job — the @@ -70,9 +71,9 @@ Build artifacts are kept for 1 day — they exist only to feed these jobs. | Job | Runner | What it does | |-----|--------|--------------| | `resolve` | `ubuntu-latest` | Runs the resolver, emits one matrix per OS family plus `has_` flags so an empty family is skipped rather than failing on an empty matrix. | -| `windows` | `A self-hosted 1ES pool` | Download artifact → `prepare-windows-host.ps1 -Backend ` → `run_ci_backend_tests.ps1 -Backend `. | -| `linux` | `A self-hosted 1ES pool` | Download artifact → `prepare-linux-host.sh ` → `run_ci_backend_tests.sh ` (under `sudo` for LXC). | -| `macos` | GitHub-hosted `${{ matrix.runner }}` | Download artifact → `chmod +x` → `run_ci_backend_tests.sh `. No host-prep step. | +| `windows` | `A self-hosted 1ES Pool` | Download artifact → `prepare-windows-host.ps1 -Backend ` → `run_backend_validation_tests.ps1 -Backend `. | +| `linux` | `A self-hosted 1ES Pool` | Download artifact → `prepare-linux-host.sh ` → `run_backend_validation_tests.sh ` (under `sudo` for LXC). | +| `macos` | GitHub-hosted `${{ matrix.runner }}` | Download artifact → `prepare-macos-host.sh ` → `chmod +x` → `run_backend_validation_tests.sh `. | Per-job display name: `, , ` (macOS omits the architecture). Job timeout 180 min; host prep 15 min; the test step 45 min @@ -128,10 +129,32 @@ A backend id is passed straight through: the matrix job hands it to the host-pre script and then to the dispatcher, which has one `switch`/`case` per id. Ids that share a suite each keep their own case so they can diverge later without a mapping table — `process-t1` and `process-t3` both run -`WinProcessContainer-Tests.ps1` today. Teaching the Process Container test suite to +`WinProcessContainer-Tests.ps1`, and `process-t3` additionally runs +`T3-Workloads.ps1`. Teaching the Process Container test suite to accept an explicit tier (so a T1 host can also be exercised -at the T3 fallback) is a worthwhile future improvement; see -[Possible future improvements](#possible-future-improvements). +at the T3 fallback) is a worthwhile future improvement. + +`process-t3` runs its two suites back to back and reports them together: a +failure in the primitives suite does not skip the workloads suite, so one job +run shows both results instead of costing a second run to triage. + +The dispatcher passes `T3-Workloads.ps1` its `-GrantDriveRoot` switch. The +pwsh- and git-driven workloads resolve the whole ancestor chain of their working +directory at startup, so granting only the scratch leaf leaves them failing +before they reach the behaviour under test. Granting the drive root covers the +chain, but at T3 a policy path becomes an inheritable ACE, so it rewrites ACLs +across the system drive on every run and again on teardown — acceptable on a +disposable runner, which is why the switch is off by default and only CI opts +in. Temporary until pwsh 7.7 leaves preview. + +The suite's git workloads also restamp the ownership of the repo they build. +The 1ES agent runs elevated, and an elevated token's *default owner* is +`BUILTIN\Administrators`, so a fixture created there is Administrators-owned; +git refuses such a repo ("detected dubious ownership") unless the caller is +itself an elevated administrator, which a contained process never is. That is a +property of the agent rather than of containment — the identical fixture is +user-owned on a dev box — so the suite reowns it to the current user and keeps +the two environments testing the same thing. An unwired backend fails loudly on purpose: adding it to a trigger produces a red job ("write the tests or remove it"), never a green no-op. The dispatchers' @@ -158,12 +181,12 @@ because Seatbelt has no wired suite. ### `backendDelayedStart` -Optional. Staggers the start of jobs for a named backend instead of letting -them all begin at once: +Optional. This section staggers the start of jobs +for a named backend instead of letting them all begin at once: ```json "backendDelayedStart": [ - { "backend": "wslc", "seconds": 300 } + { "backend": "wslc", "seconds": 30 } ] ``` @@ -173,21 +196,10 @@ traffic into a burst the moment its jobs start together. Public registries answer with rate limiting and stalled downloads. `seconds` is the gap between consecutive jobs of that backend, counted per -backend and following the resolved job order. With the entry above, four WSLC -jobs start at 0, 300, 600, and 900 seconds. +backend and following the resolved job order. With the example entry above, +four WSLC jobs would start at 0, 30, 60, and 90 seconds. -The resolver puts the offset on every matrix entry as -`startup_delay_seconds` — `0` where no stagger applies — and the job sleeps -that long before its first network step. Job timeout is a flat 180 minutes, -with plenty of room for any wait you'd reasonably configure. - -Leave the section out (or empty) and every job starts as soon as its runner is -ready. A backend id that no plan schedules is accepted; it just never applies. - -Do keep in mind that the runner is held while it sleeps — Actions can't defer -allocating a matrix job, so the wait has to happen inside it. Use no more than -the contention calls for. This spreads simultaneous load and nothing else; a -single download that stalls on its own is unaffected. +Avoid long delays since idle runners might be harvested. ## Backend status @@ -196,13 +208,13 @@ get fixed or wired. | Backend | Status | Notes | |---------|--------|-------| -| Process T1 | ✅ Good | Prerelease Windows only. Remaining failures are genuine MXC bugs or harness limitations. | -| Process T3 | ✅ Good | Non-prerelease Windows builds only, until the testing suite is updated. | +| Process T1 | ✅ Good | Prerelease Windows only. Runs the primitives suite. Remaining failures are genuine MXC bugs or harness limitations. | +| Process T3 | ✅ Good | Non-prerelease Windows builds only. Runs the primitives suite plus `T3-Workloads.ps1` (real programs — pwsh, git, node, python, cmd — on top of the T3 primitives). | | Bubblewrap | ✅ Good | | | LXC | ✅ Good | Some networking tests fail on distros other than Ubuntu 24.04; seems to be an issue with MXC. | | WSLC | ✅ Good | Might have to retry hung jobs - this is an issue with overzealous agent reclaiming. | -| IsolationSession | ⚠️ Blocked | `Feature_AgentSessionsBaseSupport` is not enabled on the pool image yet. | -| Windows Sandbox | ⛔ Not scheduled | Dispatcher case is wired; no trigger entry yet. | +| IsolationSession | ✅ Good | | +| Windows Sandbox | ⛔ Blocked | Images don't support `Containers-DisposableClientVM` opt. feature | | MicroVM | ⛔ Not working | Windows cold and warm starts hang; no Linux suite. The artifact payload is currently commented out in the build jobs. | | Hyperlight | ⛔ Not implemented | No suite on any platform. | | Seatbelt | ⛔ Not implemented | The backend itself is healthy; there is no official E2E suite to dispatch to. | @@ -231,6 +243,12 @@ Windows optional features are **verified, never enabled**: turning one on needs reboot the job cannot take, so a mis-imaged pool fails here with a pointed message instead of surfacing later as an opaque backend error. +The script does provision two things, for every backend rather than a particular +one: `Repair-Winget` re-registers the App Installer package when `winget` is on +`PATH` but cannot run, and `Install-PackagedTooling` then installs `winapp` and +`openssl` +([below](#who-installs-what)). + `prepare-linux-host.sh`: - `bubblewrap` — installs `bwrap`, `slirp4netns`, `util-linux`, and `iptables` @@ -243,7 +261,95 @@ message instead of surfacing later as an opaque backend error. - `microvm` — asserts the NanVix payload exists. - `hyperlight` — no-op. -macOS has no preparation step. +Every install above goes through two shared helpers rather than its own +package-manager chain: `resolve_package_manager` picks the first of `apt-get`, +`dnf`, `yum`, or `microdnf` on the host and caches it, and `install_packages` +holds the single `case` that knows how each one is invoked. Supporting a new +distribution family is therefore one new arm in `install_packages`, not another +branch in every installer. `install_packages` returns the package manager's own +status rather than acting on it, so each caller decides what a failure means. + +`prepare-macos-host.sh`: + +- `seatbelt` — asserts `mxc-exec-mac` shipped. The sandbox is part of the OS, so + there is nothing to install; the script exists so macOS has the same shape as + the other two and a future prerequisite has an obvious home. + +Every one of the three scripts also takes the workload-interpreter inventory +([below](#workload-interpreters)) before it dispatches on the backend id. + +### Workload interpreters + +Some suites do not just exercise MXC's primitives — they run *real programs* +inside the sandbox and assert on what those programs produce. Each preparation +script inventories those programs on the host up front, so a missing one is +reported once, as a preparation result, rather than repeatedly as a confusing +mid-suite failure. + +#### The list + +| Interpreter | Platforms | Version | Notes | +|-------------|-----------|---------|-------| +| `pwsh` | all | Latest 7.x | The only entry whose absence fails a job, and only on Windows. | +| `git` | all | Latest | | +| `node`, `npm`, `npx` | all | 24.x | `npm` and `npx` arrive with Node. | +| `python`, `pip` | all | Latest | Windows tries `python` first, Unix `python3`. | +| `dotnet` | all | Latest LTS | Currently 10.x, from the `LTS` channel — resolved at image-build time, not pinned. | +| `az` | all | Latest | No ARM64 Windows build exists; ARM64 images get the x64 one under emulation. | +| `gh` | all | Latest | | +| `openssl` | all | Latest | On Windows, installed per job — see below. | +| `nuget` | Windows | Latest | Unix reaches NuGet through `dotnet nuget`, so looking for a binary there would warn forever. | +| `winapp` | Windows | Latest | The Windows App Development CLI. Installed per job — see below. | +| `winget` | Windows | Image | The only entry allowed to resolve inside `WindowsApps`. | +| `scoop`, `choco` | Windows | Latest | | +| `brew` | macOS | Latest | | + +Nothing is pinned: every entry is whatever was current when the image was built, +within the constraint in the Version column. Only Node is held to a major +version, because the SDK targets it. + +The list is **suite-agnostic by design** and is checked for every backend, not +only the ones whose suites need it today: it describes what a validation *host* +provides, not what any one suite consumes. `T3-Workloads.ps1` is simply the +first caller, and wiring up the next one needs no change here. + +Each script carries its own copy — `Assert-WorkloadInterpreters` in +`prepare-windows-host.ps1`, `assert_workload_interpreters` in the two `.sh` +scripts — so the platform lists can diverge. + +#### Missing means a warning, not a failure + +Only `pwsh` on Windows is required; everything else warns, because suites are +expected to report their dependent cases as skipped rather than failing. Nothing +is required on Unix yet. + +The cost is that an absent interpreter silently shrinks coverage while the job +still shows green. GitHub's pass/fail icon cannot express "passed, but with less +coverage than yesterday" — a future test-analysis portal is intended to surface +skip counts so that erosion is visible. + +On Windows a command resolving under `WindowsApps` does not count. That is +normally a Microsoft Store `AppExecutionAlias` stub: a 0-byte redirect that +opens the Store rather than running. App Installer legitimately ships `winget` +that way, and a working alias is indistinguishable from a stub — both are 0-byte +reparse points — so `winget` alone opts out of the filter. + +#### Who installs what + +Every pool runs images pre-provisioned with programs installed by +`ubuntu-debian-provision.sh` / `rhel-provision.sh` for Linux and +`windows-provision.ps1` for Windows. These scripts are located in the +`validation-provision-artifacts` branch in the ADO repo. On Windows that +script covers `dotnet`, `choco`, `scoop`, `az`, `gh` and `nuget`; the remaining +runtimes (`node`, `python`, `pwsh`, `git`) come from separate image artifacts. + +During the start of a job, the installed programs are inventoried; no job +installs a workload interpreter. Windows is the one exception, installing +OpenSSL and WinApp via the repaired WinGet, because both are published as +packaged applications. + +A backend's own prerequisites are separate and are still installed per job by +`prepare-linux-host.sh` — see [Host preparation](#host-preparation). ## Log collection @@ -256,7 +362,7 @@ scratch trees, transcripts, and results files under the user's temp directory `${{ runner.temp }}` (`C:\a\_work\_temp`). Anything left in the former is simply never collected, which is why the artifact used to arrive nearly empty. -So `run_ci_backend_tests.ps1` points `TEMP` and `TMP` at `$RUNNER_TEMP` before +So `run_backend_validation_tests.ps1` points `TEMP` and `TMP` at `$RUNNER_TEMP` before it dispatches. Parameter defaults, `[System.IO.Path]::GetTempPath()`, and child processes all read those variables, so everything temp-rooted lands in the upload directory without CI having to know a single filename. @@ -300,13 +406,13 @@ test runner is allocated. 1. **Catalog:** add the id to the `backends` list of every platform/arch that can run it. There is no separate registration step — the id *is* the dispatcher argument. -2. **Dispatcher:** add a case to `run_ci_backend_tests.ps1` (`ValidateSet` + - `switch`) or `run_ci_backend_tests.sh` (`usage` + `case`), pointing at the +2. **Dispatcher:** add a case to `run_backend_validation_tests.ps1` (`ValidateSet` + + `switch`) or `run_backend_validation_tests.sh` (`usage` + `case`), pointing at the suite. Until a suite exists, leave the explicit throw / `exit 2` so accidental activation fails loudly. 3. **Host prep:** add a branch to `prepare-windows-host.ps1` (`ValidateSet` + - `switch`) or `prepare-linux-host.sh` (`usage` + `case`). Skip only if there - is genuinely nothing to install or assert. + `switch`), `prepare-linux-host.sh`, or `prepare-macos-host.sh` (`usage` + + `case`). Skip only if there is genuinely nothing to install or assert. 4. **Artifact:** make sure everything the suite needs is in the `Upload binaries` list of the relevant `Build.*.Job.yml`, and that the build enables the backend's cargo feature. @@ -326,7 +432,10 @@ passing a distinguishing argument later without touching the matrix. 3. For a Windows prerelease image set `"prerelease": true` and use a neutral `windows-prerelease-` id — the id appears in public job names. 4. For a new Linux distro, check that `prepare-linux-host.sh` handles its - package manager and service layout. + package manager and service layout. A new package-manager family needs one + arm in `install_packages` and one entry in `resolve_package_manager`; a + family whose package *names* differ also needs its column in the tables in + `install_lxc` and `install_bubblewrap`. 5. Add it to a plan's `triggers`, then resolve locally. ### Wire an unwired backend to a suite diff --git a/docs/playground-limitations.md b/docs/playground-limitations.md index 7d272a528..d08b386a5 100644 --- a/docs/playground-limitations.md +++ b/docs/playground-limitations.md @@ -46,6 +46,19 @@ work if the ACL is set: icacls C:\Python314 /grant "ALL APPLICATION PACKAGES:(OI)(CI)(RX)" /T ``` +### git and files created by an elevated process + +git refuses a repository whose owner is not the caller (`fatal: detected dubious +ownership`). It makes one exception — a repo owned by `BUILTIN\Administrators` is +accepted if the caller is *itself* an elevated administrator — and a contained +process can never qualify, because the AppContainer token drops that membership. + +An elevated process's token hands out `BUILTIN\Administrators` as the default +owner of everything it creates, so **any repo cloned or created while elevated is +unusable from inside a container**, even though the same repo works fine on the +host. Create it unelevated, reassign the owner, or add it to `safe.directory` in +protected (system/global) git config. + This is a Windows security model limitation, not an MXC bug. ## Network Limitations diff --git a/scripts/ci/prepare-linux-host.sh b/scripts/ci/prepare-linux-host.sh index 6e270ca06..2f91bd80f 100644 --- a/scripts/ci/prepare-linux-host.sh +++ b/scripts/ci/prepare-linux-host.sh @@ -25,18 +25,56 @@ apt_update() { fi } +# Resolves the host's package manager once, so supporting a new distribution +# family is a case in install_packages rather than another branch in every +# installer below. +package_manager="" +resolve_package_manager() { + if [[ -n "$package_manager" ]]; then + return 0 + fi + + local candidate + for candidate in apt-get dnf yum microdnf; do + if command -v "$candidate" >/dev/null 2>&1; then + package_manager="$candidate" + return 0 + fi + done + return 1 +} + +# install_packages ... +# Installs from the feeds the host already has configured, and returns the +# package manager's own status so each caller decides what a failure means. +install_packages() { + case "$package_manager" in + apt-get) + # sudo resets the environment, so the frontend setting has to be + # applied on the far side of it rather than exported here. + sudo env DEBIAN_FRONTEND=noninteractive \ + apt-get install -y --no-install-recommends "$@" + ;; + dnf | yum | microdnf) + sudo "$package_manager" install -y "$@" + ;; + *) + echo "Unsupported package manager: '$package_manager'." >&2 + return 1 + ;; + esac +} + # Red Hat ships no third-party content, so epel-release is not in RHEL's own # repos; the documented install is the release RPM straight from Fedora. install_epel() { - local package_manager="$1" - if command -v subscription-manager >/dev/null 2>&1; then sudo subscription-manager repos \ --enable "codeready-builder-for-rhel-10-$(arch)-rpms" || echo "WARNING: could not enable the CRB repository; EPEL packages that depend on it may fail to install." >&2 fi - sudo "$package_manager" install -y \ + install_packages \ https://dl.fedoraproject.org/pub/epel/epel-release-latest-10.noarch.rpm } @@ -46,24 +84,25 @@ install_bubblewrap() { command -v unshare >/dev/null 2>&1 && command -v nsenter >/dev/null 2>&1 && command -v iptables >/dev/null 2>&1 && + command -v ip6tables >/dev/null 2>&1 && command -v ip >/dev/null 2>&1; then return fi - if command -v apt-get >/dev/null 2>&1; then - apt_update - sudo apt-get install -y --no-install-recommends \ - bubblewrap slirp4netns util-linux iptables iproute2 - elif command -v dnf >/dev/null 2>&1; then - sudo dnf install -y bubblewrap slirp4netns util-linux iptables iproute - elif command -v yum >/dev/null 2>&1; then - sudo yum install -y bubblewrap slirp4netns util-linux iptables iproute - elif command -v microdnf >/dev/null 2>&1; then - sudo microdnf install -y bubblewrap slirp4netns util-linux iptables iproute - else + + if ! resolve_package_manager; then echo "No supported package manager found to install Bubblewrap prerequisites." >&2 exit 1 fi + + local packages=(bubblewrap slirp4netns util-linux iptables) + if [[ "$package_manager" == "apt-get" ]]; then + apt_update + packages+=(iproute2) + else + packages+=(iproute) + fi + install_packages "${packages[@]}" } # The ingress chain matches on connection state, which iptables can only @@ -87,27 +126,25 @@ install_lxc() { if command -v lxc-start >/dev/null 2>&1; then return fi - if command -v apt-get >/dev/null 2>&1; then + + if ! resolve_package_manager; then + echo "No supported package manager found to install LXC." >&2 + exit 1 + fi + + local packages + if [[ "$package_manager" == "apt-get" ]]; then apt_update + packages=(lxc dnsmasq-base iptables bridge-utils) # Debian dropped lxc-utils; Ubuntu still ships it. - local packages=(lxc dnsmasq-base iptables bridge-utils) if apt-cache show lxc-utils >/dev/null 2>&1; then packages+=(lxc-utils) fi - sudo apt-get install -y --no-install-recommends "${packages[@]}" - elif command -v dnf >/dev/null 2>&1; then - install_epel dnf - sudo dnf install -y lxc lxc-templates dnsmasq iptables - elif command -v yum >/dev/null 2>&1; then - install_epel yum - sudo yum install -y lxc lxc-templates dnsmasq iptables - elif command -v microdnf >/dev/null 2>&1; then - install_epel microdnf - sudo microdnf install -y lxc lxc-templates dnsmasq iptables else - echo "No supported package manager found to install LXC." >&2 - exit 1 + install_epel + packages=(lxc lxc-templates dnsmasq iptables) fi + install_packages "${packages[@]}" } # Start the LXC bridge and wait until it can actually serve containers. @@ -205,7 +242,67 @@ enable_bridge_netfilter() { done } +# Verifies the interpreters test suites drive inside the sandbox and reports +# what the image actually provides, so a tool the image was built without stays +# visible instead of silently absent. These are baked into the image rather +# than installed here: unlike a backend's prerequisites, they are ordinary +# developer tools that every job expects to find already in place. +# +# The check is suite-agnostic: it describes what a validation host is expected +# to provide, not what any one suite consumes, so a future suite that shells out +# to these programs needs no change here. +assert_workload_interpreters() { + # name|candidates (tried in order)|required|remedy + local interpreters=( + "pwsh|pwsh|false|install PowerShell 7 in the image" + "git|git|false|install Git in the image" + "node|node|false|install Node.js in the image" + "npm|npm|false|install Node.js in the image (npm ships with it)" + "npx|npx|false|install Node.js in the image (npx ships with it)" + "python|python3,python|false|install Python in the image" + "pip|pip3,pip|false|install Python in the image (pip ships with it)" + "dotnet|dotnet|false|install the .NET SDK in the image" + "az|az|false|install the Azure CLI in the image" + "gh|gh|false|install the GitHub CLI in the image" + "openssl|openssl|false|install OpenSSL in the image" + ) + + local missing="" entry name candidates required remedy resolved candidate + local candidate_list + for entry in "${interpreters[@]}"; do + IFS='|' read -r name candidates required remedy <<<"$entry" + + resolved="" + IFS=',' read -r -a candidate_list <<<"$candidates" + for candidate in "${candidate_list[@]}"; do + # python3 before python: on Unix a bare `python` is usually absent, + # and where it does exist it can still be Python 2. + if resolved="$(command -v "$candidate" 2>/dev/null)"; then + break + fi + resolved="" + done + + if [[ -n "$resolved" ]]; then + echo "Workload interpreter '$name' found at $resolved" + elif [[ "$required" == "true" ]]; then + missing="${missing:+$missing; }$name ($remedy)" + else + echo "::warning::Workload interpreter '$name' is absent ($remedy)" + fi + done + + if [[ -n "$missing" ]]; then + echo "::error::Workload interpreters missing from this image: $missing" + exit 1 + fi +} + chmod +x "$binary_directory/lxc-exec" + +# Run for every backend: this is host inventory, not a backend prerequisite. +assert_workload_interpreters + case "$backend" in bubblewrap) install_bubblewrap diff --git a/scripts/ci/prepare-macos-host.sh b/scripts/ci/prepare-macos-host.sh new file mode 100644 index 000000000..a3e7042c9 --- /dev/null +++ b/scripts/ci/prepare-macos-host.sh @@ -0,0 +1,90 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Prepares a macOS host for a backend's artifact-only test suite. +# +# Seatbelt needs nothing installed -- the sandbox is part of the OS -- so this +# currently only takes the host's workload-interpreter inventory. It exists so +# macOS has the same shape as the Windows and Linux preparation scripts, giving +# a future prerequisite an obvious home instead of another workflow-inline step. + +usage() { + echo "Usage: $0 " >&2 +} + +if [[ $# -ne 2 ]]; then + usage + exit 2 +fi + +backend="$1" +binary_directory="$2" + +# Verifies the interpreters test suites drive inside the sandbox, following the +# same verify-never-install rule as the rest of host preparation: a missing one +# is an image problem, not something a job can fix mid-run. +# +# The check is suite-agnostic: it describes what a validation host is expected +# to provide, not what any one suite consumes, so a future suite that shells out +# to these programs needs no change here. +assert_workload_interpreters() { + # name|candidates (tried in order)|required|remedy + local interpreters=( + "pwsh|pwsh|false|install PowerShell 7 in the image" + "git|git|false|install Git in the image" + "node|node|false|install Node.js in the image" + "npm|npm|false|install Node.js in the image (npm ships with it)" + "npx|npx|false|install Node.js in the image (npx ships with it)" + "python|python3,python|false|install Python in the image" + "pip|pip3,pip|false|install Python in the image (pip ships with it)" + "dotnet|dotnet|false|install the .NET SDK in the image" + "az|az|false|install the Azure CLI in the image" + "gh|gh|false|install the GitHub CLI in the image" + "openssl|openssl|false|install OpenSSL in the image" + # macOS-only + "brew|brew|false|install Homebrew in the image" + ) + + local missing="" entry name candidates required remedy resolved candidate + local candidate_list + for entry in "${interpreters[@]}"; do + IFS='|' read -r name candidates required remedy <<<"$entry" + + resolved="" + IFS=',' read -r -a candidate_list <<<"$candidates" + for candidate in "${candidate_list[@]}"; do + # python3 before python: on Unix a bare `python` is usually absent, + # and where it does exist it can still be Python 2. + if resolved="$(command -v "$candidate" 2>/dev/null)"; then + break + fi + resolved="" + done + + if [[ -n "$resolved" ]]; then + echo "Workload interpreter '$name' found at $resolved" + elif [[ "$required" == "true" ]]; then + missing="${missing:+$missing; }$name ($remedy)" + else + echo "::warning::Workload interpreter '$name' is absent ($remedy)" + fi + done + + if [[ -n "$missing" ]]; then + echo "::error::Workload interpreters missing from this image: $missing" + exit 1 + fi +} + +# Runs for every backend: this is host inventory, not a backend prerequisite. +assert_workload_interpreters + +case "$backend" in + seatbelt) + test -f "$binary_directory/mxc-exec-mac" + ;; + *) + usage + exit 2 + ;; +esac diff --git a/scripts/ci/prepare-windows-host.ps1 b/scripts/ci/prepare-windows-host.ps1 index 76955478d..491e861f4 100644 --- a/scripts/ci/prepare-windows-host.ps1 +++ b/scripts/ci/prepare-windows-host.ps1 @@ -35,6 +35,8 @@ param( Set-StrictMode -Version Latest $ErrorActionPreference = 'Stop' +$script:WingetPath = $null + function Exit-WithError { param([Parameter(Mandatory)][string]$Message) @@ -147,6 +149,276 @@ function Initialize-ProcessContainerHost { } } +# Verify the interpreters test suites drive inside the sandbox, following +# the same verify-never-install rule as the optional-feature assertions above: +# a missing one is an image problem, not something the job can fix mid-run. +function Assert-WorkloadInterpreters { + $interpreters = @( + @{ Name = 'pwsh'; Candidates = @('pwsh'); Required = $true; Remedy = 'install PowerShell 7 in the image' }, + @{ Name = 'git'; Candidates = @('git'); Required = $false; Remedy = 'install Git for Windows in the image' }, + @{ Name = 'node'; Candidates = @('node'); Required = $false; Remedy = 'install Node.js in the image' }, + @{ Name = 'npm'; Candidates = @('npm'); Required = $false; Remedy = 'install Node.js in the image (npm ships with it)' }, + @{ Name = 'npx'; Candidates = @('npx'); Required = $false; Remedy = 'install Node.js in the image (npx ships with it)' }, + @{ Name = 'python'; Candidates = @('python', 'python3'); Required = $false; Remedy = 'install Python in the image' }, + @{ Name = 'pip'; Candidates = @('pip', 'pip3'); Required = $false; Remedy = 'install Python in the image (pip ships with it)' }, + @{ Name = 'dotnet'; Candidates = @('dotnet'); Required = $false; Remedy = 'install the .NET SDK in the image' }, + @{ Name = 'az'; Candidates = @('az'); Required = $false; Remedy = 'install the Azure CLI in the image' }, + @{ Name = 'gh'; Candidates = @('gh'); Required = $false; Remedy = 'install the GitHub CLI in the image' }, + @{ Name = 'openssl'; Candidates = @('openssl'); Required = $false; Remedy = 'only published as a packaged application, so it is installed by Install-PackagedTooling rather than baked into the image' }, + # Windows-only + @{ Name = 'nuget'; Candidates = @('nuget'); Required = $false; Remedy = 'install the NuGet CLI in the image' }, + @{ Name = 'winapp'; Candidates = @('winapp'); Required = $false; Remedy = 'only published as a packaged application, so it is installed by Install-PackagedTooling rather than baked into the image' }, + @{ Name = 'winget'; Candidates = @('winget'); Required = $false; AllowStoreAlias = $true; Remedy = 'install the Windows Package Manager (App Installer) in the image' }, + @{ Name = 'scoop'; Candidates = @('scoop'); Required = $false; Remedy = 'install Scoop in the image' }, + @{ Name = 'choco'; Candidates = @('choco'); Required = $false; Remedy = 'install Chocolatey in the image' } + ) + + $missing = @() + foreach ($tool in $interpreters) { + $resolved = $null + foreach ($candidate in $tool.Candidates) { + $found = Get-Command $candidate -ErrorAction SilentlyContinue + # A command resolving into WindowsApps is normally a Microsoft Store + # AppExecutionAlias stub: a 0-byte redirect that opens the Store + # rather than running, which the suite deliberately ignores. + # + # WindowsApps is also how App Installer legitimately delivers winget, + # and a working alias is indistinguishable from a stub by path or by + # size (both are 0-byte reparse points). So entries that ship that + # way opt out of the filter via AllowStoreAlias; blanket-filtering + # them reports an installed tool as missing. + if ($found -and ($tool['AllowStoreAlias'] -or $found.Source -notlike '*\WindowsApps\*')) { + $resolved = $found.Source + break + } + } + if ($resolved) { + Write-Host "Workload interpreter '$($tool.Name)' found at $resolved" + } elseif ($tool.Required) { + $missing += "$($tool.Name) ($($tool.Remedy))" + } else { + Write-Host "::warning::Workload interpreter '$($tool.Name)' is absent ($($tool.Remedy))" + } + } + + if ($missing) { + Exit-WithError "Workload interpreters missing from this image: $($missing -join '; ')" + } +} + +# Returns whether winget can actually run, which is not the same as being on +# PATH: an unregistered App Installer leaves an alias that resolves and then +# fails to launch. The explicit alias path is a second candidate because +# PowerShell caches command lookups, so a repair inside this process is not +# guaranteed to be visible through Get-Command. +function Test-WingetOperational { + $candidates = @() + $command = Get-Command winget -ErrorAction SilentlyContinue + if ($command) { $candidates += $command.Source } + $candidates += Join-Path $env:LOCALAPPDATA 'Microsoft\WindowsApps\winget.exe' + + foreach ($candidate in $candidates) { + if (-not $candidate) { continue } + try { + $version = & $candidate --version 2>&1 + if ($LASTEXITCODE -eq 0 -and $version) { + Write-Host "winget is operational ($($version | Select-Object -First 1)) at $candidate" + $script:WingetPath = $candidate + return $true + } + } catch { + # A broken alias throws rather than returning an exit code. + } + } + return $false +} + +# winget ships as an AppExecutionAlias belonging to the App Installer package. +# CI images routinely carry the package while leaving it unregistered for the +# account the job runs as, which produces an alias that resolves on PATH but +# fails with "The file cannot be accessed by the system". Registering the +# package already on the image is the documented repair and downloads nothing. +# +# This is the one exception to the verify-never-install rule above, and only +# barely: it installs nothing, it re-registers what the image already shipped. +# It is best effort — winget is optional, so nothing here fails the job. +function Repair-Winget { + if (Test-WingetOperational) { + return + } + + Write-Host "winget did not run; checking whether the App Installer package is present." + + $package = $null + try { + $package = Get-AppxPackage -Name Microsoft.DesktopAppInstaller -ErrorAction Stop | + Select-Object -First 1 + } catch { + # PowerShell 7 builds without the native Appx binary module have to + # reach these cmdlets through Windows PowerShell. + try { + Import-Module Appx -UseWindowsPowerShell -ErrorAction Stop -WarningAction SilentlyContinue + $package = Get-AppxPackage -Name Microsoft.DesktopAppInstaller -ErrorAction Stop | + Select-Object -First 1 + } catch { + Write-Host "::warning::Could not query Appx packages, so winget cannot be repaired: $($_.Exception.Message)" + return + } + } + + if (-not $package) { + Write-Host "::warning::winget is unavailable and the App Installer package is absent; install it in the image." + return + } + + Write-Host "App Installer $($package.Version) is present (status $($package.Status)); registering it for the current user." + try { + # PackageFamilyName is Microsoft.DesktopAppInstaller_8wekyb3d8bbwe, read + # off the package rather than hard-coded. + Add-AppxPackage -RegisterByFamilyName -MainPackage $package.PackageFamilyName -ErrorAction Stop + } catch { + Write-Host "::warning::Could not register the App Installer package: $($_.Exception.Message)" + return + } + + if (Test-WingetOperational) { + Write-Host "winget is operational after registering App Installer." + } else { + Write-Host "::warning::winget is still not operational after registering App Installer." + } +} + +# Appends the registry PATH entries this process has not picked up yet, so a +# directory an installer just published becomes visible to a process that +# otherwise keeps the PATH it started with. +function Update-ProcessPath { + $seen = [System.Collections.Generic.HashSet[string]]::new([StringComparer]::OrdinalIgnoreCase) + foreach ($entry in $env:Path -split ';') { + if ($entry) { [void]$seen.Add($entry.TrimEnd('\')) } + } + + foreach ($scope in 'Machine', 'User') { + $value = [Environment]::GetEnvironmentVariable('Path', $scope) + if (-not $value) { continue } + foreach ($entry in $value -split ';') { + if ($entry -and $seen.Add($entry.TrimEnd('\'))) { + $env:Path = "$env:Path;$entry" + } + } + } +} + +# Resolves a command the way Assert-WorkloadInterpreters does, so a tool this +# step installs is judged by the same rule that reports it later. +function Resolve-Interpreter { + param([Parameter(Mandatory)][string[]]$Candidates) + + foreach ($candidate in $Candidates) { + $found = Get-Command $candidate -ErrorAction SilentlyContinue + if ($found -and $found.Source -notlike '*\WindowsApps\*') { + return $found.Source + } + } + return $null +} + +# openssl and the Windows App Development CLI are the two workload interpreters +# that cannot be baked into an image: both are published only as packaged +# applications, and no packaged application can be registered while an image is +# being provisioned. They are installed here instead, on the running machine, +# which is the first point at which winget works. +# +# This is the second exception to the verify-never-install rule above. It is +# best effort: both tools are optional, so a failure warns here and the +# inventory that follows reports what the machine actually ended up with. +function Install-PackagedTooling { + # 0 is success; the other two are "already installed" and "no applicable + # upgrade", which both mean the tool is present and are equally fine. + $success = @(0, -1978335135, -1978335189) + + $packages = @( + @{ + Name = 'winapp' + Candidates = @('winapp') + Id = 'Microsoft.WinAppCli' + # This package publishes a packaged build and a portable one. The + # packaged build installs behind an execution alias in WindowsApps, + # which is indistinguishable from a Store stub and is therefore not + # counted as present. The portable build puts a real executable on + # PATH instead, so it is the one to ask for. + Extra = @('--installer-type', 'zip') + PathHints = @( + (Join-Path $env:ProgramFiles 'WinGet\Links'), + (Join-Path $env:LOCALAPPDATA 'Microsoft\WinGet\Links') + ) + }, + @{ + Name = 'openssl' + Candidates = @('openssl') + Id = 'ShiningLight.OpenSSL.Light' + Extra = @() + # This installer does not publish its own bin directory. + PathHints = @( + (Join-Path $env:ProgramFiles 'OpenSSL-Win64\bin'), + (Join-Path $env:ProgramFiles 'OpenSSL\bin') + ) + } + ) + + $wanted = $packages | Where-Object { -not (Resolve-Interpreter -Candidates $_.Candidates) } + if (-not $wanted) { + Write-Host "Packaged workload tooling is already present." + $global:LASTEXITCODE = 0 + return + } + + if (-not $script:WingetPath) { + Write-Host "::warning::winget is unavailable, so $(($wanted.Name) -join ' and ') cannot be installed." + $global:LASTEXITCODE = 0 + return + } + + foreach ($package in $wanted) { + Write-Host "Installing $($package.Name) ($($package.Id))." + $arguments = @( + 'install', '--id', $package.Id, '--exact', '--silent', + '--disable-interactivity', '--accept-source-agreements', + '--accept-package-agreements' + ) + $package.Extra + + try { + $output = & $script:WingetPath @arguments 2>&1 + $code = $LASTEXITCODE + } catch { + Write-Host "::warning::Could not install $($package.Name): $(($_.Exception.Message -split "`r?`n" | Select-Object -First 1).Trim())" + continue + } + + if ($success -notcontains $code) { + $detail = ($output | Select-Object -Last 3 | Out-String).Trim() + if ($detail) { Write-Host $detail } + Write-Host "::warning::Installing $($package.Name) reported exit code $code." + continue + } + + Update-ProcessPath + foreach ($hint in $package.PathHints) { + if ((Test-Path -LiteralPath $hint) -and (($env:Path -split ';') -notcontains $hint)) { + $env:Path = "$env:Path;$hint" + } + } + + $resolved = Resolve-Interpreter -Candidates $package.Candidates + if ($resolved) { + Write-Host "$($package.Name) is available at $resolved" + } else { + Write-Host "::warning::$($package.Name) installed but still does not resolve on PATH." + } + } + + $global:LASTEXITCODE = 0 +} + function Initialize-MicroVmHost { # Staged next to wxc-exec.exe by the --features microvm build, so their # absence means a broken artifact rather than a host problem. Snapshots are @@ -314,6 +586,13 @@ $BinaryDirectory = (Resolve-Path $BinaryDirectory).Path Write-Host "Preparing Windows host for backend '$Backend' using $BinaryDirectory" +# Run for every backend: this is host inventory, not a backend prerequisite. +# The winget repair comes first so the packaged-tooling install below can use +# it, and the inventory reports the state after both. +Repair-Winget +Install-PackagedTooling +Assert-WorkloadInterpreters + switch ($Backend) { 'process-t3' { Initialize-ProcessContainerHost } 'microvm' { Initialize-MicroVmHost } diff --git a/tests/scripts/run_ci_backend_tests.ps1 b/scripts/ci/run_backend_validation_tests.ps1 similarity index 67% rename from tests/scripts/run_ci_backend_tests.ps1 rename to scripts/ci/run_backend_validation_tests.ps1 index 3770f6390..c3e33b980 100644 --- a/tests/scripts/run_ci_backend_tests.ps1 +++ b/scripts/ci/run_backend_validation_tests.ps1 @@ -31,6 +31,8 @@ param( $ErrorActionPreference = 'Stop' $scriptRoot = Split-Path -Parent $MyInvocation.MyCommand.Path +$repoRoot = Split-Path -Parent (Split-Path -Parent $scriptRoot) +$testScriptRoot = Join-Path $repoRoot 'tests\scripts' $binaryDirectoryPath = (Resolve-Path -LiteralPath $BinaryDirectory).Path $wxc = Join-Path $binaryDirectoryPath 'wxc-exec.exe' @@ -84,6 +86,14 @@ function Invoke-TestScript { Assert-File -Path $wxc function Invoke-ProcessContainerTests { + # Returns the harness exit code rather than throwing, so a caller running + # more than one suite can report both results instead of stopping at the + # first failure. The suite talks to the operator through Write-Host (which + # Out-Null does not touch), so discarding the success stream keeps the + # return value a scalar even if a phase leaks a stray object. + [OutputType([int])] + param() + # The existing harness expects separate debug and release layouts. CI # intentionally tests one release artifact, so stage it in both slots. $debugDirectory = Join-Path $binaryDirectoryPath 'debug' @@ -97,7 +107,7 @@ function Invoke-ProcessContainerTests { Copy-Item -LiteralPath $uiProbe -Destination (Join-Path $debugDirectory 'wxc-ui-probe.exe') -Force Copy-Item -LiteralPath $uiProbe -Destination (Join-Path $releaseDirectory 'wxc-ui-probe.exe') -Force - $script = Join-Path $scriptRoot 'WinProcessContainer-Tests.ps1' + $script = Join-Path $testScriptRoot 'WinProcessContainer-Tests.ps1' # -KeepArtifacts stops the harness deleting its scratch tree on a clean # run, so a passing job still uploads its per-test logs and configs. # Skip build and Cargo phases because this job consumes a previously @@ -120,28 +130,52 @@ function Invoke-ProcessContainerTests { -UiProbeDebug (Join-Path $debugDirectory 'wxc-ui-probe.exe') ` -UiProbeRelease (Join-Path $releaseDirectory 'wxc-ui-probe.exe') ` -KeepArtifacts ` - -Phases $phases - if ($LASTEXITCODE -ne 0) { - throw "Process Container tests failed with exit code $LASTEXITCODE." - } + -Phases $phases | Out-Null + return $LASTEXITCODE +} + +function Invoke-T3WorkloadTests { + [OutputType([int])] + param() + + $script = Join-Path $testScriptRoot 'T3-Workloads.ps1' + # -Wxc is required: the script's default points at a debug build that does + # not exist in a CI artifact. -KeepArtifacts preserves the per-workload + # logs and configs on a clean run so a passing job still uploads them. + # -GrantDriveRoot lets the pwsh/git workloads resolve their working + # directory's ancestor chain; it rewrites ACLs across the system drive, + # which is why the script leaves it off by default and only a disposable + # CI runner opts in. Temporary until pwsh 7.7 leaves preview. + $global:LASTEXITCODE = 0 + & $script -Wxc $wxc -KeepArtifacts -GrantDriveRoot | Out-Null + return $LASTEXITCODE } Redirect-TempToRunnerTemp switch ($Backend) { 'process-t1' { - Invoke-ProcessContainerTests + $primitives = Invoke-ProcessContainerTests + if ($primitives -ne 0) { + throw "Process Container tests failed with exit code $primitives." + } } 'process-t3' { - Invoke-ProcessContainerTests + # Run both suites before reporting. Stopping at the first failure would + # hide the other suite's result, costing an extra nightly run to triage. + $primitives = Invoke-ProcessContainerTests + $workloads = Invoke-T3WorkloadTests + if ($primitives -ne 0 -or $workloads -ne 0) { + throw "process-t3 tests failed (primitives exit=$primitives, workloads exit=$workloads)." + } } 'isolation-session' { - Invoke-TestScript -Path (Join-Path $scriptRoot 'run_isolation_session_tests.ps1') -Arguments @{ + Invoke-TestScript -Path (Join-Path $testScriptRoot 'run_isolation_session_tests.ps1') -Arguments @{ WxcExePath = $wxc } } 'windows-sandbox' { - Invoke-TestScript -Path (Join-Path $scriptRoot 'run_windows_sandbox_one_shot_tests.ps1') -Arguments @{ + Invoke-TestScript -Path (Join-Path $testScriptRoot 'run_windows_sandbox_one_shot_tests.ps1') -Arguments @{ BinDir = $binaryDirectoryPath } } @@ -150,12 +184,12 @@ switch ($Backend) { if ($Architecture -ne 'x64') { throw 'The existing WSLC test harness is not architecture-portable yet.' } - Invoke-TestScript -Path (Join-Path $scriptRoot 'run_wslc_all_tests.ps1') -Arguments @{ + Invoke-TestScript -Path (Join-Path $testScriptRoot 'run_wslc_all_tests.ps1') -Arguments @{ WxcExecPath = $wxc } } 'microvm' { - Invoke-TestScript -Path (Join-Path $scriptRoot 'run_microvm_tests.ps1') -Arguments @{ + Invoke-TestScript -Path (Join-Path $testScriptRoot 'run_microvm_tests.ps1') -Arguments @{ BinDir = $binaryDirectoryPath } } diff --git a/tests/scripts/run_ci_backend_tests.sh b/scripts/ci/run_backend_validation_tests.sh similarity index 92% rename from tests/scripts/run_ci_backend_tests.sh rename to scripts/ci/run_backend_validation_tests.sh index 2785015a7..a82734198 100644 --- a/tests/scripts/run_ci_backend_tests.sh +++ b/scripts/ci/run_backend_validation_tests.sh @@ -18,6 +18,7 @@ backend="$1" binary_directory="$(cd "$2" && pwd)" script_root="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" repo_root="$(cd "$script_root/../.." && pwd)" +test_script_root="$repo_root/tests/scripts" release_directory="$repo_root/src/target/release" case "$backend" in @@ -43,7 +44,7 @@ case "$backend" in # Strict: a skip on a provisioned runner means a prerequisite vanished, # not that the assertion held. Without this the job goes green having # verified none of the directional enforcement. - MXC_BWRAP_TESTS_REQUIRE_EXECUTION=1 bash "$script_root/run_bwrap_all_tests.sh" + MXC_BWRAP_TESTS_REQUIRE_EXECUTION=1 bash "$test_script_root/run_bwrap_all_tests.sh" # The inbound chain test needs real host CAP_NET_ADMIN to read the # sandbox's network namespace and inject a peer into it, so the non-root # suite above can only report it as skipped. Invoking it separately here @@ -59,7 +60,7 @@ case "$backend" in # the assertion passed. Translate it into an explicit failure so the # inbound guarantee can never be reported as verified without running. inbound_status=0 - sudo -n bash "$script_root/run_bwrap_inbound_deny_test.sh" || inbound_status=$? + sudo -n bash "$test_script_root/run_bwrap_inbound_deny_test.sh" || inbound_status=$? if [[ $inbound_status -eq 77 ]]; then echo "The Bubblewrap inbound default-deny test skipped for a missing prerequisite;" \ "prepare-linux-host.sh should have installed slirp4netns, nsenter, iptables," \ @@ -77,7 +78,7 @@ case "$backend" in # A skip here means a prerequisite disappeared on a runner provisioned # to execute this suite, so turn it into a failure rather than a # vacuously green gate. - MXC_LXC_TESTS_REQUIRE_EXECUTION=1 bash "$script_root/run_lxc_all_tests.sh" + MXC_LXC_TESTS_REQUIRE_EXECUTION=1 bash "$test_script_root/run_lxc_all_tests.sh" ;; seatbelt) test -x "$binary_directory/mxc-exec-mac" diff --git a/scripts/ci/validation-test-matrix.json b/scripts/ci/validation-test-matrix.json index f5f7704af..d2f00a454 100644 --- a/scripts/ci/validation-test-matrix.json +++ b/scripts/ci/validation-test-matrix.json @@ -71,7 +71,7 @@ }, { "id": "windows-canary", - "displayName": "Windows canary", + "displayName": "Windows Canary", "family": "windows", "architectures": { "x64": { @@ -79,9 +79,7 @@ "artifact": "wxc-binaries-x86_64-pc-windows-msvc", "pool": "", "backends": [ - "process-t1", "process-t3", - "isolation-session", "wslc", "windows-sandbox", "microvm", @@ -93,9 +91,7 @@ "artifact": "wxc-binaries-aarch64-pc-windows-msvc", "pool": "", "backends": [ - "process-t1", "process-t3", - "isolation-session", "wslc", "windows-sandbox" ] @@ -325,12 +321,7 @@ } } ], - "backendDelayedStart": [ - { - "backend": "wslc", - "seconds": 45 - } - ], + "backendDelayedStart": [], "triggers": { "pr": [], "nightly": [ diff --git a/tests/scripts/README.md b/tests/scripts/README.md index 62887713b..88302976c 100644 --- a/tests/scripts/README.md +++ b/tests/scripts/README.md @@ -41,6 +41,8 @@ Linux / macOS (`.sh`): | `run_windows_sandbox_one_shot_tests.ps1` | Windows Sandbox one-shot E2E suite (fresh disposable VM per test) | Windows Sandbox enabled | | `run_windows_sandbox_state_aware_tests.ps1` | Windows Sandbox state-aware lifecycle E2E (single VM held across provision/start/exec*/stop/deprovision) | Windows Sandbox enabled | | `run_processcontainer_proxy_tests.ps1` | Process container proxy tests | `wxc-exec.exe` | +| `WinProcessContainer-Tests.ps1` | Process container (AppContainer / BaseContainer) primitives suite — tier probes, rw/ro/denied matrix, UI mitigations, DACL restore, crash recovery | `wxc-exec.exe`, `wxc-ui-probe.exe` | +| `T3-Workloads.ps1` | Real workloads (pwsh, git, node, python, cmd) on top of the T3 primitives. A missing interpreter is reported as a skip, not a failure | `wxc-exec.exe`; `pwsh` / `git` / `node` / `python` each optional, gating their own cases | | `run_on_repeat.ps1` | Stress test (loops core tests) | `wxc-exec.exe` | ### Linux suites @@ -69,21 +71,23 @@ these dispatchers, which map a matrix backend id to the suites above: | Dispatcher | Platforms | Backend ids | |------------|-----------|-------------| -| `run_ci_backend_tests.ps1` | Windows | `process-t1`, `process-t3`, `isolation-session`, `windows-sandbox`, `wslc`, `microvm`, `hyperlight` | -| `run_ci_backend_tests.sh` | Linux, macOS | `bubblewrap`, `lxc`, `seatbelt`, `microvm`, `hyperlight` | +| `scripts/ci/run_backend_validation_tests.ps1` | Windows | `process-t1`, `process-t3`, `isolation-session`, `windows-sandbox`, `wslc`, `microvm`, `hyperlight` | +| `scripts/ci/run_backend_validation_tests.sh` | Linux, macOS | `bubblewrap`, `lxc`, `seatbelt`, `microvm`, `hyperlight` | Pass the backend id exactly as it appears in the catalog — there is no separate handler name. Ids that share a suite have their own case in the dispatcher: `process-t1` and `process-t3` both run `WinProcessContainer-Tests.ps1`, which determines the tier it expects from the host's own `wxc-exec --probe`. +`process-t3` additionally runs `T3-Workloads.ps1`; both suites run even if the +first one fails, and the job reports their exit codes together. ```powershell -tests\scripts\run_ci_backend_tests.ps1 -Backend process-t1 ` +scripts\ci\run_backend_validation_tests.ps1 -Backend process-t1 ` -BinaryDirectory -Architecture x64 ``` ```bash -tests/scripts/run_ci_backend_tests.sh bubblewrap +scripts/ci/run_backend_validation_tests.sh bubblewrap ``` A backend with no wired suite exits non-zero on purpose, so accidentally diff --git a/tests/scripts/T3-Workloads.ps1 b/tests/scripts/T3-Workloads.ps1 index 82e832029..2e9a614d2 100644 --- a/tests/scripts/T3-Workloads.ps1 +++ b/tests/scripts/T3-Workloads.ps1 @@ -21,6 +21,15 @@ param( # -Wxc explicitly if the layout differs. [string]$Wxc = (Join-Path (Split-Path -Parent (Split-Path -Parent $PSScriptRoot)) 'src\target\debug\wxc-exec.exe'), [string]$ScratchRoot = (Join-Path $env:TEMP 'mxc-t3-workloads'), + # Grant read-only access to the drive root (C:\) for the pwsh/git + # workloads (W4, W5, W17). Those interpreters resolve the ENTIRE + # ancestor chain of the working directory at startup, so granting the + # leaf alone is not enough. Off by default -- see the TEMPORARY note + # above `Get-DriveRootGrant` for the cost of turning it on. + [switch]$GrantDriveRoot, + # Structured results. Lives in $env:TEMP but OUTSIDE $ScratchRoot so + # `Initialize-Scratch`'s recursive nuke can never take it with it. + [string]$ResultsJson = (Join-Path $env:TEMP 'T3-Workloads.results.json'), # Subset of workloads to run. Default: all nineteen. [int[]] $Run = @(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19), # Add a few extra "kitchen sink" RO grants (TEMP, LOCALAPPDATA, ...) @@ -43,25 +52,40 @@ function Record-Workload { param( [Parameter(Mandatory)] [string]$Id, [Parameter(Mandatory)] [string]$Name, - [Parameter(Mandatory)] [bool]$Pass, + [bool]$Pass = $true, + # Visual/semantic status. When omitted, derived from -Pass for back- + # compat (pass/fail). 'skip' = the workload's interpreter isn't on this + # host, so nothing was proven either way. A skip does not fail the run, + # but it renders distinctly so absent coverage is never a green PASS. + [ValidateSet('pass', 'fail', 'skip')] [string]$Status, [int]$ExitCode = 0, [string]$Detail = '', [string]$Stderr = '' ) + if (-not $PSBoundParameters.ContainsKey('Status')) { + $Status = if ($Pass) { 'pass' } else { 'fail' } + } else { + # Keep the boolean consistent for downstream logic: only 'fail' fails. + $Pass = ($Status -ne 'fail') + } $entry = [pscustomobject]@{ Id = $Id Name = $Name Pass = $Pass + Status = $Status ExitCode = $ExitCode Detail = $Detail StderrTop = ($Stderr -split "`r?`n" | Select-Object -First 3) -join ' / ' } $Script:Results.Add($entry) | Out-Null - $tag = if ($Pass) { '[PASS]' } else { '[FAIL]' } - $color = if ($Pass) { 'Green' } else { 'Red' } + $tag, $color = switch ($Status) { + 'pass' { '[PASS]', 'Green' } + 'fail' { '[FAIL]', 'Red' } + 'skip' { '[SKIP]', 'Yellow' } + } Write-Host (" {0} {1} :: {2} (exit={3})" -f $tag, $Id, $Name, $ExitCode) -ForegroundColor $color if ($Detail) { Write-Host (" detail: {0}" -f $Detail) } - if (-not $Pass -and $entry.StderrTop) { + if ($Status -eq 'fail' -and $entry.StderrTop) { Write-Host (" stderr: {0}" -f $entry.StderrTop) -ForegroundColor DarkRed } } @@ -273,6 +297,49 @@ function Resolve-HostPaths { if ($script:PythonRoNeeded) { Write-Host (" -> auto-grant ReadOnly: {0}" -f $script:PythonRoNeeded) } } +# ----------------------------------------------------------------------- +# TEMPORARY: drive-root read-only grant for pwsh-driven workloads +# +# Opt-in via -GrantDriveRoot; off by default. +# +# pwsh before 7.7, and git run underneath it, resolve the *whole ancestor +# chain* of the working directory at startup -- not just the directory +# itself. The policy only grants the leaf (`$ScratchRoot\rw`), so every +# segment above it is denied and the interpreter fails before reaching +# the behaviour these tests exist to check: +# +# W4/W5 git: "Unable to read current working directory: Permission denied" +# W17 pwsh falls back to C:\ as its location, then Set-Location is +# denied at the first ungranted ancestor. +# +# Granting the drive root covers every segment at once, and is what MXC's +# own launch diagnostic recommends. +# +# COST, and why this is a switch rather than the default: at T3 a policy +# path becomes an inheritable ACE (`filesystem_dacl.rs` sets +# inheritable = is_dir()), so naming C:\ propagates an ACL rewrite across +# the whole system drive on every run, and again in reverse on teardown. +# That is acceptable on a disposable CI runner and unpleasant on a +# developer box, which is why the caller has to ask for it. +# +# REMOVE the switch, the grant, and the warning once pwsh 7.7+ is out of +# preview and baked into the validation images. +# ----------------------------------------------------------------------- + +# The readonlyPaths entry for the pwsh/git workloads: the drive root when +# -GrantDriveRoot is set, otherwise nothing. +function Get-DriveRootGrant { + if (-not $GrantDriveRoot) { return @() } + $root = if ($env:SystemDrive) { "$env:SystemDrive\" } else { 'C:\' } + return @($root) +} + +function Write-DriveRootGrantWarning { + param([Parameter(Mandatory)] [string]$Id) + if (-not $GrantDriveRoot) { return } + Write-Host (" [{0}] WARNING: granting read-only {1} so pwsh/git can resolve the working directory's ancestor chain. TEMPORARY -- remove once pwsh 7.7 ships out of preview." -f $Id, ((Get-DriveRootGrant) -join ', ')) -ForegroundColor Yellow +} + # ----------------------------------------------------------------------- # Workloads # ----------------------------------------------------------------------- @@ -291,7 +358,7 @@ function W1-CmdTypeMarker { function W2-PwshReadFile { Section 'W2: pwsh -NoProfile -c "Get-Content marker.txt"' if (-not $script:PwshDir) { - Record-Workload -Id 'W2' -Name 'pwsh Get-Content' -Pass $false -Detail 'pwsh not found on PATH' + Record-Workload -Id 'W2' -Name 'pwsh Get-Content' -Status 'skip' -Detail 'pwsh not found on PATH' return } # PowerShell's install dir already grants ReadAndExecute to @@ -312,11 +379,11 @@ function W2-PwshReadFile { function W3-PwshGitVersion { Section 'W3: pwsh -NoProfile -c "git --version"' if (-not $script:PwshDir) { - Record-Workload -Id 'W3' -Name 'pwsh git --version' -Pass $false -Detail 'pwsh not found' + Record-Workload -Id 'W3' -Name 'pwsh git --version' -Status 'skip' -Detail 'pwsh not found' return } if (-not $script:GitDir) { - Record-Workload -Id 'W3' -Name 'pwsh git --version' -Pass $false -Detail 'git not found' + Record-Workload -Id 'W3' -Name 'pwsh git --version' -Status 'skip' -Detail 'git not found' return } # Git install dir also grants ReadAndExecute to ALL APPLICATION @@ -335,6 +402,42 @@ function W3-PwshGitVersion { -ExitCode $r.ExitCode -Detail "stdout=$($r.Stdout.Trim())" -Stderr $r.Stderr } +function Repair-FixtureOwnership { + param([string]$Dir) + # An elevated process's token hands out BUILTIN\Administrators as the + # *default owner* of everything it creates, so on a CI agent the repo this + # harness just built is owned by Administrators rather than by the user. + # + # git then refuses it: `detected dubious ownership`. git accepts an + # Administrators-owned repo only when the caller is *itself* an elevated + # administrator, and a contained process never is -- the AppContainer token + # drops that membership by construction. So the check fires inside the + # sandbox and cannot be satisfied there. + # + # That is an artifact of who built the fixture, not of containment: on a + # non-elevated dev box the same repo is user-owned and W4/W5 pass. Restamp + # the tree so the fixture is identical in both places and the workloads + # test git-in-a-sandbox rather than git's host ownership heuristic. + $user = [System.Security.Principal.WindowsIdentity]::GetCurrent().User + $items = @(Get-Item -LiteralPath $Dir -Force) + + @(Get-ChildItem -LiteralPath $Dir -Recurse -Force -ErrorAction SilentlyContinue) + $restamped = 0 + foreach ($item in $items) { + try { + $acl = Get-Acl -LiteralPath $item.FullName + if ($acl.GetOwner([System.Security.Principal.SecurityIdentifier]) -eq $user) { continue } + $acl.SetOwner($user) + Set-Acl -LiteralPath $item.FullName -AclObject $acl + $restamped++ + } catch { + Write-Host (" WARNING: could not set owner on {0}: {1}" -f $item.FullName, $_.Exception.Message) -ForegroundColor Yellow + } + } + if ($restamped -gt 0) { + Write-Host (" reowned {0} fixture object(s) to {1} (harness is running elevated)" -f $restamped, $user.Value) + } +} + function Initialize-Repo { param([string]$Dir) # Build a tiny standalone repo via real git, then we'll exercise @@ -353,12 +456,13 @@ function Initialize-Repo { } finally { Pop-Location } + Repair-FixtureOwnership -Dir $Dir } function W4-PwshGitStatus { Section 'W4: pwsh -NoProfile -c "cd ; git status"' if (-not $script:PwshDir -or -not $script:GitDir) { - Record-Workload -Id 'W4' -Name 'pwsh git status' -Pass $false -Detail 'pwsh or git not found' + Record-Workload -Id 'W4' -Name 'pwsh git status' -Status 'skip' -Detail 'pwsh or git not found' return } $repo = "$ScratchRoot\rw\repo" @@ -370,9 +474,11 @@ function W4-PwshGitStatus { # `C:\Users\...` metadata-access checks that the AppContainer SID # doesn't have grants for. $cmd = "pwsh.exe -NoProfile -NoLogo -Command `"& git -C '$repo' status --porcelain; exit `$LASTEXITCODE`"" + Write-DriveRootGrantWarning -Id 'W4' $cfg = New-Config -Name 'w4-git-status' ` -CommandLine $cmd ` -ReadWrite @("$ScratchRoot\rw") ` + -ReadOnly (Get-DriveRootGrant) ` -Cwd "$ScratchRoot\rw" $log = "$ScratchRoot\log\w4.log" $r = Invoke-Workload -ConfigPath $cfg -LogPath $log -TimeoutSec 90 @@ -385,7 +491,7 @@ function W4-PwshGitStatus { function W5-PwshGitLog { Section 'W5: pwsh -NoProfile -c "cd ; git log --oneline -n 10"' if (-not $script:PwshDir -or -not $script:GitDir) { - Record-Workload -Id 'W5' -Name 'pwsh git log' -Pass $false -Detail 'pwsh or git not found' + Record-Workload -Id 'W5' -Name 'pwsh git log' -Status 'skip' -Detail 'pwsh or git not found' return } $repo = "$ScratchRoot\rw\repo" @@ -394,9 +500,11 @@ function W5-PwshGitLog { Initialize-Repo -Dir $repo } $cmd = "pwsh.exe -NoProfile -NoLogo -Command `"& git -C '$repo' log --oneline -n 10; exit `$LASTEXITCODE`"" + Write-DriveRootGrantWarning -Id 'W5' $cfg = New-Config -Name 'w5-git-log' ` -CommandLine $cmd ` -ReadWrite @("$ScratchRoot\rw") ` + -ReadOnly (Get-DriveRootGrant) ` -Cwd "$ScratchRoot\rw" $log = "$ScratchRoot\log\w5.log" $r = Invoke-Workload -ConfigPath $cfg -LogPath $log -TimeoutSec 90 @@ -412,7 +520,7 @@ function W5-PwshGitLog { function W6-PwshListDir { Section 'W6: pwsh Get-ChildItem on rw directory' if (-not $script:PwshDir) { - Record-Workload -Id 'W6' -Name 'pwsh Get-ChildItem' -Pass $false -Detail 'pwsh not found' + Record-Workload -Id 'W6' -Name 'pwsh Get-ChildItem' -Status 'skip' -Detail 'pwsh not found' return } $cmd = "pwsh.exe -NoProfile -NoLogo -Command `"Get-ChildItem -LiteralPath '$ScratchRoot\rw' | Select-Object -ExpandProperty Name; exit 0`"" @@ -428,7 +536,7 @@ function W6-PwshListDir { function W7-PwshInProcEval { Section 'W7: pwsh in-process script eval (math, env, pipeline)' if (-not $script:PwshDir) { - Record-Workload -Id 'W7' -Name 'pwsh in-proc eval' -Pass $false -Detail 'pwsh not found' + Record-Workload -Id 'W7' -Name 'pwsh in-proc eval' -Status 'skip' -Detail 'pwsh not found' return } $script = '$x = 6 * 7; Write-Output "answer=$x"; 1..3 | ForEach-Object { Write-Output "iter=$_" }; exit 0' @@ -445,7 +553,7 @@ function W7-PwshInProcEval { function W8-PwshSpawnCmd { Section 'W8: pwsh spawning cmd /c (no NUL)' if (-not $script:PwshDir) { - Record-Workload -Id 'W8' -Name 'pwsh spawn cmd' -Pass $false -Detail 'pwsh not found' + Record-Workload -Id 'W8' -Name 'pwsh spawn cmd' -Status 'skip' -Detail 'pwsh not found' return } # No NUL redirects in the child — just a one-line echo. @@ -462,7 +570,7 @@ function W8-PwshSpawnCmd { function W9-PwshWriteReadRoundTrip { Section 'W9: pwsh Set-Content + Get-Content round-trip' if (-not $script:PwshDir) { - Record-Workload -Id 'W9' -Name 'pwsh write+read' -Pass $false -Detail 'pwsh not found' + Record-Workload -Id 'W9' -Name 'pwsh write+read' -Status 'skip' -Detail 'pwsh not found' return } $target = "$ScratchRoot\rw\w9-out.txt" @@ -480,7 +588,7 @@ function W9-PwshWriteReadRoundTrip { function W10-PwshDotNetIo { Section 'W10: pwsh .NET [System.IO.File]::ReadAllText' if (-not $script:PwshDir) { - Record-Workload -Id 'W10' -Name 'pwsh .NET IO' -Pass $false -Detail 'pwsh not found' + Record-Workload -Id 'W10' -Name 'pwsh .NET IO' -Status 'skip' -Detail 'pwsh not found' return } $script = "Write-Output ([System.IO.File]::ReadAllText('$ScratchRoot\rw\marker.txt')); exit 0" @@ -504,7 +612,7 @@ function W10-PwshDotNetIo { function W11-NodeReadFile { Section 'W11: node -e fs.readFileSync(marker)' if (-not $script:NodeDir) { - Record-Workload -Id 'W11' -Name 'node read file' -Pass $false -Detail 'node not found on PATH' + Record-Workload -Id 'W11' -Name 'node read file' -Status 'skip' -Detail 'node not found on PATH' return } $markerFwd = ("$ScratchRoot\rw\marker.txt") -replace '\\','/' @@ -524,7 +632,7 @@ function W11-NodeReadFile { function W12-NodeEval { Section 'W12: node in-proc eval (math, loop)' if (-not $script:NodeDir) { - Record-Workload -Id 'W12' -Name 'node eval' -Pass $false -Detail 'node not found on PATH' + Record-Workload -Id 'W12' -Name 'node eval' -Status 'skip' -Detail 'node not found on PATH' return } $js = "let x=6*7;console.log('answer='+x);for(let i=1;i<=3;i++)console.log('iter='+i);" @@ -543,7 +651,7 @@ function W12-NodeEval { function W13-NodeRoundTrip { Section 'W13: node fs.writeFileSync + readFileSync round-trip' if (-not $script:NodeDir) { - Record-Workload -Id 'W13' -Name 'node write+read' -Pass $false -Detail 'node not found on PATH' + Record-Workload -Id 'W13' -Name 'node write+read' -Status 'skip' -Detail 'node not found on PATH' return } $targetFwd = ("$ScratchRoot\rw\w13-out.txt") -replace '\\','/' @@ -570,7 +678,7 @@ function W13-NodeRoundTrip { function W14-PyReadFile { Section 'W14: python -c open(marker).read()' if (-not $script:PythonExe) { - Record-Workload -Id 'W14' -Name 'python read file' -Pass $false -Detail 'python not found on PATH' + Record-Workload -Id 'W14' -Name 'python read file' -Status 'skip' -Detail 'python not found on PATH' return } $py = "import sys; sys.stdout.write(open(r'$ScratchRoot\rw\marker.txt').read())" @@ -589,7 +697,7 @@ function W14-PyReadFile { function W15-PyEval { Section 'W15: python in-proc eval (math, loop)' if (-not $script:PythonExe) { - Record-Workload -Id 'W15' -Name 'python eval' -Pass $false -Detail 'python not found on PATH' + Record-Workload -Id 'W15' -Name 'python eval' -Status 'skip' -Detail 'python not found on PATH' return } # `python -c` accepts ';' between simple statements but requires @@ -628,7 +736,7 @@ function Initialize-ChdirTarget { function W16-PyRoundTrip { Section 'W16: python open(w,write) + open(r,read) round-trip' if (-not $script:PythonExe) { - Record-Workload -Id 'W16' -Name 'python write+read' -Pass $false -Detail 'python not found on PATH' + Record-Workload -Id 'W16' -Name 'python write+read' -Status 'skip' -Detail 'python not found on PATH' return } $target = "$ScratchRoot\rw\w16-out.txt" @@ -662,13 +770,14 @@ function W16-PyRoundTrip { function W17-PwshSetLocation { Section 'W17: pwsh Set-Location into rw subdir' if (-not $script:PwshDir) { - Record-Workload -Id 'W17' -Name 'pwsh Set-Location' -Pass $false -Detail 'pwsh not found' + Record-Workload -Id 'W17' -Name 'pwsh Set-Location' -Status 'skip' -Detail 'pwsh not found' return } $target = Initialize-ChdirTarget $cmd = "pwsh.exe -NoProfile -NoLogo -Command `"Set-Location -LiteralPath '$target'; Get-ChildItem -Name; exit 0`"" + Write-DriveRootGrantWarning -Id 'W17' $cfg = New-Config -Name 'w17-pwsh-cd' -CommandLine $cmd ` - -ReadWrite @("$ScratchRoot\rw") -Cwd "$ScratchRoot\rw" + -ReadWrite @("$ScratchRoot\rw") -ReadOnly (Get-DriveRootGrant) -Cwd "$ScratchRoot\rw" $log = "$ScratchRoot\log\w17.log" $r = Invoke-Workload -ConfigPath $cfg -LogPath $log -TimeoutSec 60 $pass = ($r.ExitCode -eq 0) -and ($r.Stdout -match 'chdir-marker\.txt') @@ -679,7 +788,7 @@ function W17-PwshSetLocation { function W18-NodeChdir { Section 'W18: node process.chdir into rw subdir' if (-not $script:NodeDir) { - Record-Workload -Id 'W18' -Name 'node chdir' -Pass $false -Detail 'node not found' + Record-Workload -Id 'W18' -Name 'node chdir' -Status 'skip' -Detail 'node not found' return } $target = Initialize-ChdirTarget @@ -700,7 +809,7 @@ function W18-NodeChdir { function W19-PyChdir { Section 'W19: python os.chdir into rw subdir' if (-not $script:PythonExe) { - Record-Workload -Id 'W19' -Name 'python chdir' -Pass $false -Detail 'python not found' + Record-Workload -Id 'W19' -Name 'python chdir' -Status 'skip' -Detail 'python not found' return } $target = Initialize-ChdirTarget @@ -749,13 +858,23 @@ catch { Write-Host '' Write-Host "ABORT: $_" -ForegroundColor Red Write-Host $_.ScriptStackTrace -ForegroundColor DarkRed - exit 2 + # ABORT = FAIL test. + Record-Workload -Id 'ABORT' -Name 'harness aborted' -Status 'fail' -ExitCode 2 -Detail "$_" } finally { Section 'Summary' - $passed = @($Script:Results | Where-Object { $_.Pass }) - $failed = @($Script:Results | Where-Object { -not $_.Pass }) - Write-Host ("Total: {0} Passed: {1} Failed: {2}" -f $Script:Results.Count, $passed.Count, $failed.Count) + $passed = @($Script:Results | Where-Object { $_.Status -eq 'pass' }) + $failed = @($Script:Results | Where-Object { $_.Status -eq 'fail' }) + $skipped = @($Script:Results | Where-Object { $_.Status -eq 'skip' }) + Write-Host ("Total: {0} Passed: {1} Failed: {2} Skipped: {3}" -f ` + $Script:Results.Count, $passed.Count, $failed.Count, $skipped.Count) + if ($skipped.Count -gt 0) { + Write-Host '' + Write-Host 'Skipped (interpreter not available on this host):' -ForegroundColor Yellow + foreach ($r in $skipped) { + Write-Host (" [{0}] {1} :: {2}" -f $r.Id, $r.Name, $r.Detail) -ForegroundColor Yellow + } + } if ($failed.Count -gt 0) { Write-Host '' Write-Host 'Failures:' -ForegroundColor Red @@ -765,8 +884,40 @@ finally { if ($r.StderrTop) { Write-Host (" stderr: {0}" -f $r.StderrTop) -ForegroundColor DarkRed } } } + + # Structured results for programmatic consumption, mirroring the shape + # WinProcessContainer-Tests.ps1 writes. Every step here is guarded: this + # runs in `finally`, so an unhandled throw would skip the exit-code line + # below and report a bogus result. CIM in particular is unavailable on + # locked-down hosts. + try { + $osInfo = Get-CimInstance Win32_OperatingSystem -ErrorAction Stop + $osCaption = [string]$osInfo.Caption + $osBuild = [string]$osInfo.BuildNumber + } catch { + $osCaption = 'unknown' + $osBuild = 'unknown' + } + try { + $summary = [pscustomobject]@{ + timestamp = (Get-Date).ToString('o') + host = $env:COMPUTERNAME + os = $osCaption + osBuild = $osBuild + total = $Script:Results.Count + passed = $passed.Count + failed = $failed.Count + skipped = $skipped.Count + results = $Script:Results + } + ($summary | ConvertTo-Json -Depth 6) | Out-File -LiteralPath $ResultsJson -Encoding utf8 -Force + } catch { + Write-Host "warning: could not write JSON results: $_" -ForegroundColor Yellow + } + Write-Host '' Write-Host ("Scratch / logs: {0}" -f $ScratchRoot) + Write-Host ("JSON summary: {0}" -f $ResultsJson) if (-not $KeepArtifacts -and $failed.Count -eq 0 -and $passed.Count -gt 0 -and (Test-Path $ScratchRoot)) { # Re-validate before deletion — `Assert-SafeScratchRoot` ran # at the start of the suite, but the variable could in @@ -775,4 +926,7 @@ finally { Assert-SafeScratchRoot Remove-Item -Recurse -Force -LiteralPath $ScratchRoot -ErrorAction SilentlyContinue } + # A run that recorded nothing but skips proved nothing, so it is not a + # pass — otherwise a host missing every interpreter reports green. + if ($failed.Count -gt 0 -or $passed.Count -eq 0) { exit 1 } else { exit 0 } } diff --git a/tests/scripts/WinProcessContainer-Tests.ps1 b/tests/scripts/WinProcessContainer-Tests.ps1 index 80598afff..957cc5c34 100644 --- a/tests/scripts/WinProcessContainer-Tests.ps1 +++ b/tests/scripts/WinProcessContainer-Tests.ps1 @@ -253,8 +253,15 @@ function Format-VerdictSummary { function Test-Preflight { Section 'Pre-flight' - $os = Get-CimInstance -ClassName Win32_OperatingSystem - Write-Host ("OS: {0} (build {1})" -f $os.Caption, $os.BuildNumber) + # Informational banner only — the load-bearing safety gate is the + # bfsCompiledIn check below. CIM is unavailable on some locked-down hosts, + # so don't let a cosmetic query abort the whole harness. + try { + $os = Get-CimInstance -ClassName Win32_OperatingSystem -ErrorAction Stop + Write-Host ("OS: {0} (build {1})" -f $os.Caption, $os.BuildNumber) + } catch { + Write-Host ("OS: unknown (CIM unavailable: {0})" -f $_.Exception.Message.Trim()) + } $bfsPath = Join-Path $env:SystemRoot 'System32\bfscfg.exe' $bfsPresent = Test-Path $bfsPath @@ -1688,20 +1695,31 @@ finally { } } - # Structured JSON for programmatic consumption. - $summary = [pscustomobject]@{ - timestamp = (Get-Date).ToString('o') - host = $env:COMPUTERNAME - os = (Get-CimInstance Win32_OperatingSystem).Caption - osBuild = (Get-CimInstance Win32_OperatingSystem).BuildNumber - total = $pass + $fail + $skip + $warn - passed = $pass - failed = $fail - skipped = $skip - warnings = $warn - results = $Script:Results + # Structured JSON for programmatic consumption. Guarded end to end: this + # runs in `finally`, so an unhandled throw here would skip the exit-code + # line at the bottom and hand the CI dispatcher a bogus result. CIM is + # unavailable on locked-down hosts. + try { + $osInfo = Get-CimInstance Win32_OperatingSystem -ErrorAction Stop + $osCaption = [string]$osInfo.Caption + $osBuild = [string]$osInfo.BuildNumber + } catch { + $osCaption = 'unknown' + $osBuild = 'unknown' } try { + $summary = [pscustomobject]@{ + timestamp = (Get-Date).ToString('o') + host = $env:COMPUTERNAME + os = $osCaption + osBuild = $osBuild + total = $pass + $fail + $skip + $warn + passed = $pass + failed = $fail + skipped = $skip + warnings = $warn + results = $Script:Results + } ($summary | ConvertTo-Json -Depth 6) | Out-File -LiteralPath $ResultsJson -Encoding utf8 -Force } catch { Write-Host "warning: could not write JSON results: $_" -ForegroundColor Yellow