v5 CI: build e2e binaries once and tier the PM matrix (#291)

* Build e2e test binaries once and tier PM legs

The e2e matrix recompiled the CLI and its test binary in each of its
176 legs. e2e-build now compiles every CLI test target once per OS and
the legs run the downloaded binaries from the same checkout path. That
compile also replaces the --all-features --no-run pass in test and
test-release, so each of those compiles one feature set.

PR runs keep every named version boundary, the oldest and newest
release of each tool and every vlt era. The 31 middle e2e rows, 2
yarn-berry releases and 10 cargo toolchain x lock cells move to *-full
jobs that run on main pushes, a new nightly schedule and dispatch.

- e2e-docker (a subset of coverage-docker) runs nightly only.
- Dockerfile.base is built once per run and loaded by each docker leg.
- The hermetic maven/nuget crawl tests run in `test`; their rows and
  e2e_composer's are gone. e2e_safety_cargo_build rides cargo-vex.
- pdm-compat builds the capstone once, skips the 7 cells ci.yml runs,
  drops the Windows native rows (they never ran), and fails a cell
  whose bootstrap fails or whose rows all skip.
- vlt-compat install-proof leaves ci.yml's identical cells to it.
- npm/pnpm compatibility are path-filtered on PRs; the disarmed vlt
  serve watchdog runs daily instead of every 6 hours.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HK6zFsizxHRUtyxPvdhM4c

* Address review: env parity, filters, PR cells

- The legs that run test binaries directly set SOCKET_NO_CONFIG and
  SOCKET_NO_UPDATE_CHECK, which cargo's [env] gave `cargo test`.
- cargo 1.93.1 with its own lock (the pinned toolchain the removed
  e2e_safety_cargo_build rows ran) stays on PRs; 1.82.0 own-lock moves
  to the full tier.
- Poetry 2.0.1, the first lock 2.1 writer, stays on PRs.
- e2e-build gets 60 minutes on Windows.
- npm/pnpm filters also watch .cargo/config.toml and cache_env.rs.
- vlt install-proof notes a cell left entirely to ci.yml.
- pdm-compat saves its build cache from main only.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HK6zFsizxHRUtyxPvdhM4c

* Run the full tier on v5 pushes; review fixes

- ci.yml also runs on pushes to release/v5-prerelease, whose PRs skip
  the full tier and which has no nightly; e2e-docker runs there too.
- cargo 1.93.1 with its own lock runs on macOS and Windows on PRs,
  the cell the old e2e_safety_cargo_build rows ran there.
- e2e-build and pdm-compat print rendered compile errors
  (json-render-diagnostics).
- vlt-compat and pdm-compat also trigger on ci.yml changes, and the
  vlt dedupe only counts ci rows with the same test filter.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HK6zFsizxHRUtyxPvdhM4c

* Run the vlt agent get test in agent mode

get now defaults to hosted mode (5e5f5ed), so the real-vlt
get_and_remove leg ran a hosted get and found the installed copy
unpatched. It now passes --mode agent, like the other agent-mode
fixtures that commit updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R11RZRvYFL3fzmEnkFAU4A
(cherry picked from commit cc5f1b6)

* Give the nightly CI run its own concurrency group

A concurrency group holds one pending run. Sharing main's group let a
queued main push and the nightly cancel each other, and the nightly is
the only automatic run of e2e-docker.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HK6zFsizxHRUtyxPvdhM4c

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: mikolalysenko

.github/workflows/ci.yml

@@ -7,7 +7,28 @@ name: npm hosted/vendored compatibility
 # installs one pinned npm and runs them. See docs/testing/npm-compatibility.md.
 
 on:
+  # PRs: any crate source, but only the test files these capstones
+  # compile (a later `!` pattern excludes, a later plain one re-includes).
+  # Main pushes stay unfiltered.
   pull_request:
+    paths:
+      - '.github/actions/upload-artifact/**'
+      - 'Cargo.lock'
+      - 'Cargo.toml'
+      - 'rust-toolchain.toml'
+      - '.cargo/config.toml'
+      - '.github/workflows/npm-compatibility.yml'
+      - 'docs/testing/npm-compatibility.md'
+      - 'crates/**'
+      - '!crates/**/*.md'
+      - '!crates/socket-patch-node/**'
+      - '!crates/socket-patch-core/tests/**'
+      - '!crates/socket-patch-cli/tests/**'
+      - 'crates/socket-patch-cli/tests/vex_e2e_common/**'
+      - 'crates/socket-patch-cli/tests/e2e_redirect_npm_build.rs'
+      - 'crates/socket-patch-cli/tests/e2e_vendor_npm_build.rs'
+      - 'crates/socket-patch-cli/tests/npm_e2e_common/**'
+      - 'crates/socket-patch-cli/tests/common/cache_env.rs'
   push:
     branches: [main]
   workflow_dispatch:

.github/workflows/npm-compatibility.yml

@@ -1,10 +1,11 @@
 name: PDM patch compatibility
 
-# Native PDM installer matrix: builds the CLI once per OS, bootstraps pinned
-# PDM releases with uv, and runs `scripts/backtest-pdm.py` — hosted, vendored
-# and agent mode against the public urllib3 free patch, verifying the
-# INSTALLED bytes, lock/manifest stability, hash rejection and rollback. No
-# Socket API token is needed. See docs/testing/pdm-compatibility.md.
+# Native PDM installer matrix: builds the CLI and the capstone test binary
+# once per OS, bootstraps pinned PDM releases with uv, and runs
+# `scripts/backtest-pdm.py` — hosted, vendored and agent mode against the
+# public urllib3 free patch, verifying the INSTALLED bytes, lock/manifest
+# stability, hash rejection and rollback. No Socket API token is needed. See
+# docs/testing/pdm-compatibility.md.
 
 on:
   pull_request:
@@ -25,6 +26,8 @@ on:
       - 'crates/socket-patch-cli/tests/e2e_vex_build/main.rs'
       - 'crates/socket-patch-cli/tests/e2e_vex_build/pdm.rs'
       - 'crates/socket-patch-cli/tests/vex_pypi_real_common/**'
+      # The capstone skips the cells ci.yml's e2e rows run.
+      - '.github/workflows/ci.yml'
   push:
     branches: [main]
     paths:
@@ -60,7 +63,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        os: [ubuntu-latest, windows-latest, macos-latest]
+        os: [ubuntu-latest, macos-latest]
     runs-on: ${{ matrix.os }}
     timeout-minutes: 30
     steps:
@@ -70,13 +73,26 @@ jobs:
       - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
         with:
           key: pdm-compat
-      - run: cargo build --locked -p socket-patch-cli
+          save-if: ${{ github.ref == 'refs/heads/main' }}
+      - name: Compile the CLI and the capstone once
+        run: |
+          set -euo pipefail
+          cargo test --locked -p socket-patch-cli --test e2e_vex_build --no-run --message-format=json-render-diagnostics > target-build.json
+          python3 - <<'PY'
+          import json, pathlib, shutil
+          dest = pathlib.Path('target/pdm-e2e')
+          dest.mkdir(parents=True, exist_ok=True)
+          shutil.copy2('target/debug/socket-patch', dest / 'socket-patch')
+          for line in pathlib.Path('target-build.json').read_text().splitlines():
+              item = json.loads(line)
+              if item.get('target', {}).get('name') == 'e2e_vex_build' and item.get('executable'):
+                  shutil.copy2(item['executable'], dest / 'e2e_vex_build')
+          assert (dest / 'e2e_vex_build').is_file()
+          PY
       - uses: ./.github/actions/upload-artifact
         with:
           name: pdm-cli-${{ matrix.os }}
-          path: |
-            target/debug/socket-patch
-            target/debug/socket-patch.exe
+          path: target/pdm-e2e/
           if-no-files-found: error
           retention-days: 7
 
@@ -86,8 +102,10 @@ jobs:
       fail-fast: false
       matrix:
         # Every stable PDM major family (0.x, 1.x, 2.x) and each 2.x lock-format
-        # boundary, on Linux and Windows; macOS samples the ends of the range.
-        os: [ubuntu-latest, windows-latest]
+        # boundary on Linux; macOS samples the ends of the range. No Windows:
+        # the harness bootstraps PDM through a POSIX venv layout (bin/pdm), so
+        # every Windows cell skipped; backtest-pdm.py now fails such a cell.
+        os: [ubuntu-latest]
         pdm: ['0.12.3', '1.15.5', '2.0.3', '2.1.5', '2.3.4', '2.6.1', '2.7.4', '2.8.2', '2.9.3', '2.10.4', '2.11.2', '2.17.3', '2.20.1', '2.22.4', '2.25.9', '2.29.2']
         include:
           - { os: macos-latest, pdm: '0.12.3' }
@@ -152,25 +170,37 @@ jobs:
   # The hermetic Rust capstone (wiremock Socket API that also serves the
   # hosted wheel) over every PDM release the backtest covers: real hosted +
   # vendored flows ending in the manifest-less VEX matrix; refused lock
-  # formats (3.1, 4.0-4.2) must attest nothing.
+  # formats (3.1, 4.0-4.2) must attest nothing. The cells ci.yml's `e2e`
+  # job runs on every PR and main push are excluded here
+  # (scripts/tests/test_ci_e2e_tiers.py keeps the two lists in step).
   capstone:
+    needs: build
     strategy:
       fail-fast: false
       matrix:
         os: [ubuntu-latest, macos-latest]
         pdm: ['0.12.3', '1.0.0', '1.4.5', '1.8.5', '1.15.5', '2.0.3', '2.7.4', '2.8.2', '2.10.4', '2.11.2', '2.17.3', '2.20.1', '2.22.4', '2.25.9', '2.29.2']
+        exclude:
+          - {os: ubuntu-latest, pdm: '1.4.5'}
+          - {os: ubuntu-latest, pdm: '1.15.5'}
+          - {os: ubuntu-latest, pdm: '2.7.4'}
+          - {os: ubuntu-latest, pdm: '2.8.2'}
+          - {os: ubuntu-latest, pdm: '2.25.9'}
+          - {os: ubuntu-latest, pdm: '2.29.2'}
+          - {os: macos-latest, pdm: '2.29.2'}
     runs-on: ${{ matrix.os }}
     timeout-minutes: 30
     steps:
+      # The binaries resolve fixtures through the build job's checkout path,
+      # which is the same on every runner of one OS.
       - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
           persist-credentials: false
-      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
+      - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
         with:
-          key: pdm-vex-capstone
-          # Only main writes the cache: 30 PR matrix cells saving would churn
-          # the repo's 10 GiB budget (ci.yml's rust-cache note).
-          save-if: ${{ github.ref == 'refs/heads/main' }}
+          pattern: pdm-cli-${{ matrix.os }}*
+          merge-multiple: true
+          path: target/pdm-e2e
       - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
         with:
           python-version: '3.12'
@@ -179,4 +209,11 @@ jobs:
         env:
           SOCKET_PATCH_PDM_E2E_REQUIRED: '1'
           SOCKET_PATCH_PDM_E2E_VERSION: ${{ matrix.pdm }}
-        run: cargo test --locked -p socket-patch-cli --test e2e_vex_build -- 'pdm::' --ignored
+        run: |
+          set -euo pipefail
+          chmod +x target/pdm-e2e/*
+          mkdir -p target/debug target/tmp
+          cp target/pdm-e2e/socket-patch target/debug/socket-patch
+          cd crates/socket-patch-cli
+          export CARGO_MANIFEST_DIR="$PWD"
+          ../../target/pdm-e2e/e2e_vex_build 'pdm::' --ignored

.github/workflows/pdm-compatibility.yml

@@ -1,7 +1,26 @@
 name: pnpm hosted compatibility
 
 on:
+  # PRs: any crate source, but only the test files these capstones
+  # compile (a later `!` pattern excludes, a later plain one re-includes).
+  # Main pushes stay unfiltered.
   pull_request:
+    paths:
+      - '.github/actions/upload-artifact/**'
+      - 'Cargo.lock'
+      - 'Cargo.toml'
+      - 'rust-toolchain.toml'
+      - '.cargo/config.toml'
+      - '.github/workflows/pnpm-compatibility.yml'
+      - 'crates/**'
+      - '!crates/**/*.md'
+      - '!crates/socket-patch-node/**'
+      - '!crates/socket-patch-core/tests/**'
+      - '!crates/socket-patch-cli/tests/**'
+      - 'crates/socket-patch-cli/tests/vex_e2e_common/**'
+      - 'crates/socket-patch-cli/tests/e2e_redirect_pnpm_build.rs'
+      - 'crates/socket-patch-cli/tests/e2e_vendor_pnpm_build.rs'
+      - 'crates/socket-patch-cli/tests/common/cache_env.rs'
   push:
     branches: [main]
   workflow_dispatch:

.github/workflows/pnpm-compatibility.yml

@@ -53,6 +53,9 @@ on:
       - 'scripts/install-vlt.sh'
       - 'scripts/vlt-historical-integrity.json'
       - 'scripts/gen-vlt-collation-golden.mjs'
+      - 'scripts/ci-vlt-proof-suites.py'
+      - '.github/workflows/ci.yml'
+      - 'scripts/tests/test_ci_vlt_rows.py'
   push:
     branches: [main]
     paths:
@@ -89,6 +92,9 @@ on:
       - 'scripts/install-vlt.sh'
       - 'scripts/vlt-historical-integrity.json'
       - 'scripts/gen-vlt-collation-golden.mjs'
+      - 'scripts/ci-vlt-proof-suites.py'
+      - '.github/workflows/ci.yml'
+      - 'scripts/tests/test_ci_vlt_rows.py'
   schedule:
     # Nightly: vlt releases and the production service drift with no PR open.
     - cron: '17 4 * * *'
@@ -311,6 +317,8 @@ jobs:
           SOCKET_PATCH_VLT_E2E_STORE_LINKER: ${{ matrix.linker }}
           SOCKET_PATCH_VLT_E2E_CACHE_ROOT: ${{ matrix.cache_root }}
           VLT_SUITES: ${{ matrix.suites || 'e2e_redirect_vlt_build e2e_vendor_vlt_build mode_migration_vlt e2e_safety_vlt e2e_vlt' }}
+          MATRIX_OS: ${{ matrix.os }}
+          NODE_PIN: ${{ matrix.node }}
         run: |
           set -uo pipefail
           exe=''
@@ -326,6 +334,17 @@ jobs:
             mkdir -p "$SOCKET_PATCH_VLT_E2E_CACHE_ROOT"
           fi
           py=$(command -v python3 || command -v python)
+          # Cells ci.yml's e2e rows run identically (every PR, main push and
+          # nightly) are left to them; a dispatch runs every cell.
+          if [ "$GITHUB_EVENT_NAME" != workflow_dispatch ]; then
+            # shellcheck disable=SC2086 # VLT_SUITES is a word list
+            VLT_SUITES=$("$py" scripts/ci-vlt-proof-suites.py --os "$MATRIX_OS" --vlt "$SOCKET_PATCH_VLT_E2E_VERSION" \
+              --node "$NODE_PIN" --linker "${SOCKET_PATCH_VLT_E2E_STORE_LINKER:-}" \
+              --cache-root "${SOCKET_PATCH_VLT_E2E_CACHE_ROOT:-}" $VLT_SUITES | tr -d '\r') || exit 1
+          fi
+          if [ -z "$VLT_SUITES" ]; then
+            echo "::notice::every capstone of this cell runs in ci.yml's e2e rows; nothing left to run here"
+          fi
           status=0
           for suite in $VLT_SUITES; do
             echo "::group::$suite"

.github/workflows/vlt-compatibility.yml

@@ -12,10 +12,11 @@ name: vlt serve watchdog
 # Like installer-drift.yml it checks a deployed service, not the diff. It is
 # `continue-on-error` until the serve fix (`Cache-Control: no-transform`) is
 # verified in production; removing that line arms it (DESIGN §8.4, the depscan
-# rollout's last step).
+# rollout's last step). Until then it cannot alert, so it runs once a day to
+# record the probe; go back to every 6 hours when arming it.
 on:
   schedule:
-    - cron: '23 */6 * * *'
+    - cron: '23 4 * * *'
   workflow_dispatch:
 
 permissions:

.github/workflows/vlt-serve-watchdog.yml

@@ -10,7 +10,7 @@
 //!
 //! # Running
 //! ```sh
-//! cargo test -p socket-patch-cli --test e2e_maven -- --ignored
+//! cargo test -p socket-patch-cli --test e2e_maven
 //! ```
 
 use std::path::{Path, PathBuf};
@@ -99,7 +99,6 @@ async fn assert_proxy_served_scans(server: &MockServer, scans: usize) {
 
 /// Verify that `socket-patch scan` discovers artifacts in a fake Maven local repo.
 #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
-#[ignore = "opt-in maven crawl e2e; run with --ignored"]
 async fn scan_discovers_maven_artifacts() {
     let server = start_proxy().await;
     let proxy_url = server.uri();
@@ -226,7 +225,6 @@ async fn scan_discovers_maven_artifacts() {
 
 /// Verify that `socket-patch scan` discovers Gradle project artifacts.
 #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
-#[ignore = "opt-in maven crawl e2e; run with --ignored"]
 async fn scan_discovers_gradle_project_artifacts() {
     let server = start_proxy().await;
     let proxy_url = server.uri();

crates/socket-patch-cli/tests/e2e_maven.rs

@@ -10,7 +10,7 @@
 //!
 //! # Running
 //! ```sh
-//! cargo test -p socket-patch-cli --test e2e_nuget -- --ignored
+//! cargo test -p socket-patch-cli --test e2e_nuget
 //! ```
 
 use std::path::{Path, PathBuf};
@@ -179,7 +179,6 @@ async fn assert_proxy_served_scans(server: &MockServer, scans: usize) {
 
 /// Verify that `socket-patch scan` discovers packages in a fake global cache layout.
 #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
-#[ignore = "opt-in nuget crawl e2e; run with --ignored"]
 async fn scan_discovers_global_cache_packages() {
     let server = start_proxy().await;
     let proxy_url = server.uri();
@@ -247,7 +246,6 @@ async fn scan_discovers_global_cache_packages() {
 
 /// Verify that `socket-patch scan` discovers packages in a fake legacy packages/ layout.
 #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
-#[ignore = "opt-in nuget crawl e2e; run with --ignored"]
 async fn scan_discovers_legacy_packages() {
     let server = start_proxy().await;
     let proxy_url = server.uri();

crates/socket-patch-cli/tests/e2e_nuget.rs

@@ -105,8 +105,11 @@ the patched `urllib3/response.py` (git-blob SHA-256 matches the ledger
 `afterHash`), ordinary `pdm install` keeps the lock stable, a tampered hash is
 rejected, a relock-then-rescan keeps rollback invertible, and rollback restores
 the lock and `pyproject.toml` byte for byte. `.github/workflows/pdm-compatibility.yml`
-runs it on Linux, Windows and macOS across every PDM major family. The matrix
-needs no Socket API token (the `urllib3@1.26.18` patch is a free tier).
+runs it on Linux and macOS across every PDM major family. It does not run on
+Windows: the harness bootstraps PDM through a POSIX venv layout (`bin/pdm`), so
+every Windows cell used to skip, and a run whose cells all skip or whose PDM
+bootstrap fails is now an error. The matrix needs no Socket API token (the
+`urllib3@1.26.18` patch is a free tier).
 
 > **Note (v5.0):** the "refused vendored scan still writes a `.socket/manifest.json`
 > record" observation in the notes column below describes the 4.0.0 binary the run

docs/testing/pdm-compatibility.md

@@ -73,7 +73,8 @@ asserted and each named test or row exists.
   `install-proof` (every capstone on 31 Linux, 11 macOS and 15 Windows
   releases, the Node floors 22.22.0 / 22.13.0 / 22.7.0 / 22.0.0 with the
   collation golden, and the store linkers auto / hardlink / copy / unpack / a `/dev/shm`
-  cache root); `native` (the backtest against production, artifacts
+  cache root; a cell `ci.yml`'s `e2e` rows run identically is left to them,
+  see `scripts/ci-vlt-proof-suites.py`, except on dispatch); `native` (the backtest against production, artifacts
   `vlt-results-<os>-<vlt>` in depscan's capture `result.json` shape);
   `lock-diff` (the same cell's `vlt-lock.json` must be byte-identical on Linux,
   macOS and Windows); `matrix-coverage` (every era × suite × OS).