Flaky and misconfigured CI checks block dependency PRs

[mtrezza]

Issue

While processing a batch of ~15 dependency pull requests, CI flakiness — not the dependency changes themselves — was the dominant cost. Five distinct failure classes hit required status checks, each needing log analysis to distinguish from a genuine regression, and each costing a ~25 minute re-run. On a strict + dismiss_stale_reviews branch, every re-run also risks another PR merging first, which puts the PR behind, dismisses its approval and forces a full rebuild.

Two of these are broken tests rather than infrastructure flakes, and are worth fixing outright.

This issue is a tracking list; each item can be split into its own sub-issue.

Approach

1. Benchmarks — percentage threshold on sub-millisecond timings (required check)

.github/workflows/ci-performance.yml (~lines 214-236) flags a regression purely on relative change, with no absolute floor:

const change = ((prValue - baseValue) / baseValue * 100);
if (change > 50) { status = '❌ Much Slower'; hasRegression = true; }
else if (change > 25) { status = '⚠️ Slower'; hasRegression = true; }
...
if (hasRegression) { process.exitCode = 1; }

On benchmarks measured in fractions of a millisecond, ordinary runner noise clears 25% trivially. Observed on #10656, where the only entry over threshold was:

Benchmark	Baseline	PR	Change
Object.save (create)	0.52 ms	0.66 ms	+26.2%

A 0.14 ms difference failed a required check. The PR was a lock-file-only bump of baseline-browser-mapping, a dev-only transitive dependency that cannot affect runtime performance. The job passed unchanged on re-run.

Suggested fix: require an absolute delta floor (e.g. ignore changes below ~2-5 ms, or below some multiple of measured run-to-run variance) before applying the percentage thresholds. Alternatively run N iterations and compare medians.

2. Docker Build — QEMU linux/arm64 crash

.github/workflows/ci.yml builds platforms: linux/amd64, linux/arm64/v8. The emulated arm64 leg intermittently dies:

#20 [linux/arm64 build 6/6] RUN npm ci --omit=dev --ignore-scripts && cp -R node_modules prod_node_modules && npm ci && npm run build
#20 34.76 qemu: uncaught target signal 4 (Illegal instruction) - core dumped
#20 41.32 Illegal instruction (core dumped)
#20 ERROR: ... did not complete successfully: exit code: 132

The linux/amd64 leg completes normally in the same run. Exit code 132 is SIGILL under QEMU, not a build error. Observed on #10661, #10650, #10673 and #10674.

Not currently a required check, so it does not block merges, but it produces a persistent red signal that has to be triaged every time. Suggested fix: native arm64 runners if available, or retry the arm64 leg, or drop arm64 from PR builds and keep it only for release builds.

3. spec/index.spec.js:641 — should reload masterKey if ttl is set and expired

This test is genuinely broken, not flaky infrastructure:

const masterKeySpy = jasmine.createSpy()
  .and.returnValues(Promise.resolve('firstMasterKey'), Promise.resolve('secondMasterKey'));

await reconfigureServer({ masterKey: masterKeySpy, masterKeyTtl: 1 / 1000 }); // 1ms

With a 1 ms TTL the cached key is expired essentially always, so any additional internal loadMasterKey() call inflates the spy count. The stub supplies exactly two return values, so the third call returns undefined and both assertions fail together:

Expected spy unknown to have been called 2 times. It was called 4 times.
Expected undefined to equal 'secondMasterKey'.

Observed failing on two PRs with entirely unrelated diffs — an lru-cache change (PostgreSQL 18) and an undici lock-file bump (MongoDB 7). masterKeyCache is a plain object with its own expiresAt comparison in src/Config.js, so neither change can reach it.

Suggested fix: use a TTL large enough to be deterministic and advance time explicitly (or stub the clock) rather than relying on a 1 ms real-time expiry, and make the spy return a stable value rather than a fixed-length list.

4. Cancelled required jobs

PostgreSQL matrix jobs intermittently come back CANCELLED rather than passing or failing. Because they are required contexts, a cancelled job blocks merge exactly like a failure, but carries no diagnostic output.

Observed: PostgreSQL 16, PostGIS 3.5 on #10676; PostgreSQL 18, PostGIS 3.6 on #10674 and #10651. All passed on re-run.

Worth investigating whether these are runner preemptions, concurrency-group cancellations, or timeouts.

5. Network-dependent E2E tests in required jobs

Two specs call third-party services directly and fail on network conditions:

1) test validate_receipt endpoint should fail at appstore validation
   Error: Timeout - Async function did not complete within 20000ms

2) LineAdapter LineAdapter E2E Test should handle error when no code is provided
   Error: socket hang up  { code: 'ECONNRESET' }

Both hit external endpoints (Apple App Store receipt validation, LINE OAuth) from required matrix jobs, so an outage or slow response on someone else's infrastructure fails the build. Observed on #10651 (MongoDB 8, ReplicaSet); both passed on re-run.

Suggested fix: mock these endpoints, or move genuine E2E coverage to a separate non-required job.

6. codecov/project fails on nearly every PR

Not a required check (codecov/patch is), but it reports red on almost every dependency PR, adding noise that has to be checked and dismissed each time. Worth either making the threshold tolerant of small fluctuations or removing the status entirely if codecov/patch is the intended gate.

Tasks

• Benchmarks: add an absolute floor before applying percentage thresholds #10684
• Docker Build: address QEMU arm64 SIGILL crashes
• Fix should reload masterKey if ttl is set and expired
• Investigate cancelled PostgreSQL matrix jobs
• Remove third-party network calls from required test jobs
• Decide whether codecov/project should report on PRs

[parse-github-assistant]

🚀 Thanks for opening this issue!