What Happened
Three independent findings landed on the same day across a 46-repository fleet on Pest 5. Individually each looked like bad luck. Together they are one shape: a Pest 5 run that forks produces a different, quieter answer than the same run serial — and in two of the three the wrong answer is the more flattering one.
I am filing them together deliberately, because the pattern is the point. Each is separable and carries its own evidence.
One of the three has since been confirmed and fixed by a maintainer, which is why I think the other two are worth a look rather than a shrug.
1 — A parallel coverage run loses a line that is demonstrably covered
Status: reproducible in place, deterministic, not yet minimised.
|
|
| serial |
pest --coverage --min=100 --exclude-group=browser → 100.0% |
| parallel |
pest --coverage --min=100 --exclude-group=browser --parallel --processes=4 → 99.9% |
| the line |
EmailTwoFactorChallengeController line 197, reported uncovered |
| repeats |
same line, every parallel run |
| control |
on Pest 4, this exact parallel command was green |
Same tree, same tests, same driver settings (PCOV, XDEBUG_MODE=off exported as an env var so workers inherit it — not -d). The only variable is --parallel, and php-code-coverage moving 12.5.7 → 14.2.3 with the major.
It is not a threshold artefact: serial says the line is covered, parallel says it is not, and it picks the same line each time. That reads like a coverage merge losing one worker's contribution rather than a flaky collection.
The workaround in place is to run coverage serially, which costs wall-clock and nothing else — but it means every repository that trusts --parallel --coverage on Pest 5 may be enforcing a floor against a number that is slightly too low, and would fail a --min=100 gate for a line that is fine.
I have not minimised this to a standalone package yet. If a smaller reproducer is what unblocks it, say so and I will build one — I did not want to sit on the field evidence while doing that.
2 — --mutate --parallel reports surviving mutants as killed
Status: hand-verified, and the wrong number is the higher one.
Same class, same tests, same command apart from --parallel:
|
MSI |
pest --mutate (serial) |
89% |
pest --mutate --parallel |
100% |
The survivors that vanish under --parallel are real: verified by hand against the serial run. So the parallel score is not noisier — it is inflated, in the direction nobody double-checks. A score that reads 100% is the one least likely to be questioned.
This is now a standing rule across the fleet: mutation runs serially, always, and parallelism is expressed as sharding at the container level (one serial --mutate per shard) rather than --parallel. That is a workaround for consumers; upstream it seems worth either fixing or refusing the flag combination outright, because the failure mode is a fabricated pass.
This is adjacent to #1316 but not the same direction: there, tested code is reported UNTESTED; here, untested code is reported killed.
3 — The type-coverage cache rewrote itself (already fixed — included as the confirmation)
pest --type-coverage --min=100, no --parallel, died with:
ParseError: syntax error, unexpected single-quoted string " => "
at vendor/pestphp/pest-plugin-type-coverage/.temp/v3.php:14
12▕ );baef33c5698f9926572068f087c' =>
13▕ array (
➜ 14▕ 0 => '/…/app/Models/SystemFile.php',
The cache file is torn mid array-literal: two writes interleaved. Stack trace named Pokio\Runtime\Fork\ThrowableCapsule::__unserialize(). Turning concurrency off at the command line does not escape it, because the plugin forks internally. And it is a race — the next run over the identical tree was green.
This one is already fixed, in pest-plugin-type-coverage v5.0.1 (fix: parallel corruption file). The old withinLock() tried flock(LOCK_EX|LOCK_NB) a hundred times over 100 ms and then, on giving up, ran the callback anyway — an unlocked write. Making the lock blocking closed it.
I include it because it is the same family and it was real, so it is some evidence that 1 and 2 are worth measuring rather than dismissing as environment.
How to reproduce
For 1 and 2: any suite large enough to have a coverage merge worth doing. Run each command twice, once with --parallel and once without, and diff the reports — the difference is stable, not flaky.
Sample Code / Versions
Pest 5, php-code-coverage 14.2.3, PCOV, PHP 8.4/8.5, Linux containers (4 processes) and macOS both. Finding 3 on pest-plugin-type-coverage v5.0.0, fixed in v5.0.1.
What Happened
Three independent findings landed on the same day across a 46-repository fleet on Pest 5. Individually each looked like bad luck. Together they are one shape: a Pest 5 run that forks produces a different, quieter answer than the same run serial — and in two of the three the wrong answer is the more flattering one.
I am filing them together deliberately, because the pattern is the point. Each is separable and carries its own evidence.
One of the three has since been confirmed and fixed by a maintainer, which is why I think the other two are worth a look rather than a shrug.
1 — A parallel coverage run loses a line that is demonstrably covered
Status: reproducible in place, deterministic, not yet minimised.
pest --coverage --min=100 --exclude-group=browser→ 100.0%pest --coverage --min=100 --exclude-group=browser --parallel --processes=4→ 99.9%EmailTwoFactorChallengeControllerline 197, reported uncoveredSame tree, same tests, same driver settings (PCOV,
XDEBUG_MODE=offexported as an env var so workers inherit it — not-d). The only variable is--parallel, andphp-code-coveragemoving 12.5.7 → 14.2.3 with the major.It is not a threshold artefact: serial says the line is covered, parallel says it is not, and it picks the same line each time. That reads like a coverage merge losing one worker's contribution rather than a flaky collection.
The workaround in place is to run coverage serially, which costs wall-clock and nothing else — but it means every repository that trusts
--parallel --coverageon Pest 5 may be enforcing a floor against a number that is slightly too low, and would fail a--min=100gate for a line that is fine.I have not minimised this to a standalone package yet. If a smaller reproducer is what unblocks it, say so and I will build one — I did not want to sit on the field evidence while doing that.
2 —
--mutate --parallelreports surviving mutants as killedStatus: hand-verified, and the wrong number is the higher one.
Same class, same tests, same command apart from
--parallel:pest --mutate(serial)pest --mutate --parallelThe survivors that vanish under
--parallelare real: verified by hand against the serial run. So the parallel score is not noisier — it is inflated, in the direction nobody double-checks. A score that reads 100% is the one least likely to be questioned.This is now a standing rule across the fleet: mutation runs serially, always, and parallelism is expressed as sharding at the container level (one serial
--mutateper shard) rather than--parallel. That is a workaround for consumers; upstream it seems worth either fixing or refusing the flag combination outright, because the failure mode is a fabricated pass.This is adjacent to #1316 but not the same direction: there, tested code is reported UNTESTED; here, untested code is reported killed.
3 — The type-coverage cache rewrote itself (already fixed — included as the confirmation)
pest --type-coverage --min=100, no--parallel, died with:The cache file is torn mid array-literal: two writes interleaved. Stack trace named
Pokio\Runtime\Fork\ThrowableCapsule::__unserialize(). Turning concurrency off at the command line does not escape it, because the plugin forks internally. And it is a race — the next run over the identical tree was green.This one is already fixed, in
pest-plugin-type-coveragev5.0.1 (fix: parallel corruption file). The oldwithinLock()triedflock(LOCK_EX|LOCK_NB)a hundred times over 100 ms and then, on giving up, ran the callback anyway — an unlocked write. Making the lock blocking closed it.I include it because it is the same family and it was real, so it is some evidence that 1 and 2 are worth measuring rather than dismissing as environment.
How to reproduce
For 1 and 2: any suite large enough to have a coverage merge worth doing. Run each command twice, once with
--paralleland once without, and diff the reports — the difference is stable, not flaky.Sample Code / Versions
Pest 5,
php-code-coverage14.2.3, PCOV, PHP 8.4/8.5, Linux containers (4 processes) and macOS both. Finding 3 onpest-plugin-type-coveragev5.0.0, fixed in v5.0.1.