Skip to content

fix: exclude mutating actions from the expectation retry loop - #249

Open
ibrasho wants to merge 1 commit into
pestphp:5.xfrom
ibrasho:fix/actions-not-awaitable
Open

ibrasho wants to merge 1 commit into
pestphp:5.xfrom
ibrasho:fix/actions-not-awaitable

Conversation

@ibrasho

@ibrasho ibrasho commented Aug 14, 2026

Copy link
Copy Markdown

What this fixes

AwaitableWebpage::__call() routes every page method through Execution::waitForExpectation(), which re-runs the whole call under a hard-coded 1s cap per attempt until the configured timeout runs out, then makes one final uncapped attempt. Every Playwright error surfaces as ExpectationFailedException, so a timeout is retried exactly like a failed assertion.

That is correct for an assertion and wrong for an action. A click changes the page it clicks. When the first attempt needs more than 1s to complete — routine on a 2-core CI runner with a heavy SPA page — the click has already fired and a dialog is open, but the attempt is judged failed. The retry then aims at a button the dialog's overlay now covers, which can never become actionable. The test spends the whole timeout plus one uncapped attempt and fails with Timeout Nms exceeded on a click that succeeded.

In practice this shows up as browser suites that are green locally going red on CI on a shifting set of tests (whichever click crossed 1s that run), and raising the timeout makes each failure take longer instead of fixing it. Related report: pestphp/pest#1511.

The change

typeSlowly is already excluded from the retry loop, with the comment "Retrying this action would append the value to what was already typed." This PR extends the same treatment to the other methods that repeat their effect when re-fired:

  • click / rightClick — the retry aims at a page the first attempt already changed
  • drag — the retry starts from an element the first attempt already moved
  • keys / press / pressAndWaitFor / withKeyDown — a re-sent Enter submits the form again; a multi-key sequence is replayed from the start
  • append — the identical hazard typeSlowly was excluded for

Deliberately NOT excluded: the state-setting inputs (fill, type, clear, check, uncheck, radio, select, attach, hover), where a retry is harmless, and every assertion, where retry-until-timeout is exactly right.

The excluded branch calls the method directly, so the action gets the full configured timeout in one attempt. That is also the stronger wait: Playwright's actionability check (attached → visible → stable → receives events → enabled) runs inside the one call, so it waits without ever firing twice.

Verification

The included regression test simulates a slow main thread: the button's first pointerdown busy-loops 1.2s, and the click shows an overlay covering the button.

  • Before this change: the test fails — the capped first attempt is judged a timeout after the click already fired, every retry aims at the covered button, and the run ends with a timeout. The page state at failure proves the double-fire hazard: window.downs === 1, overlay open.
  • After this change: the test passes in ~1s.
  • Full suite: 358 passed. pint and phpstan clean.

No signatures change. Configurations with a timeout ≤ 1000ms already bypass the loop entirely, so they see no behavior change.

If you would prefer a smaller first step, click + drag alone fixes the loudest failure mode — the keyboard methods and append are the same hazard class, just quieter. Happy to adjust.

waitForExpectation() re-runs the whole call under a 1s cap per attempt.
For an assertion that is the right tool. For an action it re-fires a
side effect: a click whose first attempt needs more than 1s has already
opened its dialog, and the retry then aims at a button the overlay now
covers. The test burns the whole timeout and fails on a click that
worked. A retried Enter double-submits a form; a retried drag starts
from an element the first attempt already moved.

typeSlowly is already excluded for exactly this reason. Extend the same
treatment to the other methods that repeat their effect when re-fired.
The excluded branch calls the method directly, so the action gets the
full timeout in one attempt — and Playwright's actionability check
(attached, visible, stable, receives events, enabled) waits inside that
one call, so nothing fires twice.
@Pawel-IT

Pawel-IT commented Aug 21, 2026 •

Copy link
Copy Markdown

Confirmed on unmodified v5.0.1. Here's a minimal reproduction in case it helps review, drop it in tests/Browser/Webpage/ and it counts how many times a single click() actually reaches the page:

use Illuminate\Support\Facades\Route;
use Pest\Browser\Playwright\Playwright;

it('issues click() exactly once even when the handler is slower than the 1000ms attempt timeout', function (): void {
    Playwright::setTimeout(5000); // AwaitableWebpage skips the retry loop at <= 1000ms

    Route::get('/', fn (): string => '
        <input type="checkbox" id="toggle">
        <script>
            window.clickCount = 0;
            document.getElementById("toggle").addEventListener("click", function () {
                window.clickCount++;
                var start = Date.now();
                while (Date.now() - start < 1500) {} // blocks past the 1000ms cap
            });
        </script>
    ');

    $page = visit('/');
    $page->click('#toggle');

    expect($page->script('() => window.clickCount'))->toBe(1);
});

On v5.0.1 that gives Failed asserting that 5 is identical to 1. One click() fired five times. Counting in-page matters here: an odd number of retries leaves the checkbox looking correct, which is probably part of why this reads as flakiness rather than a bug.

No competing fix intended — this PR and #227 already cover it.

@yeapea

yeapea commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Diagnosis looks right to me. Execution::waitForExpectation() does exactly what you describe, and all nine names you added are real methods on Webpage. Three things I'd add to the list though.

submit() is a click. It's literally $this->guessLocator('[type="submit"]')->click(), so it's the case your own description calls out ("a re-sent Enter submits the form again") — and it's the nastiest one, because a repeat can write a second row server-side. It belongs next to click.

The bigger one is withinFrame(). It isn't in the list, so it still goes through the retry, and retrying it re-runs the whole callback — every click inside it. The inner page doesn't save you either: during a retried attempt Playwright::timeout() is 1000, so the inner __call takes the <= 1000 branch and skips the list entirely. Upshot is that excluding click doesn't protect a click you wrote inside a withinFrame.

back() and forward() have the same shape. They send goBack / goForward, and if the navigation outruns the 1s cap the attempt counts as failed — except the history already moved, so the retry moves again. I don't know how often that cap actually gets crossed in the wild, but nothing stops the second goBack when it does.

Leaving type() retryable looks right, by the way. It goes through fill(), which replaces instead of appending, unlike typeSlowly().

@likemusic

likemusic commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Here is a browser-level reproduction of the data corruption this PR prevents, on current 5.x (a78ede5), using only the plugin's own suite. It fails without the change and passes with it.

append() is the clearest demonstration, because it is read-then-write: it reads the field's current value and fills the concatenation.

it('appends the value only once even when the field is slow to react', function (): void {
    Route::get('/', fn (): string => '<input id="field" type="text" value="A">
        <script>
            document.getElementById("field").addEventListener("input", () => {
                const end = Date.now() + 1500;
                while (Date.now() < end) {}
            });
        </script>');

    $page = visit('/');

    $page->append('#field', 'B');

    expect($page->script('document.getElementById("field").value'))->toBe('AB');
});

The input handler blocks the renderer for 1500ms. Execution::waitForExpectation() pins the client timeout to 1000ms for the duration of an attempt, and Client::execute() stamps that value onto every Playwright command it sends — so fill() itself is cut off at one second, after it has landed but before it returns. The loop retries, and append() now reads the value it just wrote:

5.x as it stands:   -'AB'  +'ABBB'      (fail, ~5.3s — the value was appended three times)
5.x with this PR:    'AB'               (pass, ~2.1s)

ABBB rather than ABB because the default suite timeout allows several retries inside the window.

What I think this adds: the failure is silent and lands far from its cause. Nothing in the eventual failure points at a retry loop — the field holds a value no one typed, whatever validates that value rejects it, and the assertion that finally fails is several steps downstream, usually about a missing element. Worth keeping in mind when judging how much this matters in practice.

The one case already fixed proves the shape: typeSlowly was added to $nonAwaitableMethods in 5.x with the comment "Retrying this action would append the value to what was already typed" — exactly the failure above, recognised and fixed for one method. append(), keys(), click(), press(), check() and the rest still go through the loop, so the same thing is still reachable; this PR closes the class rather than the instance.

Related: pestphp/pest#1755 describes the hardcoded 1000ms — applied to each Playwright command inside a retry attempt — that makes this reachable at all, independently of the configured timeout.


Update, 29 September 2026 — the reproduction above is green now, and that is worth saying out loud.

#269 landed on 5.x on 24 September and added append to $nonAwaitableMethods, among six other actions. The test above therefore passes at c98e8a5: 1.73s, one execution, measured on that checkout. Anyone copying it today would conclude the bug is gone, which is why I am not leaving it without this note. It passes because the specific method was taken out of the loop, not because actions stopped being retried.

What remains is everything #269 did not name, still cut off at a fixed second. Measured on the same c98e8a5 with the suite's own setTimeout(2000), a script() doing 1.5s of work runs three times back to back and the call then fails with Timeout 2000ms exceeded — 3 runs of 3, identical (enter@51 | leave@1551 | enter@1552 | leave@3052 | enter@3053 | leave@4553, 4.5s). Details and the instrumented test are in #276, which fixes the fixed second itself by giving each attempt what is left of the configured timeout.

So this PR still has ground of its own: it excludes more actions than #269 did. The reproduction above just no longer demonstrates it.

@Pawel-IT

Copy link
Copy Markdown

Any chance this gets merged soon? I'm assuming a ton of people are running into this issue and it really is a major time waste. It also makes tests extremely flaky

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants