test(e2e): surface stalled spec and capture BrowserStack diagnostics - #236
Merged
Conversation
The in-app jasmine suite intermittently never renders all-tests-done, and a stalled run left nothing to diagnose: no spec name in CI output, no device logs, no screenshot (Maestro's extendedWaitUntil dies before the screenshot step). In the e2e app, always render the currently-running spec (testID current-spec), give the progress counter a testID (test-progress), and wrap jasmineEnv.execute() in a 240s watchdog — inside Maestro's 300s window — that renders all-tests-done plus a 'Suite timed out during: <spec>' failure, so a stall reaches the screenshot step and names the culprit. Replace the NoopGlobalErrors stub with a real implementation over ErrorUtils and Hermes' rejection tracker (neither is wired up in Release), so uncaught errors and unhandled rejections are console.error'd, routed to jasmine, and surfaced in the UI instead of silently hanging the suite. In the run-browserstack-maestro action, on the final failed attempt download each failed session's device logs and screenshots (URLs verified against a real past build's session JSON), print the app-tagged device-log lines into the job log, and upload everything as a browserstack-diagnostics-<platform> artifact. Collection is best-effort and the upload step runs only after the run step has already failed, so diagnostics can never mask the real failure or change the action's exit semantics.
gmaclennan
added
the
run-e2e
github-actions
Bot
added
the
maintenance
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When the in-app suite stalls (issue #233), CI currently yields nothing: Maestro's
extendedWaitUntilonall-tests-doneexpires before the screenshot step, the job log carries no spec output, and the device logs BrowserStack records are never fetched. Every occurrence costs a re-run and yields no diagnosis. This PR makes a stall name its spec through three layers.In the e2e app, the currently-running spec is always rendered (
current-spectestID) so any screenshot names it, the progress counter gets atest-progresstestID, andjasmineEnv.execute()races a 240-second watchdog — inside Maestro's 300-second window — that on expiry rendersall-tests-doneplus a "Suite timed out during: " failure, so a stalled run still reaches Maestro's screenshot step and fails onall-tests-passedwith evidence. TheNoopGlobalErrorsstub, which silently swallowed errors outside specs (a plausible cause of the hang), is replaced with a real implementation over React Native'sErrorUtilsand Hermes' promise-rejection tracker: uncaught errors and unhandled rejections are logged, routed into jasmine as failures, and listed in the UI.In the
run-browserstack-maestroaction, the final failed attempt now downloads each failed session's device logs and screenshots (field names verified against a real past build's session JSON), prints the app-tagged device-log lines into the job log — on Android that includes the[e2e] spec startedlines, which answers "which spec" directly — and uploads everything as abrowserstack-diagnostics-<platform>artifact. Collection is best-effort and the upload step runs only after the run step has already failed, so diagnostics can never mask the real failure or change the action's exit code or retry classification.The failure path can only be exercised by a real BrowserStack run; the collection script was validated GET-only against a past build's artifacts, and this PR's own labeled e2e run tests the happy path end to end.
Closes #233