Skip to content

Migrate e2e tests from cavy to react-native-cavynext, add web e2e support - #893

Open
tvanlaerhoven wants to merge 30 commits into
developfrom
feature/cavynext-migration
Open

Migrate e2e tests from cavy to react-native-cavynext, add web e2e support#893
tvanlaerhoven wants to merge 30 commits into
developfrom
feature/cavynext-migration

Conversation

@tvanlaerhoven

@tvanlaerhoven tvanlaerhoven commented Aug 6, 2026

Copy link
Copy Markdown
Member

Summary

Replaces the forked cavy/cavy-cli packages in the e2e app with react-native-cavynext / react-native-cavynext-cli (published on npm), modernises the specs, and adds web (react-native-web) e2e support.

Test framework migration

  • cavy / cavy-cli (github forks) → react-native-cavynext / react-native-cavynext-cli ^0.1.0 from npm
  • Specs use cavynext's Jest-style expect (the hand-rolled expect in utils/Actions.ts is removed)
  • Platform-specific suites via spec.describeIf(...): presentation-mode tests skip on web (fullscreen needs a user gesture), rendering-target tests run on Android only; skips show up in the report
  • Presentation-mode and rendering-target switching run on plain sources only: an IMA pre-roll added an ad-server dependency and ~10s of ad play-out per case without covering anything those specs assert

Player test layer (e2e/src/utils)

  • The player under test is registered in cavynext's TestHookStore by TestableTHEOplayerView instead of a module-level global; specs obtain it with getTestPlayer(spec). It is unregistered synchronously on unmount, so a test can never grab the previous, about-to-be-destroyed player
  • getTestPlayer gives a freshly created player a settle time before a spec sets a source. The runner re-mounts the view before every test and cavynext awaits that re-mount, so without it a source lands on a player that is milliseconds old while its predecessor is still being released natively — which made iOS/tvOS fail with MEDIA_AVPLAYER_ERROR (5003) on the mp4 source. Forked cavy hid this behind the 1s poll interval of its player lookup:
     const player = await spec.findComponent(PLAYER_HOOK_ID);
    +await new Promise((resolve) => setTimeout(resolve, PLAYER_SETTLE_TIME));
     return player;
  • waitForPlayerEvents rewritten: listeners and the timeout are always cleaned up, rejections are real Errors carrying a player-state snapshot, and unrelated events no longer break in-order matching
  • New seekTo(player, time) helper
  • The test app is no longer wrapped in <StrictMode>: its double-invoked mounts created and destroyed a second native player per test, and made React DevTools log every line twice

Web e2e support

  • npm run test:e2e:web runs the suite in Chrome via cavynext run-web + webpack dev server (e2e/web/), launched with a dedicated user-data-dir and throttling disabled so background timers/playback keep running
  • mutedAutoplay: 'all' on web (browser autoplay policy) and transmux workers copied from the repo root's theoplayer package so SDK and worker versions match

Library

  • THEOplayerAdapter clears its cached paused state on play, not only on playing, so player.paused no longer stays true for the whole playplaying gap

CI

  • pr_ios.yml / pr_android.yml updated for the cavynext CLI and cavynext_results.md
  • The tvOS pods target gets its Nielsen framework name rewritten to NielsenTVAppApi again, without which the tvOS app fails to link (ld: framework 'NielsenAppApi' not found)

Test results

Platform Result
Web ✅ 28 examples, 0 failures, 12 skipped
Android (emulator, release) ✅ 28 examples, 0 failures
iOS (simulator)
tvOS (simulator)

Notes

  • Rebased on develop (RN 0.86.2)
  • The e2e lockfile is based on develop's with cavy removed and cavynext added incrementally: a from-scratch npm install fails on the $react-native override reference (pre-existing on develop, worth a separate fix)
  • e2e/ios/Podfile.lock was regenerated without e2e/scripts/patch_ios_podspec.sh having run first, so it lacks the IMA/Chromecast/THEOads/Millicast integrations. CI patches the podspec and runs pod update, so only a plain local pod install is affected

Link to Devin session: https://dolby.devinenterprise.com/sessions/69e1051f15d743c690a56c0a7bd9018b
Requested by: @tvanlaerhoven


Open in Devin Review

tvanlaerhoven and others added 8 commits August 6, 2026 13:26
Replace all cavy imports with react-native-cavynext, update test scripts to
use the cavynext CLI, and point dev dependencies at the local cavynext
workspace packages. Update workflows and gitignore to look for
cavynext_results.md.

Add a placeholder player-specific support package at
e2e/packages/react-native-theoplayer-cavynext for THEOplayer-aware test
helpers.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Remove the empty react-native-theoplayer-cavynext stub and replace the local
file:../../ paths with a git submodule at e2e/packages/react-native-cavynext.
The e2e package.json now points file: deps into the submodule, and
package-lock.json was regenerated to match.

After cloning, run `git submodule update --init` and build the cavynext
packages (`npm run build` in the submodule) so lib/ output exists.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Replace the git submodule under e2e/packages with GitHub branch
references in e2e/package.json:
  - react-native-cavynext -> github:tvanlaerhoven/react-native-cavynext#core-dist
  - react-native-cavynext-cli -> github:tvanlaerhoven/react-native-cavynext#cli-dist

Regenerate package-lock.json and verify `npm install` succeeds; the
cavynext package lib/ and cavynext CLI bin are now installed from the
dist branches.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Picks up expect/component matchers, by.* selectors, richer interactions,
lifecycle hooks and the CLI --json flag.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Specs now use the cavynext expect API instead of the hand-rolled one in
utils/Actions (which is removed), drop the manual START/END log
boilerplate the runner already provides, use spec.platform() instead of
Platform.OS, and assert on received event counts in the ad tests.

Web support: a webpack config and index.html under e2e/web,
index.web.js / index.test.web.js entry points following the cavynext
run-web convention, a web-aware libraryLocation player config, a
theoplayer devDependency for the web SDK, and test:e2e:web / build:web
scripts. The web bundle builds cleanly with the test app as entry.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
- The player under test is now registered in cavynext's TestHookStore by
  TestableTHEOplayerView (unregistered synchronously on unmount via a
  layout effect) instead of living in a module-level global. Specs get it
  through getTestPlayer(spec), which rides cavynext's component lookup.
- waitForPlayerEvents rewritten: all listeners (including the ERROR and
  AD_EVENT watchers) are removed however the wait ends, the timeout is
  cleared, rejections are real Errors (PlayerEventTimeoutError) carrying
  a player state snapshot, and unrelated events of an expected type no
  longer break in-order matching.
- New seekTo(player, time) helper; preparePlayerWithSource takes spec.
- Platform-specific suites use spec.describeIf: presentation mode tests
  skip on web (fullscreen needs a user gesture), rendering target tests
  run on Android only.
- Web run: full suite green (28 examples, 0 failures, 12 skipped).

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
react-native-cavynext and react-native-cavynext-cli 0.1.0 are on npm
now; the github dist-branch references are gone. Full web e2e run green
against the npm packages (28 examples, 0 failures, 12 skipped).

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Based on develop lockfile with cavy removed and the cavynext npm packages added incrementally, because a from-scratch npm install fails on the react-native override reference (pre-existing on develop). Full web e2e run green on RN 0.86.
devin-ai-integration[bot]

This comment was marked as resolved.

tvanlaerhoven and others added 2 commits August 6, 2026 15:26
New e2e job in pr_web.yml running the suite headless in Chrome, with
the markdown summary appended to the job summary. run_web_e2e.sh adds
the headless/no-sandbox flags when CI is set. Verified locally in
headless mode: 28 examples, 0 failures, 12 skipped.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Nothing imports it; its autolinked native module registers battery and
brightness observers that emit through RCTDeviceEventEmitter after the
runtime tears down, producing the "Module has not been registered as
callable" error at the end of an iOS test run.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
devin-ai-integration[bot]

This comment was marked as resolved.

The workflows cat cavynext_results.md into the job summary, but the
cavynext CLI only writes it when --markdown is passed; the web job
already did.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>
devin-ai-integration[bot]

This comment was marked as resolved.

tom.vanlaerhoven@dolby.com and others added 2 commits August 12, 2026 16:42
devin-ai-integration[bot]

This comment was marked as resolved.

Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>
devin-ai-integration[bot]

This comment was marked as resolved.

cavynext 0.2.0 adds describe.each / it.each with label interpolation;
the specs now use it instead of wrapping describes in a forEach. The
TestSources builder is flattened into plain plainSources(platform) /
adSources(platform) functions, making platform-based source selection
explicit at the spec level. Web run green (28 examples, 0 failures,
12 skipped).

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Install the four analytics connector packages and re-enable
Connector.spec in the test set. The comscore web SDK references node
builtins behind runtime guards, so webpack resolves os/http/https to
nothing. react-native-device-info returns as an explicit devDependency:
the adobe connector peer-requires it, and without the native module the
app crashes on Android.

Verified: web 32 examples / 0 failures / 12 skipped; Android (release)
32 examples / 0 failures.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
devin-ai-integration[bot]

This comment was marked as resolved.

tvanlaerhoven and others added 4 commits August 12, 2026 11:01
NielsenAppSDK-XC, depended on by the nielsen analytics connector, is distributed through Nielsen's own CocoaPods specs repo rather than the CDN. Regenerate the stale, pre-rebase Podfile.lock while at it.
pr_web had no paths filter and ran on every PR update; pr_android and pr_ios triggered on each other's native subtrees under e2e. Web now has a filter, and each platform excludes the other's e2e native folder.
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 4 commits August 12, 2026 18:50
Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>
Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>
Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>
Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 3 commits August 12, 2026 20:01
Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>
Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>
Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

Open in Devin Review

Comment thread e2e/src/res/mp4.json
{
"sources": {
"src": "https://cdn.theoplayer.com/video/elephants-dream.mp4",
"src": "https://cdn.theoplayer.com/demos/me.mp4",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Shorter mp4 test asset may end before later assertions run

The mp4 fixture switched to demos/me.mp4. Several specs assume playback is still ongoing several seconds in: PresentationMode.spec sleeps 500ms after each mode switch and asserts player.paused is falsy, and Basic.spec seeks to 5s. If the new clip is short, ended (which pauses the player) could make those assertions flaky on slower runners. Worth confirming the asset's duration is comfortably longer than the ad pre-roll + seek + mode-switch sequence.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked: demos/me.mp4 is 10.03s (629 KB, 640x360 h264/aac, ffprobe-confirmed), and no spec needs anywhere near that much play-out. The presentation-mode case is two mode switches with one 500ms sleep (~1-2s after playing), the Android rendering-target case is 2x 500ms plus a timeupdate, and Basic.spec now seeks to 5s instead of 10s — that seek change is part of this commit precisely because 10s was the clip's duration. The Ads spec uses ads.json (VAST tag), not this fixture, so the pre-roll doesn't eat into it.

The switch is deliberate: the 47 MB elephants-dream.mp4 was the direct cause of the iOS 5003 failures. Each test cancelled its 46 MB range read mid-body (finished with error [-999]), Fastly closed those sockets, and the next player picked one out of CFNetwork's pool (connection=18, reused=1) → error code: 303 [4:-2205]CFByteFlume: Read failed during non-interactive download. Give up immediately.-12935FailedToBecomeReadyForPlayback. A fixture that completes in a single request leaves nothing in flight to abort.

Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

Open in Devin Review

Comment on lines +83 to +85
# Stay alive as long as the dev server runs: the cavynext CLI stops this script
# when the test run ends, and the trap then takes the browser down with it.
wait "${WEBPACK_PID}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Dev-server script exits non-zero when terminated by the runner

With set -euo pipefail, wait "${WEBPACK_PID}" returns non-zero when the script is signalled/terminated by the cavynext CLI at the end of the run, so the build command exits with a failure status (e.g. 143). Whether that is surfaced as a failed test run depends on how cavynext run-web treats the exit code of --buildCmd; if it does, CI would report a spurious failure after a successful run.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hedge resolves empirically: the e2e (web) job is green on every run of this branch, and it always ends this way — cavynext terminates the --buildCmd process itself after reporting results, so the 143 goes to a process whose exit code the CLI doesn't propagate. The job's own exit code comes from cavynext run-web, which is what pr_web.yml waits on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant