Migrate e2e tests from cavy to react-native-cavynext, add web e2e support - #893
Migrate e2e tests from cavy to react-native-cavynext, add web e2e support#893tvanlaerhoven wants to merge 30 commits into
Conversation
Replace all cavy imports with react-native-cavynext, update test scripts to use the cavynext CLI, and point dev dependencies at the local cavynext workspace packages. Update workflows and gitignore to look for cavynext_results.md. Add a placeholder player-specific support package at e2e/packages/react-native-theoplayer-cavynext for THEOplayer-aware test helpers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Remove the empty react-native-theoplayer-cavynext stub and replace the local file:../../ paths with a git submodule at e2e/packages/react-native-cavynext. The e2e package.json now points file: deps into the submodule, and package-lock.json was regenerated to match. After cloning, run `git submodule update --init` and build the cavynext packages (`npm run build` in the submodule) so lib/ output exists. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Replace the git submodule under e2e/packages with GitHub branch references in e2e/package.json: - react-native-cavynext -> github:tvanlaerhoven/react-native-cavynext#core-dist - react-native-cavynext-cli -> github:tvanlaerhoven/react-native-cavynext#cli-dist Regenerate package-lock.json and verify `npm install` succeeds; the cavynext package lib/ and cavynext CLI bin are now installed from the dist branches. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Picks up expect/component matchers, by.* selectors, richer interactions, lifecycle hooks and the CLI --json flag. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Specs now use the cavynext expect API instead of the hand-rolled one in utils/Actions (which is removed), drop the manual START/END log boilerplate the runner already provides, use spec.platform() instead of Platform.OS, and assert on received event counts in the ad tests. Web support: a webpack config and index.html under e2e/web, index.web.js / index.test.web.js entry points following the cavynext run-web convention, a web-aware libraryLocation player config, a theoplayer devDependency for the web SDK, and test:e2e:web / build:web scripts. The web bundle builds cleanly with the test app as entry. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
- The player under test is now registered in cavynext's TestHookStore by TestableTHEOplayerView (unregistered synchronously on unmount via a layout effect) instead of living in a module-level global. Specs get it through getTestPlayer(spec), which rides cavynext's component lookup. - waitForPlayerEvents rewritten: all listeners (including the ERROR and AD_EVENT watchers) are removed however the wait ends, the timeout is cleared, rejections are real Errors (PlayerEventTimeoutError) carrying a player state snapshot, and unrelated events of an expected type no longer break in-order matching. - New seekTo(player, time) helper; preparePlayerWithSource takes spec. - Platform-specific suites use spec.describeIf: presentation mode tests skip on web (fullscreen needs a user gesture), rendering target tests run on Android only. - Web run: full suite green (28 examples, 0 failures, 12 skipped). Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
react-native-cavynext and react-native-cavynext-cli 0.1.0 are on npm now; the github dist-branch references are gone. Full web e2e run green against the npm packages (28 examples, 0 failures, 12 skipped). Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Based on develop lockfile with cavy removed and the cavynext npm packages added incrementally, because a from-scratch npm install fails on the react-native override reference (pre-existing on develop). Full web e2e run green on RN 0.86.
New e2e job in pr_web.yml running the suite headless in Chrome, with the markdown summary appended to the job summary. run_web_e2e.sh adds the headless/no-sandbox flags when CI is set. Verified locally in headless mode: 28 examples, 0 failures, 12 skipped. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Nothing imports it; its autolinked native module registers battery and brightness observers that emit through RCTDeviceEventEmitter after the runtime tears down, producing the "Module has not been registered as callable" error at the end of an iOS test run. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The workflows cat cavynext_results.md into the job summary, but the cavynext CLI only writes it when --markdown is passed; the web job already did. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>
…dules Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>
Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>
cavynext 0.2.0 adds describe.each / it.each with label interpolation; the specs now use it instead of wrapping describes in a forEach. The TestSources builder is flattened into plain plainSources(platform) / adSources(platform) functions, making platform-based source selection explicit at the spec level. Web run green (28 examples, 0 failures, 12 skipped). Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Install the four analytics connector packages and re-enable Connector.spec in the test set. The comscore web SDK references node builtins behind runtime guards, so webpack resolves os/http/https to nothing. react-native-device-info returns as an explicit devDependency: the adobe connector peer-requires it, and without the native module the app crashes on Android. Verified: web 32 examples / 0 failures / 12 skipped; Android (release) 32 examples / 0 failures. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
NielsenAppSDK-XC, depended on by the nielsen analytics connector, is distributed through Nielsen's own CocoaPods specs repo rather than the CDN. Regenerate the stale, pre-rebase Podfile.lock while at it.
pr_web had no paths filter and ran on every PR update; pr_android and pr_ios triggered on each other's native subtrees under e2e. Web now has a filter, and each platform excludes the other's e2e native folder.
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>
Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>
Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>
Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>
Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>
Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>
Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>
Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>
| { | ||
| "sources": { | ||
| "src": "https://cdn.theoplayer.com/video/elephants-dream.mp4", | ||
| "src": "https://cdn.theoplayer.com/demos/me.mp4", |
There was a problem hiding this comment.
🔍 Shorter mp4 test asset may end before later assertions run
The mp4 fixture switched to demos/me.mp4. Several specs assume playback is still ongoing several seconds in: PresentationMode.spec sleeps 500ms after each mode switch and asserts player.paused is falsy, and Basic.spec seeks to 5s. If the new clip is short, ended (which pauses the player) could make those assertions flaky on slower runners. Worth confirming the asset's duration is comfortably longer than the ad pre-roll + seek + mode-switch sequence.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Checked: demos/me.mp4 is 10.03s (629 KB, 640x360 h264/aac, ffprobe-confirmed), and no spec needs anywhere near that much play-out. The presentation-mode case is two mode switches with one 500ms sleep (~1-2s after playing), the Android rendering-target case is 2x 500ms plus a timeupdate, and Basic.spec now seeks to 5s instead of 10s — that seek change is part of this commit precisely because 10s was the clip's duration. The Ads spec uses ads.json (VAST tag), not this fixture, so the pre-roll doesn't eat into it.
The switch is deliberate: the 47 MB elephants-dream.mp4 was the direct cause of the iOS 5003 failures. Each test cancelled its 46 MB range read mid-body (finished with error [-999]), Fastly closed those sockets, and the next player picked one out of CFNetwork's pool (connection=18, reused=1) → error code: 303 [4:-2205] → CFByteFlume: Read failed during non-interactive download. Give up immediately. → -12935 → FailedToBecomeReadyForPlayback. A fixture that completes in a single request leaves nothing in flight to abort.
Co-Authored-By: tom.vanlaerhoven <tom.vanlaerhoven@dolby.com>
| # Stay alive as long as the dev server runs: the cavynext CLI stops this script | ||
| # when the test run ends, and the trap then takes the browser down with it. | ||
| wait "${WEBPACK_PID}" |
There was a problem hiding this comment.
🔍 Dev-server script exits non-zero when terminated by the runner
With set -euo pipefail, wait "${WEBPACK_PID}" returns non-zero when the script is signalled/terminated by the cavynext CLI at the end of the run, so the build command exits with a failure status (e.g. 143). Whether that is surfaced as a failed test run depends on how cavynext run-web treats the exit code of --buildCmd; if it does, CI would report a spurious failure after a successful run.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Hedge resolves empirically: the e2e (web) job is green on every run of this branch, and it always ends this way — cavynext terminates the --buildCmd process itself after reporting results, so the 143 goes to a process whose exit code the CLI doesn't propagate. The job's own exit code comes from cavynext run-web, which is what pr_web.yml waits on.
Summary
Replaces the forked
cavy/cavy-clipackages in the e2e app withreact-native-cavynext/react-native-cavynext-cli(published on npm), modernises the specs, and adds web (react-native-web) e2e support.Test framework migration
cavy/cavy-cli(github forks) →react-native-cavynext/react-native-cavynext-cli^0.1.0from npmexpect(the hand-rolledexpectinutils/Actions.tsis removed)spec.describeIf(...): presentation-mode tests skip on web (fullscreen needs a user gesture), rendering-target tests run on Android only; skips show up in the reportPlayer test layer (
e2e/src/utils)TestHookStorebyTestableTHEOplayerViewinstead of a module-level global; specs obtain it withgetTestPlayer(spec). It is unregistered synchronously on unmount, so a test can never grab the previous, about-to-be-destroyed playergetTestPlayergives a freshly created player a settle time before a spec sets a source. The runner re-mounts the view before every test and cavynextawaits that re-mount, so without it a source lands on a player that is milliseconds old while its predecessor is still being released natively — which made iOS/tvOS fail withMEDIA_AVPLAYER_ERROR(5003) on the mp4 source. Forked cavy hid this behind the 1s poll interval of its player lookup:const player = await spec.findComponent(PLAYER_HOOK_ID); +await new Promise((resolve) => setTimeout(resolve, PLAYER_SETTLE_TIME)); return player;waitForPlayerEventsrewritten: listeners and the timeout are always cleaned up, rejections are realErrors carrying a player-state snapshot, and unrelated events no longer break in-order matchingseekTo(player, time)helper<StrictMode>: its double-invoked mounts created and destroyed a second native player per test, and made React DevTools log every line twiceWeb e2e support
npm run test:e2e:webruns the suite in Chrome viacavynext run-web+ webpack dev server (e2e/web/), launched with a dedicated user-data-dir and throttling disabled so background timers/playback keep runningmutedAutoplay: 'all'on web (browser autoplay policy) and transmux workers copied from the repo root'stheoplayerpackage so SDK and worker versions matchLibrary
THEOplayerAdapterclears its cachedpausedstate onplay, not only onplaying, soplayer.pausedno longer staystruefor the wholeplay→playinggapCI
pr_ios.yml/pr_android.ymlupdated for thecavynextCLI andcavynext_results.mdNielsenTVAppApiagain, without which the tvOS app fails to link (ld: framework 'NielsenAppApi' not found)Test results
Notes
npm installfails on the$react-nativeoverride reference (pre-existing on develop, worth a separate fix)e2e/ios/Podfile.lockwas regenerated withoute2e/scripts/patch_ios_podspec.shhaving run first, so it lacks the IMA/Chromecast/THEOads/Millicast integrations. CI patches the podspec and runspod update, so only a plain localpod installis affectedLink to Devin session: https://dolby.devinenterprise.com/sessions/69e1051f15d743c690a56c0a7bd9018b
Requested by: @tvanlaerhoven