Skip to content

chore(workflow): migrate shared, computer, android and ios to rstest - #2958

Open
fi3ework wants to merge 4 commits into
web-infra-dev:mainfrom
fi3ework:rstest-migration/03-mid-tier
Open

chore(workflow): migrate shared, computer, android and ios to rstest#2958
fi3ework wants to merge 4 commits into
web-infra-dev:mainfrom
fi3ework:rstest-migration/03-mid-tier

Conversation

@fi3ework

@fi3ework fi3ework commented Aug 11, 2026

Copy link
Copy Markdown
Member

Third batch of the incremental vitest 3.0.5 -> rstest 0.11.5 migration, after #2921 (four micro-packages) and #2945 (six cold packages). This one covers shared, computer, android and ios — 117 test files, 1047 unit tests.

Remaining after this: cli + web-integration, then core, then studio plus the vitest teardown.

Performance

Roughly break-even overall, with the win concentrated in shared. Short version: rstest's advantage is compilation, and three of these four packages barely compile — they mostly sleep. ios spends 1.3% of its run building and the rest waiting on real timers inside the device code under test, so there is nothing for a faster bundler to reclaim. shared is 441 fast pure-function tests where compilation is the dominant cost, and it doubles. Breakdown below the table.

Median of 3 runs per cell, runners alternated each round so machine drift hits both equally. Cold clears the bundler caches first. pnpm -s test equivalent, Apple M3 Max, Node 26.5.0. Both runners exit 0 on every sample.

Package Cold vitest Cold rstest Warm vitest Warm rstest
shared 2.96s 1.66s 1.78x 2.88s 1.41s 2.04x
computer 11.92s 10.73s 1.11x 11.60s 10.96s 1.06x
android 13.44s 12.70s 1.06x 13.45s 12.73s 1.06x
ios 18.04s 18.30s 0.99x 17.83s 18.14s 0.98x
total 46.36s 43.39s 1.07x 45.76s 43.24s 1.06x

The spread is not noise, and it is the same rule the chrome-extension result in #2945 followed: the gain tracks the compile-to-execute ratio, not the file count. rstest's advantage is compilation, so a suite that barely compiles and mostly waits has nothing to win. rstest reports the split itself:

Package build tests build share
shared 71ms 886ms 7.4%
computer 219ms 10.1s 2.1%
android 275ms 12.8s 2.1%
ios 229ms 17.2s 1.3%

shared is 441 fast pure-function tests, so compilation is the largest movable cost and it doubles warm. ios spends 1.3% of the run compiling, which caps any bundler-side gain at about 1.3% — inside the measurement noise.

Where ios actually spends its 17s, measured file by file: device.test.ts alone takes 17.38s wall while the whole package takes 17.4s, so it is the critical path and the other 12 files finish alongside it. Inside that file, the 48 cases account for only 2.26s; the remaining ~15s is afterEach calling await device.destroy() 48 times, about 315ms of real waiting per test, because src/device.ts sleeps for real (sleep(100), sleep(500), sleep(800), sleep(2000)) with no fake timers in play. computer and android are the same shape.

These four are here for uniformity, not speed; the batches that moved the needle were the cold ones in #2945. Note the numbers above are unit tests only — tests/ai/** is excluded from the default include, so no model or network calls are involved.

Test discovery is identical, case by case

Not just the totals — I diffed the full file > test name list produced by rstest list against vitest list on the pre-migration tree, for every include mode each package has:

Package Mode Files Tests
shared unit 34 441 identical
computer unit 14 91 identical
android unit 18 358 identical
ios unit 13 157 identical
shared AITEST=1 34 441 identical
computer AI_TEST_TYPE=computer 15 29 identical
computer AI_TEST_TYPE=computer-rdp 1 1 identical
android AI_TEST_TYPE=android 9 14 identical
ios AI_TEST_TYPE=iOS 6 9 identical

The AI suites need real devices and models so they are not executed here, but rstest list compiles them, which is what caught two of the three type problems below.

One formatting difference, not a discovery difference: for test.each names, vitest quotes interpolated string values (retries 'a missing dump file') and rstest does not. Normalising the quotes makes all eight modes byte-identical.

What is not a mechanical rename

112 of the 117 test files are a pure vitest -> @rstest/core specifier swap plus vi. -> rs.. The rest:

1. Async mock factories have no rstest equivalent (android, 3 sites). rstest's MockFactory is synchronous, so vi.mock('x', async (importOriginal) => ...) is a hard error. These switch to the import-attribute recipe already used by harmony in #2945:

import * as fsActual from 'node:fs' with { rstest: 'importActual' };
rs.mock('node:fs', () => { const original = fsActual; /* ... */ });

page.test.ts x2, scrcpy-adapter.test.ts x1.

2. ios/tests/unit-test/agent.test.ts no longer mocks the device-class override. Two blockers stack here. agentFromWebDriverAgent resolves the override through await import(overrideModule) where the specifier is a user-supplied runtime value, which rstest's build-time mock transform cannot reach (web-infra-dev/rstest#1454). And rstest has no virtual-module mocking at all — its MockModuleOptions is only { spy: true } | { mock: true }.

The two override cases now point at real .mjs fixtures under tests/unit-test/fixtures/ and observe them through a global counter, which is what crosses the module-realm split between rstest's registry and the native loader. No test is skipped, and the suite gains an assertion that the default device class stays unused — something the mock-based version could not express.

Worth noting for reviewers: the old code cast vi.doMock to a signature vitest does not have ({ virtual: true } is a Jest option; vitest's MockOptions is { spy?: boolean }). Removing that argument on the pre-migration tree leaves all 13 cases passing, so it was dead weight. It also meant the two success-path cases never exercised real module resolution while the failure-path case did.

3. Three type errors that neither lint nor nx test sees. The AI suites are compiled but not type-checked by either, so these only surface under tsc:

  • android/tests/ai/merge-reports.test.ts read ctx.task.result.state and .startTime. rstest's TestResult names the field status and carries no start timestamp, so the duration now derives from the performance.now() stamp the suite already takes in beforeEach. Same fix as harmony in chore(workflow): migrate six packages to rstest #2945.
  • computer/tests/ai/chrome-extension-bridge.test.ts passed { timeout: 20 * 60 * 1000, retry: 0 } as the third argument. rstest takes options in the second position; in the third position the object is silently ignored, so the suite would have quietly fallen back to the config's 3-minute testTimeout. Verified with a standalone repro: third position leaves retryCount: 0, second position honours it.
  • android/tsconfig.json pinned module: ES2020, which rejects the import-attribute syntax item 1 needs. The pin moves to tsconfig.build.json so the production build keeps its module semantics while the tests inherit ESNext. A stale tsconfig.tsbuildinfo masks this error, which is worth knowing if you reproduce locally.

Config translation

Per package: test.* moves to the top level, ssr.external becomes output.externals, define becomes source.define, fileParallelism: false becomes pool: { maxWorkers: 1 } (computer, android), and dangerouslyIgnoreUnhandledErrors becomes errors: { unhandled: false } (android, ios). computer's explicit environment: 'node' is preserved as testEnvironment: 'node' rather than dropped as redundant.

scripts/rstest-shared.ts is new: the photon external and the __VERSION__ define were repeated verbatim in every node-target config. With five consumers it is now defined once, and harmony moves over too. This was deliberately deferred from #2945, where harmony was the only consumer.

Also in this PR, the follow-ups from the #2945 review: harmony/tsconfig.build.json pins module: ES2020 again so the production build is decoupled from the test-only import-attribute syntax, the two stale @ts-ignore comments are gone, and scripts/rstest-style-stub.ts types its rspack callback as Rspack.Configuration (the suggested satisfies Pick<RstestConfig, 'tools'> alone catches nothing, because tools.rspack is a ConfigChain union).

Validation

npx nx test @midscene/shared   --skip-nx-cache   # 34 files / 441 tests
npx nx test @midscene/computer --skip-nx-cache   # 14 files /  91 tests
npx nx test @midscene/android  --skip-nx-cache   # 18 files / 358 tests
npx nx test @midscene/ios      --skip-nx-cache   # 13 files / 157 tests
npx nx test @midscene/harmony  --skip-nx-cache   #  7 files / 246 tests (shared-helper regression)

npx tsc --noEmit -p packages/{shared,computer,android,ios}/tsconfig.json
npx tsc --noEmit -p packages/{shared,computer,android,ios}/tsconfig.build.json

npx nx build @midscene/shared  --skip-nx-cache
npx nx build @midscene/android --skip-nx-cache
npx nx build @midscene/harmony --skip-nx-cache

pnpm run lint

All green. Discovery parity was checked with rstest list / vitest list against a worktree of the pre-migration tree, as tabled above.

Not run: the AI suites themselves, which need real devices and model credentials.

Decouple harmony's production TypeScript config from the test-only import
attribute syntax: tsconfig.build.json pins module=ES2020 again so the build
output semantics stay put, while tsconfig.json keeps inheriting ESNext for the
tests. Drop the two @ts-ignore comments that described the old arrangement.

Type the shared less-stub helper's rspack callback as Rspack.Configuration.
The satisfies clause alone gives no protection here because tools.rspack is a
ConfigChain union, so an explicit annotation is what actually catches typos.
The photon external and the __VERSION__ define are repeated verbatim in every
node-target test config. With five consumers after this batch, hoist them into
scripts/rstest-shared.ts. defineVersion also settles the encoding, which had
drifted between `'${version}'` and JSON.stringify(version) across packages.
Mechanical for 116 of 121 test files: the vitest specifier becomes @rstest/core
and the vi binding becomes rs. The per-package configs move test.* to the
top level, fileParallelism: false to pool.maxWorkers: 1, and
dangerouslyIgnoreUnhandledErrors to errors.unhandled.

Five files needed hand work, all for the same reason: rstest mock factories are
synchronous, so vitest's async (importOriginal) has no counterpart. Four of them
switch to an import attribute (import * as x from 'y' with { rstest:
'importActual' }) plus a synchronous spread, the recipe already used in
harmony.

The fifth is ios agent.test.ts. agentFromWebDriverAgent loads the device class
override through await import(overrideModule) where the specifier is a
user-supplied runtime value, which rstest's build-time mock transform cannot
reach (web-infra-dev/rstest#1454). vi.doMock(..., { virtual: true }) has no
rstest equivalent either, so the two override cases now point at real on-disk
fixture modules and observe them through a global counter. The suite also gains
an assertion that the default device class stays unused, which the mock-based
version could not express.

Test discovery is unchanged from vitest, package for package:
shared 34/441, android 18/357, ios 13/155, computer 14/91.
… see

Three problems that only surface under tsc: neither biome nor nx test type-checks
the AI suites, and the import attribute error needs a cleared tsbuildinfo to
reproduce.

android merge-reports.test.ts read ctx.task.result.state and .startTime. Rstest's
TestResult names the field status and does not carry a start timestamp, so the
duration now derives from the performance.now() stamp the suite already takes in
beforeEach. Same fix already applied to harmony.

computer chrome-extension-bridge.test.ts passed { timeout, retry } as the third
argument. Rstest takes test options in the second position instead.

android's tsconfig.json pinned module=ES2020, which rejects the import attribute
syntax the page and scrcpy-adapter suites now need. Move the pin to
tsconfig.build.json so the production build keeps its module semantics while the
tests inherit ESNext.
@fi3ework
fi3ework force-pushed the rstest-migration/03-mid-tier branch from cd359bf to 5c21c84 Compare August 12, 2026 09:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant