Skip to content

Fix module-level state leaks that broke randomized test order - #362

Merged
TheGreatAxios merged 5 commits into
mainfrom
cl-5579-test-ordering
Aug 7, 2026
Merged

Fix module-level state leaks that broke randomized test order#362
TheGreatAxios merged 5 commits into
mainfrom
cl-5579-test-ordering

Conversation

@TheGreatAxios

@TheGreatAxios TheGreatAxios commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

bun test --randomize failed with over a hundred failures spread
across unrelated files (measured at seed 42 on unmodified main:
~107-119 failures across repeated runs — the count is not perfectly
stable run-to-run, meaning some failures are timing-sensitive rather
than purely a function of the shuffle order). All of it traced to
two shared-state bugs, plus two further instances of the first one:

  • mock.module restores that captured "the real module" via a live
    namespace binding — either await import() or a static
    import * as ns — were no-ops, because Bun mutates that same
    namespace object in place when the module gets mocked. The
    captured reference silently turns into the mock too, so a later
    mock.module(path, () => capturedReference) "restore" changes
    nothing. This broke:
    • tests/unit/tui/agent-tools.test.ts (authz/verify/secret-guard/
      read-file-guard plugins collapsed to fakes for every file that
      ran afterward)
    • src/tools/web-search.test.ts (dropped unwrapToolContent and
      connectMCPServers from mcp/client.js process-wide)
    • tests/unit/config.test.ts (node:os capture via a static
      import * as nodeOs, same defect, different capture shape)
    • mouse-reporting-disabled.test.ts (latent, not yet triggering a
      failure — fixed proactively)
    • src/session/state.test.ts mocked node:fs/promises with no
      restore at all; harmless in practice because its delegate was
      captured before the mock took effect, but still fixed to restore
      Fixed by shallow-copying the capture immediately ({ ...ns }),
      regardless of which import form produced it, and spreading the
      real module into the mock instead of replacing it outright. Also
      removed a dead mock of a src/web/plugin.js module that no longer
      exists.
  • schedulePricingMetadataRefresh's one-shot latch is module-level
    and shared by the whole test process; any file that exercises the
    real loadConfig bootstrap path leaves it set, so
    pricing-metadata.test.ts's own dedupe test silently no-op'd when
    it ran after one of those files. Fixed with a beforeEach reset
    alongside the existing afterEach one.

AGENTS.md documents the rule in terms of any live namespace
binding, not just the dynamic-import idiom, so the next author
following the guidance doesn't reproduce the config.test.ts bug in a
different shape. CI keeps the deterministic fixed-seed step and adds
a nightly job with a fresh random seed each run, printed up front so
a failure reproduces locally with the same --seed.

No assertions were weakened. Each fix restores real isolation — the
tests that exercise the fixed plugins/latch still fail if that
isolation regresses.

Verification

  • bun run typecheck, bun run build, bun run test all pass
  • bun test ./src ./tests ./evals --randomize passes across 20
    seeds (42, 1337, 1361063567, 1-10, 100, 200, 300, 7, 555, 909090,
    2026, 314159, 8675309, 424242)
  • No tests quarantined

Closes CL-5579

@linear-code

linear-code Bot commented Aug 7, 2026

Copy link
Copy Markdown

CL-5579

Bun swaps a mocked module's exports onto the same namespace object
it hands back from import(), so a file that captures "the real
module" via a bare import() before mocking is holding a reference
that later mutates into the mock. Restoring with that reference in
afterAll was therefore a no-op, and files loaded after left every
other test in the process running against the mock: authz, verify,
secret-guard, and read-file-guard plugins collapsed to their fakes,
mcpClientToAgentTools lost several exports, and any file statically
importing mcp/client.js after web-search.test.ts lost
unwrapToolContent and connectMCPServers entirely.

Shallow-copying the captured exports at capture time freezes a real
snapshot before the mock overwrites the shared object, so afterAll
can actually put the original back. The web-search mock also now
spreads the real module instead of replacing it outright, and the
mouse-reporting-disabled mock (already spreading real exports) gets
the same snapshot fix so its existing afterAll restore is not
silently defeated the same way.

The dead mock of a nonexistent src/web/plugin.js module is removed
along with its unusable real-module capture.
schedulePricingMetadataRefresh guards itself with a module-level
one-shot flag shared by the whole test process. Any other file that
exercises loadConfig's real bootstrap path leaves that flag set to
true and never clears it, so this file's own first test silently
inherited "already scheduled" from whichever file ran before it and
its offlineFetch was never called.
A file order that happens to work is not evidence of isolation; only
running with --randomize catches a test that leans on another file's
side effects. Fixing the leaks this uncovered is only durable if the
check keeps running, so add a CI step alongside the canonical run and
write the underlying rule into AGENTS.md.
config.test.ts captured node:os with a static `import * as nodeOs`
and later "restored" it with `mock.module("node:os", () => nodeOs)`
-- the same defect as the dynamic-import case already fixed
elsewhere in this branch, since Bun mutates that namespace object in
place regardless of how it was imported. The capture is now a
shallow copy taken before anything mocks node:os.

state.test.ts mocked node:fs/promises with no restore at all. Its
delegate functions were captured before the mock took effect so the
leak was inert here, but the file still violated the isolation rule
this branch exists to enforce, so it gets the same shallow-copy
capture and an afterAll restore.
The AGENTS.md wording only mentioned the dynamic-import capture
shape, which reads as though that is the only form at risk -- it is
not; a static `import * as ns` binding is mutated by mock.module the
same way, and config.test.ts had exactly that bug. State the rule in
terms of any live namespace binding instead of one idiom.

The fixed-seed CI step is deterministic and worth keeping, but it
only ever exercises one shuffle of the suite, so a leak that
particular order does not disturb stays invisible forever. Add a
nightly job with a fresh random seed each run, printed up front so a
failure reproduces locally with the exact same --seed.
@TheGreatAxios
TheGreatAxios force-pushed the cl-5579-test-ordering branch from 13e1701 to 9e3b9db Compare August 7, 2026 08:26
@TheGreatAxios
TheGreatAxios merged commit 9cb88ed into main Aug 7, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant