Skip to content

test: wait for the setup compile deterministically in workspace init - #552

Merged
magnus-madsen merged 1 commit into
masterfrom
test/deterministic-workspace-setup
Jul 1, 2026
Merged

test: wait for the setup compile deterministically in workspace init#552
magnus-madsen merged 1 commit into
masterfrom
test/deterministic-workspace-setup

Conversation

@magnus-madsen

Copy link
Copy Markdown
Member

Problem

CI still shows intermittent VS Code extension test failures (example run):

1) RenameProvider Should rename variable:      Error: No result.
2) RenameProvider Should rename variable-use:   Error: No result.

Error: No result. means the vscode.executeDocumentRenameProvider request came back empty — the server's index had no symbol at that position because the workspace hadn't been compiled yet when the test ran.

Root cause

#545 converted in-test file mutations to wait on lsp/check via awaitCheck, but the per-suite workspace swap in init still used fixed sleeps (processFileChange: sleep(1000)awaitIdlesleep(1000)).

awaitIdle (flix.allJobsFinished) resolves immediately whenever the compiler's queue is momentarily empty — which it is if the file-system watcher hasn't fired yet after copyDirContents. Under CI load, init could return before the check triggered by copying the suite's files had finished compiling them.

RenameProvider (suite #12) does no in-test mutation, so it never benefits from awaitCheck — its correctness depends entirely on init leaving the workspace compiled and idle. Hence the residual flakiness.

Fix

Make the setup path deterministic, the same way the mutation path already is:

  • settleAfterChange(baseline) waits for checkCount to advance past a baseline captured before the change (proving the watcher fired and a check completed), then repeatedly drains to idle until the check count is stable across a reconcile-debounce window (the client schedules a 300 ms-debounced reconciliation that can enqueue a follow-up check).
  • init baselines and settles separately for the clear and the copy, and waits for the removals to be observed before copying — so VS Code reports the new files as creates rather than coalescing a delete-then-create of the same path into a single (unhandled) change event.
  • The first suite (extension not yet active) has no watcher/baseline; its copied files are picked up by the initial workspace scan on ext.activate(), and settleAfterChange(0) waits for that compile.
  • Removes the now-unused processFileChange.

Net effect: the fixed ~4 s of per-suite sleeps are replaced by a deterministic wait anchored to observable compiler progress (typically faster, and no longer racy).

Testing

npm run build and tsc --build ./test pass. The change is timing-only in the test harness; validated end-to-end on CI (local single runs can't reproduce load-dependent flakiness).

PR #545 made in-test file mutations wait on `lsp/check` via `awaitCheck`,
but the per-suite workspace swap in `init` still used fixed `sleep`s
(`processFileChange`: sleep 1s, awaitIdle, sleep 1s). `awaitIdle`
(`flix.allJobsFinished`) resolves immediately whenever the queue is
momentarily empty, which it is if the file-system watcher hasn't fired
yet. Under CI load `init` could therefore return before the check
triggered by copying the suite's files had compiled them.

Suites that only query after `init` (e.g. RenameProvider, which does no
in-test mutation) depend entirely on this wait, so the rename request hit
a server whose index lacked the symbol and failed with "Error: No
result." — the residual flakiness seen in CI.

Make the setup path deterministic like the mutation path:

- `settleAfterChange(baseline)` waits for `checkCount` to advance past a
  baseline captured before the change (proving the watcher fired and a
  check completed), then drains to idle repeatedly until the check count
  is stable across a reconcile-debounce window (the client schedules a
  300ms-debounced reconciliation that can enqueue a follow-up check).
- `init` baselines and settles separately for the clear and the copy, and
  waits for the removals to be observed before copying so VS Code reports
  the new files as creates rather than coalescing a delete-then-create
  into an unhandled change event.
- The first suite (extension not yet active) has no watcher/baseline, so
  the copied files are picked up by the initial workspace scan on
  `ext.activate()`; `settleAfterChange(0)` then waits for that compile.

Removes the now-unused `processFileChange`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@magnus-madsen
magnus-madsen merged commit ced1075 into master Jul 1, 2026
7 checks passed
@magnus-madsen
magnus-madsen deleted the test/deterministic-workspace-setup branch July 28, 2026 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant