Skip to content

ci(e2e): keep Playwright traces and outputs from bazel-remote-gates as a run artifact - #269

Draft
Jesssullivan wants to merge 2 commits into
mainfrom
jess/e2e-evidence-artifacts
Draft

Jesssullivan wants to merge 2 commits into
mainfrom
jess/e2e-evidence-artifacts

Conversation

@Jesssullivan

Copy link
Copy Markdown
Owner

First bounded PR from the QA LOOK-loop design (2026-08-28): the blog's e2e already runs remotely on the overlay dind pool inside bazel-remote-gates, but a failure leaves no evidence — PLAYWRIGHT_TRACE_MODE defaulted to off and the job had zero upload-artifact steps (the one in ci.yml belongs to Lighthouse in build-and-test). Today's 1/330 e2e timeout on #263 could only be read from stdout.

  • .bazelrc: test --test_env=PLAYWRIGHT_TRACE_MODE (alongside the existing Chromium env passthrough).
  • playwright.bazel.config.ts: outputDir = $TEST_UNDECLARED_OUTPUTS_DIR/playwright under bazel test (falls back to test-results), so traces/screenshots/error-contexts survive the sandbox in bazel-testlogs/playwright_chromium_e2e/test.outputs/outputs.zip.
  • ci.yml gates job: PLAYWRIGHT_TRACE_MODE: retain-on-failure; an if: always() step copies bazel-testlogs/playwright_chromium_e2e (dereferenced, zip expanded) into artifacts/playwright-e2e, then actions/upload-artifact@v7 (SHA-pinned) uploads it, 7-day retention, if-no-files-found: warn.

No new permissions or secrets; nothing about what is tested changes. Local: node scripts/test-workflow-authority.mjs green (file reads only). Hosted CI is the proof — the artifact should appear on this PR's own gates run.

An e2e failure in bazel-remote-gates left nothing behind: trace mode was
off and the job uploaded no artifact, so today's 1/330 timeout could only be
read from ephemeral stdout. Now PLAYWRIGHT_TRACE_MODE=retain-on-failure
reaches the test action (.bazelrc --test_env), the Bazel config writes
Playwright's output directory into TEST_UNDECLARED_OUTPUTS_DIR so Bazel keeps
it in bazel-testlogs/playwright_chromium_e2e/test.outputs/outputs.zip, and an
always-run step copies that directory (dereferenced, zip expanded) into a run
artifact retained for seven days. No new permissions, no secrets, no change
to what is tested.
@Jesssullivan

Copy link
Copy Markdown
Owner Author

Scope note against TIN-3130 / R336 (GF is retiring mandatory GitHub-Actions-artifact evidence latches because retention quota can block publishers): this upload is evidence, not a latch — no job, gate, or publisher consumes it, if-no-files-found: warn, 7-day retention on a personal-account public repo. Written exit per R336: when the gf-qa-look RustFS bucket + its write identity exist (GF #1668 design), the Upload e2e evidence step is replaced by the bucket upload in the same job; the Bazel-side change (undeclared outputs + trace mode) is venue-independent and stays.

Run 33203042020 kept the Playwright trace for the failing spec but Bazel
then refused the test: 'output tree artifact playwright_chromium_e2e/
test.outputs was not created'. Bazel 8 materializes undeclared outputs as
an unzipped tree artifact by default and the spec directory names are
non-ASCII. --zip_undeclared_test_outputs makes it one outputs.zip, which
the evidence step already expands.
@Jesssullivan

Copy link
Copy Markdown
Owner Author

Run 33203042020 proved the plumbing (the failing spec's trace.zip and error-context.md were retained under test.outputs/playwright/…) and then Bazel refused the test: output tree artifact playwright_chromium_e2e/test.outputs was not created. Bazel 8 keeps undeclared outputs as an unzipped tree artifact by default; the Playwright spec directory names are non-ASCII (search-Search-—-color-mode-…). HEAD adds test --zip_undeclared_test_outputs so it is a single outputs.zip (the evidence step already unzips it).

Note for the record: this is the second 30 s locator.evaluate timeout today in e2e/search.spec.ts › color mode … in light and dark modes (also #263's first gates run) — that is an intermittent, not a one-off; its trace will now be attached.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant