Skip to content

The test job gets twelve minutes, not seven, as the suite nears 3700 tests - #534

Merged
joaocarvoli merged 3 commits into
mainfrom
joao/eng-1092-the-shema-api-test-job-hits-its-7-minute-timeout-as-the
Sep 25, 2026
Merged

joaocarvoli merged 3 commits into
mainfrom
joao/eng-1092-the-shema-api-test-job-hits-its-7-minute-timeout-as-the

Conversation

@joaocarvoli

Copy link
Copy Markdown
Member

The test job gets twelve minutes instead of seven.

What happens today

.github/workflows/test.yml caps the test job at 7 minutes. A normal run takes about three, but the time spreads unevenly across runners: PR #527 (run 35973409359) and PR #531 (run 35987830425) were cancelled at 7m15s with the suite at 98% and no failure, and the test check read as failed. The suite is close to 3700 tests.

What changes

timeout-minutes: 7 → 12. The ceiling stays on the job (ENG-913's reason: a hung run must turn red, not sit pending for six hours), with headroom for the suite's growth. The comment above it is unchanged: its measurements are still true and the reason for a ceiling is the same.

Not done here: sharding the suite or hunting the slowest files (--durations); the ticket lists them as alternatives, and the one-line ceiling is enough today.

Invisible; CI only.

🤖 Generated with Claude Code

…ears 3700 tests

A normal run takes about three minutes, but the time spreads unevenly
across runners: PR #527 (run 35973409359) and PR #531 (run 35987830425)
were cancelled at 7m15s with the suite at 98% and no failure, and the
`test` check read as failed. Twelve minutes leaves headroom without
hiding a hang, which the per-test bounds still catch.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Sep 24, 2026

Copy link
Copy Markdown

ENG-1092

# The ceiling is on the job. Without one it inherits GitHub's 360-minute default, which
# is how a hung run stayed "pending" for six hours instead of turning red (ENG-913).
timeout-minutes: 7
timeout-minutes: 12

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tests/test_ci_gates.py:151 pins this value — ("test.yml", "test"): 7 — and the case at :166 asserts timeout == minutes. That case carries no mark, so it runs in this job's own selection, not migration and not fresh_interpreter: the test check turns red on the one-line change that exists to stop it turning red. Bash here would not run pytest for me, so that is read from the file rather than executed. The pin has to move to 12 with the workflow.

Comment thread .github/workflows/test.yml Outdated
@@ -25,7 +25,7 @@ jobs:
# 2m32-3m06 (five runs: 35653484356, 35653965653, 35654431223, 35655564859, 35655963815).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two prose records of seven survive the change. This comment block opens on line 23 with "Seven, over a Run tests step measured at 2m04-2m31…" — the measurements are still true, as the PR body says, but the first word is the ceiling itself, and it now reads Seven five lines above 12. docs/ci.md:20 carries the other: "under a 7-minute ceiling, against a step measured at 2m04-2m31 on the runner since ENG-980". Either edit both, or drop the leading number here and let the line under it be the value.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pin moved in 2b8618ae; these did not. At head .github/workflows/test.yml:23 still opens "Seven, over a Run tests step measured at 2m04-2m31…" five lines above timeout-minutes: 12, and docs/ci.md:20 still reads "under a 7-minute ceiling, against a step measured at 2m04-2m31 on the runner since ENG-980". grep -rn '7-minute|timeout-minutes: 7' docs/ .github/ tests/ returns those two and migrations.yml:45, which is its own job and not in scope. The same commit also added a third record that disagrees with both: tests/test_ci_gates.py:149 now says the step "measured 6m21s on #534". Same either/or as before — edit both, or drop the leading number here and let the line under it be the value.

@little-henok little-henok Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the one-line ceiling change and everything in the tree that records that ceiling. Two findings, and they share a cause: the number lives in four places and this moves one of them. The pinned one is the one I would least want found by CI — tests/test_ci_gates.py asserts 7, runs in this very job's selection, and turns the test check red on the change written to stop it turning red. Not covering the alternatives the PR body sets aside, sharding and --durations. That is the complete list: two, nothing else.

The test that keeps every job under a timeout still pinned the test job at
seven minutes, so raising the workflow alone turned that test red on the
PR's own run. The pin follows the workflow, with the measurement that
justified it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@joaocarvoli
joaocarvoli requested review from henokteixeira and removed request for henokteixeira September 25, 2026 02:09

@little-henok little-henok Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pass 2 — re-checked the two findings from pass 1 against 2b8618ae, nothing else. The gate pin moved with the workflow. The prose records did not: test.yml:23 still opens on Seven and docs/ci.md:20 still says a 7-minute ceiling, and the commit that fixed the pin added a third measurement that disagrees with both. Replied on that thread. One of two still open, no new findings, and the set from pass 1 is those two with nothing else in it. Not covering the other jobs' ceilings in that table — only the test row moved here.

…ured it

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@joaocarvoli
joaocarvoli requested review from henokteixeira and removed request for henokteixeira September 25, 2026 02:15

@little-henok little-henok Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the three changed files at 911c10a7, against the two points from the first pass. Both are closed: tests/test_ci_gates.py:153 pins 12, and grep -rn '7-minute|seven-minute|timeout-minutes: 7' docs/ .github/ tests/ now returns only migrations.yml:45, that job's own ceiling and not in scope, and the new comment's narration of the old value. Nothing new from me — everything added since 2b8618ae is comment and documentation prose. That is the whole set; there is nothing else outstanding here.

@joaocarvoli
joaocarvoli merged commit 7740493 into main Sep 25, 2026
6 checks passed
@joaocarvoli
joaocarvoli deleted the joao/eng-1092-the-shema-api-test-job-hits-its-7-minute-timeout-as-the branch September 25, 2026 02:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant