Skip to content

ci(codecov): hold coverage statuses until both uploads land - #668

Draft
Acruve15 wants to merge 1 commit into
mainfrom
alexis/codecov-wait-for-both-uploads
Draft

ci(codecov): hold coverage statuses until both uploads land#668
Acruve15 wants to merge 1 commit into
mainfrom
alexis/codecov-wait-for-both-uploads

Conversation

@Acruve15

Copy link
Copy Markdown
Collaborator

The codecov/project check fails spuriously on most pushes, then fixes itself a couple of minutes later.

Cause

Coverage arrives in two uploads from two independent jobs:

Job Command Report
pytest pytest --cov=app 59 files, ~3106 lines
pytest-client pytest --cov=pyroclient 3 files, ~135 lines

There is no codecov.yml in the repo, so Codecov runs on defaults and evaluates the project status as soon as the first upload lands. The client job is much faster, so it usually wins, and the status then compares a 3-file report against the full base:

Files          59        3      -56
Lines        3106      135    -2971
- Coverage   93.56%   91.11%   -2.45%

The check goes red, the PR looks broken, and once the backend upload arrives the same check silently flips to success. This produced four spurious failures across two PRs in a single day (#661 twice, #664 twice), every one of them resolving on its own. The practical harm is that it trains reviewers to ignore a red check.

Fix

codecov.notify.after_n_builds: 2 holds statuses and the PR comment until both uploads are in, so they are only ever computed on a complete report.

Both upload steps now also carry a flags: value (backend / client). That is not cosmetic: with flags, each job's contribution is identifiable in the Codecov UI, and an upload that genuinely goes missing shows up as a missing flag rather than silently dragging the total down (which is the failure mode this PR exists to fix, just permanent instead of transient).

The project status additionally gets threshold: 0.5% so a refactor that moves covered lines around does not block a PR over noise.

Verification

Validated against Codecov's own validator rather than assumed to parse:

curl -s --data-binary @codecov.yml https://codecov.io/validate

Returns Valid!. Worth recording one finding from that: after_n_builds is not accepted under coverage.status.project.default or coverage.status.patch.default (the validator rejects it as an unknown field there). It is only valid under codecov.notify, which is what this config uses. My first attempt had it per-status and was invalid.

The real proof is behavioural and only observable after merge: pushes should stop producing a transient red codecov/project. If it still flaps, the next lever is codecov.require_ci_to_pass or raising after_n_builds.

Notes for review

  • Both statuses are still enforced, just delayed. This does not weaken the coverage gate, and codecov/patch keeps requiring the diff to be covered.
  • after_n_builds: 2 is coupled to there being exactly two coverage uploads. If a third coverage-producing job is ever added, this number needs bumping, otherwise statuses would fire early again. The comment in codecov.yml says so.

Coverage arrives in two uploads from two independent jobs: the backend suite
(--cov=app, ~3100 lines) and the client suite (--cov=pyroclient, ~135 lines).
With no codecov.yml in the repo the project status was evaluated on the first
upload to arrive, so whenever the client job finished first the check compared a
3-file report against the full base, reported a ~2.5% project drop and failed the
PR, then silently recovered once the backend upload landed.

That produced four spurious failures across two PRs in a day, each resolving on
its own within minutes, which trains reviewers to ignore a red check.

notify.after_n_builds holds statuses and the PR comment until both uploads are
in, so they are only ever computed on a complete report. Both upload steps now
carry a flag so each job's contribution is identifiable and a genuinely missing
upload shows up as such rather than silently lowering the total. The project
status also gets a 0.5% threshold so small unavoidable swings do not block.

Validated against https://codecov.io/validate ("Valid!"); note after_n_builds is
only accepted under codecov.notify, not per status.
@codecov

codecov Bot commented Jul 31, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.73%. Comparing base (5c73e72) to head (1d6c059).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #668   +/-   ##
=======================================
  Coverage   93.73%   93.73%           
=======================================
  Files          59       59           
  Lines        3143     3143           
=======================================
  Hits         2946     2946           
  Misses        197      197           
Flag Coverage Δ
backend 93.84% <ø> (?)
client 91.11% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@MateoLostanlen MateoLostanlen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good for me thnaks for this fix !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants