ci(codecov): hold coverage statuses until both uploads land - #668
Draft
Acruve15 wants to merge 1 commit into
Draft
ci(codecov): hold coverage statuses until both uploads land#668Acruve15 wants to merge 1 commit into
Acruve15 wants to merge 1 commit into
Conversation
Coverage arrives in two uploads from two independent jobs: the backend suite (--cov=app, ~3100 lines) and the client suite (--cov=pyroclient, ~135 lines). With no codecov.yml in the repo the project status was evaluated on the first upload to arrive, so whenever the client job finished first the check compared a 3-file report against the full base, reported a ~2.5% project drop and failed the PR, then silently recovered once the backend upload landed. That produced four spurious failures across two PRs in a day, each resolving on its own within minutes, which trains reviewers to ignore a red check. notify.after_n_builds holds statuses and the PR comment until both uploads are in, so they are only ever computed on a complete report. Both upload steps now carry a flag so each job's contribution is identifiable and a genuinely missing upload shows up as such rather than silently lowering the total. The project status also gets a 0.5% threshold so small unavoidable swings do not block. Validated against https://codecov.io/validate ("Valid!"); note after_n_builds is only accepted under codecov.notify, not per status.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #668 +/- ##
=======================================
Coverage 93.73% 93.73%
=======================================
Files 59 59
Lines 3143 3143
=======================================
Hits 2946 2946
Misses 197 197
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
MateoLostanlen
approved these changes
Aug 12, 2026
MateoLostanlen
left a comment
Member
There was a problem hiding this comment.
All good for me thnaks for this fix !
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
codecov/projectcheck fails spuriously on most pushes, then fixes itself a couple of minutes later.Cause
Coverage arrives in two uploads from two independent jobs:
pytestpytest --cov=apppytest-clientpytest --cov=pyroclientThere is no
codecov.ymlin the repo, so Codecov runs on defaults and evaluates the project status as soon as the first upload lands. The client job is much faster, so it usually wins, and the status then compares a 3-file report against the full base:The check goes red, the PR looks broken, and once the backend upload arrives the same check silently flips to success. This produced four spurious failures across two PRs in a single day (#661 twice, #664 twice), every one of them resolving on its own. The practical harm is that it trains reviewers to ignore a red check.
Fix
codecov.notify.after_n_builds: 2holds statuses and the PR comment until both uploads are in, so they are only ever computed on a complete report.Both upload steps now also carry a
flags:value (backend/client). That is not cosmetic: with flags, each job's contribution is identifiable in the Codecov UI, and an upload that genuinely goes missing shows up as a missing flag rather than silently dragging the total down (which is the failure mode this PR exists to fix, just permanent instead of transient).The project status additionally gets
threshold: 0.5%so a refactor that moves covered lines around does not block a PR over noise.Verification
Validated against Codecov's own validator rather than assumed to parse:
Returns
Valid!. Worth recording one finding from that:after_n_buildsis not accepted undercoverage.status.project.defaultorcoverage.status.patch.default(the validator rejects it as an unknown field there). It is only valid undercodecov.notify, which is what this config uses. My first attempt had it per-status and was invalid.The real proof is behavioural and only observable after merge: pushes should stop producing a transient red
codecov/project. If it still flaps, the next lever iscodecov.require_ci_to_passor raisingafter_n_builds.Notes for review
codecov/patchkeeps requiring the diff to be covered.after_n_builds: 2is coupled to there being exactly two coverage uploads. If a third coverage-producing job is ever added, this number needs bumping, otherwise statuses would fire early again. The comment incodecov.ymlsays so.