fix(ci): stop the release audit calling a healthy publish broken - #567
Merged
Merged
Conversation
Two of the release verification checks read a registry once, or for only a minute, and report a release that shipped correctly as failed. The v0.36.0 run went red on both while every artifact was fine, which is the state that trains everyone to ignore a red release. - verify_npm_sdk.sh polled npm for 60 s. a03d2b0 already measured that as too short and widened verify_npm.sh to five minutes, but never applied it here, so this script false-failed v0.35.1 and v0.36.0: npm logged `+ crw-sdk@<version>` and warned the package "may take a few minutes to become available", then the check gave up at 60 s. Brought to parity with its sibling, existence poll and install-smoke retry both. - verify_docker.sh read each tag once. The v0.36.0 audit called ghcr.io/us/crw:latest missing while :0.36.0 and :0.36, pushed in the same operation, passed; latest resolved to the same digest as 0.36.0 minutes later. Now polls, with the deadline shared across tags so a genuinely failed release is still reported in five minutes rather than fifteen. Measured against the live registries, both directions: verify_npm_sdk.sh 0.36.0 -> exit 0 in 2 s verify_npm_sdk.sh 99.99.99 -> exit 1 in 325 s verify_docker.sh 0.36.0 -> exit 0, all three tags, both arches verify_docker.sh 99.99.99 -> exit 1 in 318 s The v0.36.0 audit's third finding, crw-mcp-linux-x64@0.36.0 missing from npm, was a true failure and is not masked by this: that package published on a re-run and a linux/x64 install now resolves it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The v0.36.0 release audit went red on three findings. One was real. Two were the
verification checks reading a registry too eagerly and calling a healthy publish broken.
That mix is the problem: a release that is red for reasons nobody trusts is a release
whose real failure nobody sees, and today the real one was only found by hand.
What was wrong
verify_npm_sdk.shpolled npm for 60 seconds.a03d2b0already measured that as tooshort on the 0.34.0 release and widened
verify_npm.shto five minutes, but the change wasnever applied to this script. So it false-failed both v0.35.1 and v0.36.0 in exactly the
same way: npm logged
+ crw-sdk@<version>, warned that the package "may take a few minutesto become available", and the check gave up at 60 s while the release itself was fine.
crw-sdk@0.36.0is on npm and installs correctly.verify_docker.shread each tag once. The audit calledghcr.io/us/crw:latestmissingwhile
:0.36.0and:0.36, pushed in the same operation, both passed.latestnowresolves to
sha256:8c74c910..., byte for byte the0.36.0digest, so the tag was simplynot readable yet when the check ran.
What changed
verify_npm_sdk.shcomes to parity with its sibling: the same 30 x 10 s existence polland the same install-smoke retry.
verify_docker.shpolls each tag, with the deadline shared across tags rather thanper tag, so a genuinely failed release is still reported in five minutes and not in
fifteen.
Measured, against the live registries, both directions
Both still fail for a genuinely missing artifact, which is the point: the window moved, the
verdict did not.
The real finding is not masked by this
crw-mcp-linux-x64@0.36.0was genuinely absent from npm even though the publish step logged+ crw-mcp-linux-x64@0.36.0.verify_npm.shalready polls five minutes and reported itcorrectly, so that check needed no change. It has since published on a workflow re-run, and
a linux/x64 install now resolves the platform package:
Before the re-run that install produced the wrapper alone, and the launcher's
GitHub-download fallback would have covered it at the cost of a 10 MB download on first run.
Cost
A genuinely failed release now takes up to five minutes per check to report instead of
failing fast. That is the trade this PR is making on purpose.