ci: make the toolchain currency report able to say clean - #29
Merged
Conversation
The shape of `mise outdated --json` depends on whether the tools are installed.
This job reports without installing, where mise emits one entry per tool
carrying a null bump rather than an empty object — so comparing the payload
against {} was true on every run, and the report has never been able to say
clean. It has run three times and failed all three: twice with nothing behind
at all, and once where a pin genuinely had moved, which is the only reason the
mechanism ever looked like it worked.
Keying on bump answers the question in either state, and the pins it finds are
printed into the run log so a reader need not re-derive them from the table
above.
An empty payload is asserted against rather than read as a verdict, because
that is the failure direction this monitor cannot survive. Nothing notifies on
a green run, so a reporter that has gone blind — an unreachable mise.toml, a
changed working directory — would otherwise report clean indefinitely. Each jq
call is guarded and names its own failure, for the same reason: an unguarded
one exits under errexit with no annotation, and a shared message would name the
wrong cause.
The report distinguishes a stale pin from a broken reporter, but the security policy still described red as meaning only the first — so a reader hitting a reporter failure had nothing telling them the pins were never assessed. The schema-drift gap is named alongside it, since that one fails quiet and a reader has no other way to learn it exists.
There was a problem hiding this comment.
Pull request overview
Fixes the Toolchain currency GitHub Actions workflow so it can accurately report when mise-managed pins are behind upstream (instead of failing unconditionally), and updates SECURITY.md to document the reporter’s new failure modes and limitations.
Changes:
- Update the workflow to determine “behind” status by checking
bump != nullper tool (robust across mise’s differing JSON shapes). - Add guarded
jqprocessing with explicit, cause-specific failure annotations, and print the specific behind pins when failing. - Clarify in
SECURITY.mdwhat a red run means (pin behind vs reporter failure vs processing failure) and note the remaining schema-change blind spot.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
SECURITY.md |
Updates CI/toolchain supply-chain documentation to match the corrected workflow behavior and clarify interpretation of failures. |
.github/workflows/toolchain.yml |
Fixes the “pins behind upstream” detection logic and improves failure attribution/visibility in the toolchain currency report. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Toolchain currencyreported "pins are behind upstream" on every run whatever the pins actually were, so its red carried no information. It had run three times and failed all three — twice with nothing behind, and once where a pin genuinely had moved, which is the only reason the mechanism ever looked like it worked.mise outdated --jsonchanges shape with installation state. This job reports without installing, and in that state mise emits an entry per pinned tool carrying a nullbumprather than an empty object — so the!= "{}"comparison the check decided on was true unconditionally. Keying onbumpanswers the question in either state, and the pins it finds are now printed so a reader need not re-derive them from the table above.Three claims elsewhere in the repo become true with this, having been unsatisfiable before:
SECURITY.mdandCONTRIBUTING.mdboth state that the report fails when a pin is behind, and.claude/rules/toolchain-ci-parity.mdstates that forgetting a workflow pin "leaves that report green the following week."How it works
A monitor's two failure directions are not equals. Loudly wrong wastes attention; quietly blind destroys the signal while still looking healthy — and nothing notifies on a green run, so the quiet direction is the one this cannot survive. An empty payload is therefore asserted against rather than read as a verdict, which holds because reporting without installing guarantees an entry per pinned tool. Each
jqcall is guarded and names its own failure for the same reason: unguarded, one would exit under errexit with no annotation, and a shared message would name the wrong cause.One gap stays open, and is stated in both the workflow comment and
SECURITY.mdrather than left implicit: a renamed or droppedbumpfield would leave the report matching nothing and falling quiet. A run that stops failing across a mise upgrade is the symptom to distrust.Issue references
Related to #16 — this adds more shell to a file that no pre-merge check exercises. Worth bounding the connection, though: the defects here were runtime-semantic, so a linter would not have caught them.