From 9c60642b8de67ae88c95fe01393f3326d18334fa Mon Sep 17 00:00:00 2001 From: "Michael C. Ferguson" Date: Thu, 16 Jul 2026 12:43:21 -0500 Subject: [PATCH 1/2] ci: add the fork-sync backport queue This repo is the source of truth for the MiSTer kernel, but the fork is a live repo other people commit to. Anything landing there after our last reconciliation is something we do not have and nobody has decided about -- and nothing ever forces that question. That is not hypothetical: the fork sat on 5.15.1 through 210 stable releases partly because no moment ever said "these N commits are unaccounted for", and this session alone hit two stale-baseline bugs, one of which nearly shipped an auto-overclocking kernel. Three pieces: - docs/kernel-recon/fork-sync.conf -- the last RECONCILED commit per fork branch. Previously this existed only as prose in docs/patch-provenance.md's header ("HEAD f0fb626acadd..."), which cannot be diffed against anything. Recorded at 794e6f002 (v5.15) and d9ac12a691 (v6.18); the header spells out that advancing a line without dispositioning what it skips is the one thing that breaks the mechanism -- it would stop meaning "reconciled" and start meaning "seen". - scripts/check-fork-sync.sh -- diffs that file against the fork's live HEADs via the compare API. One call per branch, no clone: the kernel repo is ~300MB and answering "what is new" does not need it. Exit 0 reconciled / 1 triage / 2 error. - .github/workflows/fork-sync.yml -- weekly, plus dispatch. Seconds per run, no build, no checkout of the kernel; far below the noise floor of build.yml's ~3h cold path, which is what makes a schedule affordable here at all. Watches BOTH branches. MiSTer-v6.18 is the one that matters most: once #75 merges, commits landing there are changes made to OUR series by other people, and they must flow back into linux-patches/ or the next regeneration silently erases them. Design points that are deliberate, not incidental: - ONE issue, edited in place, closed when the queue empties. A weekly job that files a fresh issue is a job people mute, and a muted queue is the same as no queue. - Exit 1 (commits to triage) does NOT fail the job -- a red X every week trains people to ignore it. Only exit 2 (missing branch, malformed conf) fails. - `gh issue create` prints a URL and has no --json/--jq; the number comes off the URL rather than from a `||` fallback around a second create, which is how you file two issues. Verified: reports "fully reconciled" today (exit 0); rewinding the v5.15 pointer to the old recon baseline f0fb626ac correctly flags 794e6f002 "New driver for RTL8821CU" (exit 1) and renders the markdown issue body. YAML parses, both embedded run blocks pass `bash -n`, shellcheck clean. Co-Authored-By: Claude Opus 4.8 --- .github/workflows/fork-sync.yml | 127 +++++++++++++++++++++++++++++++ docs/kernel-recon/fork-sync.conf | 51 +++++++++++++ scripts/check-fork-sync.sh | 121 +++++++++++++++++++++++++++++ 3 files changed, 299 insertions(+) create mode 100644 .github/workflows/fork-sync.yml create mode 100644 docs/kernel-recon/fork-sync.conf create mode 100755 scripts/check-fork-sync.sh diff --git a/.github/workflows/fork-sync.yml b/.github/workflows/fork-sync.yml new file mode 100644 index 0000000..e23d47b --- /dev/null +++ b/.github/workflows/fork-sync.yml @@ -0,0 +1,127 @@ +################################################################################ +# +# fork-sync.yml — weekly backport queue for MiSTer-devel/Linux-Kernel_MiSTer. +# +# This repo is the source of truth for the MiSTer kernel, but the fork is a live repo +# other people commit to. Anything landing there after our last reconciliation is +# something we do not have and nobody has decided about. Nothing else forces that +# question -- and the question does not get asked on its own: the fork sat on 5.15.1 +# through 210 stable releases partly because no moment ever said "these N commits are +# unaccounted for". +# +# So: scripts/check-fork-sync.sh diffs docs/kernel-recon/fork-sync.conf (the last +# RECONCILED commit per fork branch) against the fork's live HEADs, and this opens an +# issue listing whatever has landed since. The queue becomes a fact rather than a memory. +# +# The direction that matters most is MiSTer-v6.18: once #75 merges upstream, commits +# landing there are changes made to OUR series by other people, and they must flow back +# into board/mister/de10nano/linux-patches/ or the next `make export` silently erases +# them. That is the failure this file exists to prevent. +# +# COST: two compare API calls and no clone -- seconds, weekly. It deliberately does NOT +# check out the kernel (~300MB) or build anything; "what is new upstream" does not need +# either. This is why it can be a schedule rather than something manual: it is far below +# the noise floor of the build workflow. (Contrast build.yml, whose cold path is ~3h.) +# +# ONE ISSUE, UPDATED -- not one per run. A weekly job that opens a fresh issue is a job +# people mute, and a muted queue is the same as no queue. It finds its own open issue by +# label, edits it in place, and closes it when the queue empties. +# +################################################################################ + +name: Fork sync queue + +on: + schedule: + # Mondays 06:00 UTC. Weekly is matched to how fast the fork actually moves (11 + # commits across 2026 so far) -- daily would just re-post the same list six more + # times before anyone acts on it. + - cron: "0 6 * * 1" + workflow_dispatch: + +# Only the issue write. No checkout of anything but this repo, no packages, no build. +permissions: + contents: read + issues: write + +concurrency: + group: fork-sync + cancel-in-progress: false + +jobs: + check: + name: Check fork for unreconciled commits + runs-on: ubuntu-latest + timeout-minutes: 5 + + steps: + - name: Checkout + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + + # Exit 1 from the script means "commits need triage", which is a normal state and + # must not fail the job -- a red X every week trains people to ignore it. The + # report is the product; the exit code only routes it. + - name: Diff fork HEADs against the reconciled sync points + id: check + env: + GH_TOKEN: ${{ github.token }} + run: | + set -o pipefail + if scripts/check-fork-sync.sh > text-report.txt 2>&1; then + echo "drift=false" >> "$GITHUB_OUTPUT" + else + rc=$? + # 2 is a real error (missing branch, malformed conf) -- that SHOULD fail. + if [ "$rc" -ne 1 ]; then + echo "::error title=fork-sync check failed::scripts/check-fork-sync.sh exited $rc" + cat text-report.txt + exit "$rc" + fi + echo "drift=true" >> "$GITHUB_OUTPUT" + fi + cat text-report.txt + scripts/check-fork-sync.sh --markdown > body.md || true + + - name: Open or update the backport-queue issue + env: + GH_TOKEN: ${{ github.token }} + DRIFT: ${{ steps.check.outputs.drift }} + REPO: ${{ github.repository }} + run: | + set -euo pipefail + TITLE="Backport queue: unreconciled commits in Linux-Kernel_MiSTer" + LABEL="fork-sync" + + # Label may not exist yet on a fresh repo; create it once, idempotently. + gh label create "$LABEL" --repo "$REPO" --color 0E8A16 \ + --description "Upstream fork commits awaiting a disposition" 2>/dev/null || true + + existing="$(gh issue list --repo "$REPO" --label "$LABEL" --state open \ + --json number --jq '.[0].number // empty')" + + if [ "$DRIFT" = "true" ]; then + { + cat body.md + printf '\n---\nGenerated by `.github/workflows/fork-sync.yml` — ' + printf 'edited in place each run, so this is always the current queue. ' + printf 'Closes itself when everything has a disposition.\n' + } > issue.md + + if [ -n "$existing" ]; then + gh issue edit "$existing" --repo "$REPO" --body-file issue.md + echo "::notice title=Backport queue updated::Issue #$existing refreshed." + else + # `gh issue create` has no --json/--jq; it prints the issue URL. Take the + # number off the end rather than retrying the command -- a `||` fallback + # around a create is how you end up filing two issues. + url="$(gh issue create --repo "$REPO" --title "$TITLE" --label "$LABEL" \ + --body-file issue.md)" + echo "::notice title=Backport queue opened::Issue ${url##*/} lists commits needing triage." + fi + elif [ -n "$existing" ]; then + gh issue close "$existing" --repo "$REPO" \ + --comment "Everything on the watched fork branches now has a disposition in \`docs/patch-provenance.md\`, and \`docs/kernel-recon/fork-sync.conf\` has been advanced past it. Reopening automatically if the fork moves again." + echo "::notice title=Backport queue empty::Closed issue #$existing." + else + echo "::notice title=Backport queue empty::Fork fully reconciled; nothing to do." + fi diff --git a/docs/kernel-recon/fork-sync.conf b/docs/kernel-recon/fork-sync.conf new file mode 100644 index 0000000..7513626 --- /dev/null +++ b/docs/kernel-recon/fork-sync.conf @@ -0,0 +1,51 @@ +# fork-sync.conf — how far we have reconciled MiSTer-devel/Linux-Kernel_MiSTer. +# +# WHY THIS FILE EXISTS +# -------------------- +# This repo is the source of truth for the MiSTer kernel: the patch series in +# board/mister/de10nano/linux-patches/ is what gets built, and scripts/export-kernel-tree.sh +# renders it into the fork's tree format. But the fork is a live repo that other people +# commit to. Anything landing there after our last reconciliation is, by definition, +# something this repo does not have and nobody has decided about. +# +# Left to humans that decision never gets made -- not through neglect, but because there +# is no moment that forces the question. The fork sat on 5.15.1 for 210 stable releases +# for exactly this reason: nothing ever said "these N commits are unaccounted for". +# +# So this file records the last commit on each fork branch whose content has a disposition +# in docs/patch-provenance.md, and scripts/check-fork-sync.sh diffs it against the fork's +# live HEADs. .github/workflows/fork-sync.yml runs that weekly and opens an issue when the +# answer is non-empty. The queue is then a fact rather than a memory. +# +# FORMAT +# +# Comments and blank lines ignored. Full SHAs only -- an abbreviation that is unambiguous +# today can collide later, and this is compared, not just displayed. +# +# UPDATING +# Move a line ONLY when the commits it skips past have been dispositioned in +# docs/patch-provenance.md -- carried into the series, or recorded as deliberately dropped +# with a reason. Advancing it to silence the issue is the one thing that breaks this: the +# file stops meaning "reconciled" and starts meaning "seen", which is what it exists to +# prevent. +# +# WHY BOTH BRANCHES +# MiSTer-v5.15 -- the fork's historical line and the baseline the whole reconciliation +# was performed against (docs/patch-provenance.md's header names f0fb626ac; this is +# now ahead of that, see below). Still actively committed to. +# MiSTer-v6.18 -- upstream's own vanilla 6.18.38 base, created 2026-07-16, which our +# export now replays onto (MiSTer-devel/Linux-Kernel_MiSTer#75). Once that merges, +# commits landing here are changes made to OUR series by other people, and they have +# to flow back into linux-patches/ or the next regeneration silently erases them. +# This is the direction that matters most for keeping this repo the source of truth. + +# Reconciled: docs/patch-provenance.md dispositioned every commit up to f0fb626ac. +# Advanced to 794e6f002 on 2026-07-16: the single commit since (794e6f002, "New driver for +# RTL8821CU") is dispositioned -- that chip is covered by mainline rtw88_8821cu +# (CONFIG_RTW88_8821CU=m), which is why BR2_PACKAGE_RTL8821CU_MORROWNR is deliberately off +# in configs/mister_de10nano_defconfig. Nothing to carry. +MiSTer-v5.15 794e6f002d0f655c504733c126a01f8c1f0bc1d4 + +# Upstream's pristine v6.18.38 base commit, and the parent our export replays onto. +# Nothing of ours or theirs sits on top of it upstream yet. +MiSTer-v6.18 d9ac12a691ead295c8bc6438754767b94c0f26a2 diff --git a/scripts/check-fork-sync.sh b/scripts/check-fork-sync.sh new file mode 100755 index 0000000..ee446cf --- /dev/null +++ b/scripts/check-fork-sync.sh @@ -0,0 +1,121 @@ +#!/usr/bin/env bash +# +# check-fork-sync.sh — report fork commits we have not reconciled yet. +# +# Diffs docs/kernel-recon/fork-sync.conf (the last reconciled commit per fork branch) +# against MiSTer-devel/Linux-Kernel_MiSTer's live HEADs, and prints what has landed since. +# That list is the backport queue: each commit needs a disposition in +# docs/patch-provenance.md -- carried into board/mister/de10nano/linux-patches/, or +# recorded as deliberately dropped with a reason. +# +# Read fork-sync.conf's header for why this exists at all; the short version is that +# nothing else ever forces the question, and the fork sat on 5.15.1 for 210 stable +# releases partly because of that. +# +# Cheap by construction: one compare API call per branch, no clone. The kernel repo is +# ~300MB and there is no reason to fetch it to answer "what is new". +# +# Usage: scripts/check-fork-sync.sh [--markdown] +# --markdown emit a GitHub-flavoured report (used by the workflow for issue bodies) +# +# Exit: 0 = fully reconciled; 1 = commits need triage (report on stdout); 2 = error. +# The workflow keys off 1 vs 0, so do not make drift fatal-looking; it is normal. + +set -o errexit +set -o nounset +set -o pipefail + +# Assigned then marked readonly separately: `readonly X="$(cmd)"` masks cmd's exit status +# (shellcheck SC2155), and the rest of scripts/ avoids that pattern. +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +readonly REPO_ROOT +readonly CONF="$REPO_ROOT/docs/kernel-recon/fork-sync.conf" +readonly FORK="${FORK_REPO:-MiSTer-devel/Linux-Kernel_MiSTer}" + +markdown=false +[[ ${1:-} == --markdown ]] && markdown=true + +err() { printf 'check-fork-sync: %s\n' "$*" >&2; } + +command -v gh >/dev/null || { err 'gh CLI not found'; exit 2; } +[[ -f $CONF ]] || { err "no such file: $CONF"; exit 2; } + +drift=0 +report='' + +# One line per branch: " ". Comments and blanks stripped. +while read -r branch sync; do + [[ -n ${branch:-} ]] || continue + + [[ ${#sync} -eq 40 ]] || { err "$branch: sync point must be a full 40-char SHA, got '$sync'"; exit 2; } + + # compare gives us ahead_by + the commit list without cloning anything. `...` is + # three-dot on purpose: we want commits reachable from the branch but not from the + # sync point, which is exactly "what landed since we last looked". + if ! cmp_json="$(gh api "repos/$FORK/compare/$sync...$branch" 2>/dev/null)"; then + # A missing branch is worth failing on: it means the fork restructured and this + # file is now describing something that does not exist. + err "cannot compare $sync...$branch in $FORK (branch gone, or SHA not an ancestor?)" + exit 2 + fi + + ahead="$(jq -r '.ahead_by' <<<"$cmp_json")" + + if [[ $ahead -eq 0 ]]; then + if $markdown; then + report+="- ✅ \`$branch\` — reconciled through \`${sync:0:9}\`, nothing new. +" + else + report+=" [ok] $branch — reconciled through ${sync:0:9}, nothing new +" + fi + continue + fi + + drift=$((drift + 1)) + if $markdown; then + report+=" +### \`$branch\` — $ahead commit(s) to triage + +Reconciled through [\`${sync:0:9}\`](https://github.com/$FORK/commit/$sync). Since then: + +| commit | subject | author | date | +|---|---|---|---| +" + report+="$(jq -r --arg f "$FORK" '.commits[] | + "| [`\(.sha[0:9])`](https://github.com/\($f)/commit/\(.sha)) | \(.commit.message | split("\n")[0] | gsub("\\|"; "\\\\|")) | \(.commit.author.name) | \(.commit.author.date[0:10]) |"' <<<"$cmp_json") +" + else + report+=" + [TRIAGE] $branch — $ahead commit(s) since ${sync:0:9} +" + report+="$(jq -r '.commits[] | " \(.sha[0:9]) \(.commit.author.date[0:10]) \(.commit.message | split("\n")[0])"' <<<"$cmp_json") +" + fi +done < <(grep -vE '^\s*(#|$)' "$CONF") + +if $markdown; then + if ((drift)); then + printf '%s\n' "The fork has commits with no disposition in this repo. Each needs one of: + +- **carried** → a patch in \`board/mister/de10nano/linux-patches/\`, or +- **dropped** → a row in \`docs/patch-provenance.md\` saying so, and why (superseded upstream, packaged separately, obsolete…). + +Then advance the branch's line in \`docs/kernel-recon/fork-sync.conf\`. + +> Advancing that file to silence this issue is the one thing that breaks the mechanism — it would stop meaning *reconciled* and start meaning *seen*. +$report" + else + printf '%s\n' "$report +Nothing to triage." + fi +else + printf '=== fork sync: %s\n%s\n' "$FORK" "$report" + if ((drift)); then + printf 'RESULT: %d branch(es) need triage — see docs/patch-provenance.md\n' "$drift" + else + printf 'RESULT: fully reconciled\n' + fi +fi + +((drift == 0)) From 57a67792393b115126f48d5a75cdf06bab5ef7fc Mon Sep 17 00:00:00 2001 From: "Michael C. Ferguson" Date: Thu, 16 Jul 2026 12:54:35 -0500 Subject: [PATCH 2/2] fork-sync: refuse to report "reconciled" from a response we cannot read Four review findings on #28, all valid. One of them was worse than reported. `ahead_by` was used unguarded. Bash arithmetic treats a bare word as an unset variable, so BOTH `[[ null -eq 0 ]]` and `[[ "" -eq 0 ]]` evaluate TRUE -- verified. The review said an unexpected response could produce a bogus triage report; it is the opposite and worse. Any response that parsed but had no ahead_by (auth failure, rate limit, partial body) made the script print "nothing new" and exit 0: a silent all-clear. A tool whose entire purpose is to stop commits going unaccounted for had a path where it silently said all-clear because it could not tell. Now the value must match ^[0-9]+$ or it exits 2. Verified with a stand-in `gh` returning {"message":"Bad credentials"}: exits 2 naming the branch, instead of reporting reconciled. jq was invoked three times and checked zero. Now checked alongside gh, so a missing dep is exit 2 with context rather than 127 from somewhere inside a pipeline. The workflow called the script TWICE -- plain for the log, --markdown for the issue body with `|| true` to swallow the expected exit 1. Wrong twice over: `|| true` also swallows exit 2, so a broken run could file an issue with an empty body and still go green; and two invocations are two sets of API calls that can disagree if one hits a transient failure. Now one --markdown invocation feeds both, with the exit code routed through a case (0 clean / 1 triage / 2 fail), and an explicit assertion that the body is non-empty before it can be posted -- which is the failure the single invocation exists to close, so it is checked rather than assumed. Halves the API calls as a side effect. `gh issue list` now passes --limit 1: only the first match is used, and the default page of 30 is API work nobody looks at. Verified: shellcheck clean, YAML parses, both run blocks pass `bash -n`, the normal path still reports fully-reconciled (exit 0), and the markdown body renders. Co-Authored-By: Claude Opus 4.8 --- .github/workflows/fork-sync.yml | 44 ++++++++++++++++++++++----------- scripts/check-fork-sync.sh | 13 ++++++++++ 2 files changed, 43 insertions(+), 14 deletions(-) diff --git a/.github/workflows/fork-sync.yml b/.github/workflows/fork-sync.yml index e23d47b..9180985 100644 --- a/.github/workflows/fork-sync.yml +++ b/.github/workflows/fork-sync.yml @@ -59,28 +59,42 @@ jobs: uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 # Exit 1 from the script means "commits need triage", which is a normal state and - # must not fail the job -- a red X every week trains people to ignore it. The - # report is the product; the exit code only routes it. + # must not fail the job -- a red X every week trains people to ignore it. Exit 2 is + # a real error and MUST fail. The report is the product; the exit code only routes it. + # + # Run ONCE, with --markdown, and use that output for both the log and the issue + # body. An earlier cut called the script twice -- plain for the log, --markdown for + # the body with `|| true` to swallow the expected exit 1 -- which was wrong twice + # over: `|| true` also swallows exit 2, so a broken run could file an issue with an + # empty body and still go green; and two invocations are two sets of API calls that + # can disagree if one hits a transient failure. - name: Diff fork HEADs against the reconciled sync points id: check env: GH_TOKEN: ${{ github.token }} run: | set -o pipefail - if scripts/check-fork-sync.sh > text-report.txt 2>&1; then - echo "drift=false" >> "$GITHUB_OUTPUT" - else - rc=$? - # 2 is a real error (missing branch, malformed conf) -- that SHOULD fail. - if [ "$rc" -ne 1 ]; then + rc=0 + scripts/check-fork-sync.sh --markdown > body.md 2> err.txt || rc=$? + + case "$rc" in + 0) echo "drift=false" >> "$GITHUB_OUTPUT" ;; + 1) echo "drift=true" >> "$GITHUB_OUTPUT" ;; + *) echo "::error title=fork-sync check failed::scripts/check-fork-sync.sh exited $rc" - cat text-report.txt + cat err.txt >&2 exit "$rc" - fi - echo "drift=true" >> "$GITHUB_OUTPUT" + ;; + esac + + # A body we would post must not be empty -- that is the failure mode the single + # invocation is meant to close, so assert it rather than assume it. + if [ ! -s body.md ]; then + echo "::error title=fork-sync produced an empty report::exit was $rc but body.md is empty" + exit 2 fi - cat text-report.txt - scripts/check-fork-sync.sh --markdown > body.md || true + + cat body.md - name: Open or update the backport-queue issue env: @@ -96,8 +110,10 @@ jobs: gh label create "$LABEL" --repo "$REPO" --color 0E8A16 \ --description "Upstream fork commits awaiting a disposition" 2>/dev/null || true + # --limit 1: only the first match is used, and the default page of 30 is API + # work we never look at. existing="$(gh issue list --repo "$REPO" --label "$LABEL" --state open \ - --json number --jq '.[0].number // empty')" + --limit 1 --json number --jq '.[0].number // empty')" if [ "$DRIFT" = "true" ]; then { diff --git a/scripts/check-fork-sync.sh b/scripts/check-fork-sync.sh index ee446cf..a81d702 100755 --- a/scripts/check-fork-sync.sh +++ b/scripts/check-fork-sync.sh @@ -38,6 +38,7 @@ markdown=false err() { printf 'check-fork-sync: %s\n' "$*" >&2; } command -v gh >/dev/null || { err 'gh CLI not found'; exit 2; } +command -v jq >/dev/null || { err 'jq not found (this script parses the compare API with it)'; exit 2; } [[ -f $CONF ]] || { err "no such file: $CONF"; exit 2; } drift=0 @@ -61,6 +62,18 @@ while read -r branch sync; do ahead="$(jq -r '.ahead_by' <<<"$cmp_json")" + # Insist on an integer before comparing. This is not defensive padding: bash + # arithmetic treats a bare word as an unset variable, so BOTH `[[ null -eq 0 ]]` and + # `[[ "" -eq 0 ]]` evaluate TRUE. A response that parsed but had no ahead_by -- an + # auth failure, a rate limit, a partial body -- would therefore report "nothing new" + # and exit 0. A tool whose entire job is to stop commits going unaccounted for must + # not have a path where it silently says all-clear because it could not tell. + [[ $ahead =~ ^[0-9]+$ ]] || { + err "$branch: compare API returned no usable ahead_by (got '$ahead')." + err "Refusing to report 'reconciled' from a response we cannot read." + exit 2 + } + if [[ $ahead -eq 0 ]]; then if $markdown; then report+="- ✅ \`$branch\` — reconciled through \`${sync:0:9}\`, nothing new.