From b5dafabfaa78f08d471eb5ecbf120a3024f108eb Mon Sep 17 00:00:00 2001 From: "Michael C. Ferguson" Date: Thu, 16 Jul 2026 10:12:31 -0500 Subject: [PATCH 1/8] kernel-patches: make the carried series `git am`-able MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 0013-hid-flydigi-vader.patch carried `From: Alexey Melnikov` with no . `git am` cannot build an author identity from that and hard-fails the whole series on it: fatal: empty ident name (for <>) not allowed Buildroot never noticed because it applies patches with `patch -p1`, which reads the diff hunks and ignores the mail headers entirely. The series therefore builds a correct kernel and ships, and the defect only surfaces when the patches are replayed as git history — which is exactly what an export to a Linux-Kernel_MiSTer-style tree does. The correct identity is not a guess. The patch's own mbox line names its provenance, commit b1b168eb6 in MiSTer-devel/Linux-Kernel_MiSTer, whose author is `Sorgelig ` with a matching Date. (The two names are one person — the fork's merge commit for PR #42 is authored `Alexey Melnikov `. Whoever wrote 0013 took the real name and dropped the address.) Using the fork's own ident also makes 0013 consistent with the other 15 Sorgelig-authored patches in the series. Add scripts/lint-kernel-patches.sh so this cannot regress. It validates with `git mailinfo` — the parser `git am` itself uses — rather than a regex that approximates one, so it tests am-ability for real. It needs no kernel tree and no network. Wire it into build.yml before the build step rather than into ci-tests.sh, which runs post-build: the check takes about a second, so in the existing job it costs no additional runner minutes and fails fast instead of after a possible 300-minute build. Verified: with this fix the full series `git am`s cleanly onto v6.18.38, 31/31 on master (0038 lands with the diag branch and already parses). Co-Authored-By: Claude Opus 4.8 --- .github/workflows/build.yml | 12 ++ .../0013-hid-flydigi-vader.patch | 2 +- scripts/lint-kernel-patches.sh | 107 ++++++++++++++++++ 3 files changed, 120 insertions(+), 1 deletion(-) create mode 100755 scripts/lint-kernel-patches.sh diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 5bc75fe..2d46407 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -57,6 +57,18 @@ jobs: - name: Checkout uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + # Deliberately BEFORE the build, and deliberately in this job rather than a + # job of its own: it needs no kernel tree and no network, runs in about a + # second, and so costs no extra runner minutes here while failing fast + # instead of after a possible 300-minute build. + # + # It guards an interface the build itself cannot: Buildroot applies these + # patches with `patch -p1`, which ignores mail headers, so a malformed + # `From:` builds green and only breaks when the series is replayed as git + # history with `git am`. See the script's header. + - name: Lint kernel patch headers (git am-ability) + run: scripts/lint-kernel-patches.sh + # Everything from "prepare the runner" to "linux.img + zImage_dtb exist". - name: Build the image uses: ./.github/actions/buildroot-build diff --git a/board/mister/de10nano/linux-patches/0013-hid-flydigi-vader.patch b/board/mister/de10nano/linux-patches/0013-hid-flydigi-vader.patch index 0996729..5614d95 100644 --- a/board/mister/de10nano/linux-patches/0013-hid-flydigi-vader.patch +++ b/board/mister/de10nano/linux-patches/0013-hid-flydigi-vader.patch @@ -1,5 +1,5 @@ From b1b168eb64d0f34952cbf4ed05af74b5f00f630b Mon Sep 17 00:00:00 2001 -From: Alexey Melnikov +From: Sorgelig Date: Wed, 15 Apr 2026 01:02:42 +0800 Subject: [PATCH 1/1] hid: add Flydigi Vader 4 Pro BT D-Input remap driver diff --git a/scripts/lint-kernel-patches.sh b/scripts/lint-kernel-patches.sh new file mode 100755 index 0000000..0d1c1fb --- /dev/null +++ b/scripts/lint-kernel-patches.sh @@ -0,0 +1,107 @@ +#!/usr/bin/env bash +# +# lint-kernel-patches.sh — assert the carried kernel patches are `git am`-able. +# +# WHY THIS EXISTS +# --------------- +# Buildroot applies these patches with `patch -p1` (support/scripts/apply-patches.sh), +# which reads only the diff hunks and ignores the mail headers entirely. So a patch +# can carry a malformed `From:` line, build a perfectly good kernel, and ship — the +# defect is invisible to every other check in this repo. +# +# It stops being invisible the moment the series is replayed as git history, which is +# what any export to a Linux-Kernel_MiSTer-style tree does (`git am`). `git am` needs +# a parseable author identity to write a commit and hard-fails without one: +# +# fatal: empty ident name (for <>) not allowed +# +# That is a real bug we shipped: 0013-hid-flydigi-vader.patch carried +# `From: Alexey Melnikov` with no , and `git am` of the series died on it. +# +# So the mail headers are an interface — one the primary build does not exercise. +# This script exercises it. +# +# HOW +# --- +# `git mailinfo` IS the parser `git am` uses to split a patch into identity + message +# + diff. Running it is therefore a real test of am-ability rather than a regex that +# approximates one, and it needs no kernel tree and no network — it is fast enough to +# run before the build rather than after it. +# +# A patch that fails to parse yields an empty Author/Email, which is exactly the +# condition that kills `git am`. +# +# Usage: scripts/lint-kernel-patches.sh [patch-dir] +# patch-dir defaults to the series named by BR2_LINUX_KERNEL_PATCH in +# configs/mister_de10nano_defconfig. +# +# Exit: 0 = every patch is am-able; 1 = at least one is not (details on stderr). + +set -o errexit +set -o nounset +set -o pipefail + +readonly REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +readonly PATCH_DIR="${1:-$REPO_ROOT/board/mister/de10nano/linux-patches}" + +if [[ ! -d $PATCH_DIR ]]; then + printf 'lint-kernel-patches: no such patch directory: %s\n' "$PATCH_DIR" >&2 + exit 1 +fi + +# mailinfo writes the split message body and diff out as files; we only care about +# the identity summary it prints, so they go to a scratch dir we discard. +scratch="$(mktemp -d)" +trap 'rm -rf "$scratch"' EXIT + +checked=0 +failed=0 + +for patch in "$PATCH_DIR"/*.patch; do + [[ -e $patch ]] || continue + name="$(basename "$patch")" + checked=$((checked + 1)) + + # `git mailinfo ` reads the mail on stdin and prints Author/Email/ + # Subject/Date. It exits 0 even when it cannot parse an identity, so the empty + # field — not the exit code — is the signal. + if ! info="$(git mailinfo "$scratch/msg" "$scratch/patch" <"$patch" 2>"$scratch/err")"; then + printf 'FAIL %s\n git mailinfo could not parse this patch:\n' "$name" >&2 + sed 's/^/ /' "$scratch/err" >&2 + failed=$((failed + 1)) + continue + fi + + author="$(sed -n 's/^Author: //p' <<<"$info")" + email="$(sed -n 's/^Email: //p' <<<"$info")" + subject="$(sed -n 's/^Subject: //p' <<<"$info")" + + problems=() + [[ -n $author ]] || problems+=('no author name — `From:` needs `Name `') + [[ -n $email ]] || problems+=('no author email — `From:` needs `Name `') + [[ -n $subject ]] || problems+=('no subject — `Subject:` is the commit message') + + if ((${#problems[@]})); then + printf 'FAIL %s\n' "$name" >&2 + printf ' %s\n' "${problems[@]}" >&2 + printf ' got: %s\n' "$(grep -m1 '^From:' "$patch" || echo '(no From: line at all)')" >&2 + failed=$((failed + 1)) + else + printf 'ok %-52s %s <%s>\n' "$name" "$author" "$email" + fi +done + +if ((checked == 0)); then + printf 'lint-kernel-patches: no *.patch files found in %s\n' "$PATCH_DIR" >&2 + exit 1 +fi + +printf '\n' +if ((failed)); then + printf 'RESULT: FAIL — %d of %d patch(es) are not `git am`-able.\n' "$failed" "$checked" >&2 + printf 'These build fine under Buildroot (`patch -p1` ignores mail headers) but\n' >&2 + printf 'cannot be replayed as git history. Fix the `From:` line to `Name `.\n' >&2 + exit 1 +fi + +printf 'RESULT: PASS — all %d patches are `git am`-able.\n' "$checked" From e1e83b142ded7f41a496825bf6d6acbe964e50fa Mon Sep 17 00:00:00 2001 From: "Michael C. Ferguson" Date: Thu, 16 Jul 2026 10:26:59 -0500 Subject: [PATCH 2/8] =?UTF-8?q?scripts:=20add=20export-kernel-tree.sh=20?= =?UTF-8?q?=E2=80=94=20render=20the=20series=20as=20a=20git=20tree?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MiSTer-devel/Linux-Kernel_MiSTer keeps the kernel as a materialized git tree: a squashed tarball commit (`v5.15.1`) with MiSTer commits replayed on top. We keep it as {pinned version + hash} + an ordered patch series. Those are the same model — tarball base plus ordered series — differing only in whether the base is stored as a hash or as 283MB of blobs. (Verified: that repo's `v5.15.1` base is byte-for-byte pristine kernel.org v5.15.1. Its only delta is 11 absent files, all `tools/`/`Documentation/` dotfiles lost to `.gitignore` during a `git add` after extraction — no MiSTer change hides in the base.) So the two formats are convertible, and this renders one into the other: a pristine hash-verified tarball as one base commit, then one commit per carried patch with authorship preserved, then an in-tree defconfig and an EXPORT.md. It is a build output, not a second source of truth. Edits belong in the patch series; this regenerates from it. Design notes, all verified rather than assumed: - Deterministic. `git am --committer-date-is-author-date` plus a base-commit date taken from the extracted Makefile's mtime (kernel.org tarballs come from `git archive`, so every file carries the tag's commit time — stable across machines, unlike download time). Two runs produce identical SHAs, so regenerating after no change is a no-op rather than a force-push of fresh SHAs. This is what makes a 6.18.39 bump cheap: edit the version and the hash, rerun. - Fails closed. The tarball is checked against linux.hash and refuses to proceed without a matching entry — verified by bumping the version with the tarball cached but no hash present. - Ships arch/arm/configs/MiSTer_defconfig, the path the fork already uses, so the tree builds without Buildroot (`make ARCH=arm MiSTer_defconfig && make zImage`) — the thing `make linux` inside Buildroot cannot hand anyone. Verified that `make ARCH=arm MiSTer_defconfig` yields a .config byte-identical to Buildroot's own `cp linux.config .config && make olddefconfig`, so the exported tree cannot build a different kernel than the image. Written in the kernel's minimized form, unlike the fork's 4247-line expanded .config, which bakes CONFIG_CC_VERSION_TEXT — one machine's gcc build string — into a file whose purpose is portability. - Emits an ORPHAN branch. Attaching it to the 5.15 history needs a merge whose tree ignores its first parent, which makes `git log` list ~113 commits whose changes are absent from the tree: a reader sees "xone: update driver" and concludes xone is present, when it is a Buildroot package now. A log listing absent changes is worse than an absent ancestor. Disposition per fork commit lives in MISTER-KERNEL-PATCH-RECON.md, which cites the superseding vanilla commit — something no git command can produce, since across this much context drift `git patch-id` matches nothing. - Never touches a remote. Publishing is an explicit fetch-into-a-fork plus push, spelled out in the generated EXPORT.md. Verified end to end: exports Linux 6.18.38 + 31 patches, 34 commits, clean tree. Co-Authored-By: Claude Opus 4.8 --- scripts/export-kernel-tree.sh | 367 ++++++++++++++++++++++++++++++++++ 1 file changed, 367 insertions(+) create mode 100755 scripts/export-kernel-tree.sh diff --git a/scripts/export-kernel-tree.sh b/scripts/export-kernel-tree.sh new file mode 100755 index 0000000..de24082 --- /dev/null +++ b/scripts/export-kernel-tree.sh @@ -0,0 +1,367 @@ +#!/usr/bin/env bash +# +# export-kernel-tree.sh — render the carried kernel series as a Linux-Kernel_MiSTer-style +# git tree: a pristine upstream tarball as one base commit, then one commit per patch. +# +# WHY +# --- +# This repo keeps the MiSTer kernel as {pinned upstream version + hash} + an ordered +# patch series. MiSTer-devel/Linux-Kernel_MiSTer keeps it as a materialized git tree: +# a squashed tarball commit (`v5.15.1`) with MiSTer commits replayed on top. Those are +# the SAME MODEL — tarball base plus ordered series — differing only in whether the +# base is stored as a hash or as 283MB of blobs. This script renders one into the other. +# +# It exists so the rendered tree is a BUILD OUTPUT, not a second source of truth. Edits +# belong in the patch series; this regenerates from it. Given the same inputs it emits +# byte-identical commits (see DETERMINISM), so re-running after no change is a no-op +# rather than a force-push of fresh SHAs. +# +# WHAT YOU GET +# ------------ +# / a fresh git repo, branch MiSTer-v, containing: +# - one base commit "Linux " — pristine upstream, hash-verified +# - one commit per carried patch, original authorship preserved +# - arch/arm/configs/MiSTer_defconfig — so the tree builds standalone: +# make ARCH=arm MiSTer_defconfig && make ARCH=arm zImage +# which is the thing `make linux` inside Buildroot cannot hand someone. +# - EXPORT.md — states it is generated, names the source of truth, and records +# the fork commit we last reconciled against. +# - tag mister- +# +# The branch is an ORPHAN lineage on purpose. It shares no ancestor with MiSTer-v5.15, +# because a merge whose tree ignores its first parent would make `git log` list ~113 +# commits whose changes are NOT in the tree (someone reads "xone: update driver" and +# concludes xone is in; it is a Buildroot package now). A log that lists absent changes +# is worse than an absent ancestor. What was dropped is recorded in +# MISTER-KERNEL-PATCH-RECON.md, which cites the superseding vanilla commit per fork +# commit — something no git command can produce. +# +# This script NEVER touches a fork or a remote. To publish, fetch the orphan branch +# into a fork and push from there (see EXPORT.md, which spells out the two commands). +# +# DETERMINISM +# ----------- +# Reproducibility comes from two choices: +# - `git am --committer-date-is-author-date`, so committer dates come from the +# patches rather than from the clock; +# - the base commit's date is the extracted Makefile's mtime. kernel.org tarballs are +# produced with `git archive`, so every file carries the tag's commit time — stable +# across machines and meaningful, unlike download time. Override with +# SOURCE_DATE_EPOCH. +# +# Usage: scripts/export-kernel-tree.sh --output DIR [--fork-sync SHA] [--tarball FILE] +# +# --output DIR where to build the tree (must not already exist) +# --fork-sync SHA fork commit this export was reconciled against; recorded in +# EXPORT.md as the backport-queue starting point +# --tarball FILE use this tarball instead of the dl/ cache or a download +# +# Exit: 0 = tree built and verified; non-zero = anything failed (fails closed). + +set -o errexit +set -o nounset +set -o pipefail + +readonly REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +readonly DEFCONFIG="$REPO_ROOT/configs/mister_de10nano_defconfig" +readonly HASH_FILE="$REPO_ROOT/board/mister/de10nano/patches/linux/linux.hash" + +# Committer identity for the generated commits. Patch AUTHORS are preserved by `git am`; +# this only says who mechanically produced the tree, and it must be explicit so the +# script works on a runner with no git config. +readonly EXPORT_NAME="${EXPORT_COMMITTER_NAME:-MiSTer Buildroot export}" +readonly EXPORT_EMAIL="${EXPORT_COMMITTER_EMAIL:-export@mister-devel.invalid}" + +die() { printf 'export-kernel-tree: %s\n' "$*" >&2; exit 1; } +say() { printf '\n=== %s\n' "$*"; } + +output='' +fork_sync='' +tarball_override='' + +while (($#)); do + case "$1" in + --output) output="${2:-}"; shift 2 ;; + --fork-sync) fork_sync="${2:-}"; shift 2 ;; + --tarball) tarball_override="${2:-}"; shift 2 ;; + -h | --help) sed -n '/^# Usage:/,/^# Exit:/p' "${BASH_SOURCE[0]}" | sed 's/^# \?//'; exit 0 ;; + *) die "unknown argument: $1 (try --help)" ;; + esac +done + +[[ -n $output ]] || die 'missing --output DIR (try --help)' +[[ ! -e $output ]] || die "--output already exists: $output" + +# --- 1. Read the pinned inputs out of the defconfig ----------------------------------- +# The defconfig is the single source of truth for what we build; nothing here is +# hardcoded, so a version bump is a one-line defconfig edit and this script follows. + +defconfig_value() { + # Values look like: BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="6.18.38" + sed -n "s/^$1=\"\(.*\)\"$/\1/p" "$DEFCONFIG" | tail -1 +} + +# Buildroot spells the external tree's own path as a make variable inside the defconfig; +# resolve it the way Buildroot would. +resolve_br_path() { + printf '%s' "${1//\$(BR2_EXTERNAL_MISTER_PATH)/$REPO_ROOT}" +} + +[[ -f $DEFCONFIG ]] || die "no defconfig at $DEFCONFIG" + +version="$(defconfig_value BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE)" +[[ -n $version ]] || die 'BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE not set in defconfig' + +patch_dir="$(resolve_br_path "$(defconfig_value BR2_LINUX_KERNEL_PATCH)")" +config_file="$(resolve_br_path "$(defconfig_value BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE)")" +fragments="$(resolve_br_path "$(defconfig_value BR2_LINUX_KERNEL_CONFIG_FRAGMENT_FILES)")" + +[[ -d $patch_dir ]] || die "patch dir not found: $patch_dir" +[[ -f $config_file ]] || die "kernel config not found: $config_file" + +series=("$patch_dir"/*.patch) +((${#series[@]})) || die "no patches in $patch_dir" + +branch="MiSTer-v${version%.*}" # 6.18.38 -> MiSTer-v6.18, matching the fork's convention +tag="mister-${version}" + +say "Exporting Linux $version + ${#series[@]} carried patches -> $output (branch $branch)" + +# --- 2. Get the tarball, and verify it against the signed-manifest hash ---------------- +# Fails closed: an unverified kernel tarball is the whole reason linux.hash exists. + +tarball="$tarball_override" +if [[ -z $tarball ]]; then + cached="$REPO_ROOT/dl/linux/linux-$version.tar.xz" + if [[ -f $cached ]]; then + tarball="$cached" + say "Using cached tarball: $tarball" + else + tarball="$(mktemp -d)/linux-$version.tar.xz" + url="https://cdn.kernel.org/pub/linux/kernel/v${version%%.*}.x/linux-$version.tar.xz" + say "Downloading $url" + curl --fail --location --silent --show-error --output "$tarball" "$url" || + die "download failed: $url" + fi +fi +[[ -f $tarball ]] || die "no such tarball: $tarball" + +expected="$(sed -n "s/^sha256[[:space:]]\+\([0-9a-f]\{64\}\)[[:space:]]\+linux-$version\.tar\.xz$/\1/p" "$HASH_FILE" | tail -1)" +[[ -n $expected ]] || die "no sha256 for linux-$version.tar.xz in $HASH_FILE — bump the hash from kernel.org's signed manifest" + +actual="$(sha256sum "$tarball" | cut -d' ' -f1)" +[[ $actual == "$expected" ]] || die "tarball hash mismatch for linux-$version.tar.xz + expected $expected (from $HASH_FILE) + actual $actual" +say "Tarball verified: sha256 $actual" + +# --- 3. Extract ------------------------------------------------------------------------ + +mkdir -p "$output" +say "Extracting" +tar -xf "$tarball" -C "$output" --strip-components=1 + +# kernel.org tarballs come from `git archive`, so every file's mtime is the tag's commit +# time. That makes this stable across machines, unlike the download time. +if [[ -n ${SOURCE_DATE_EPOCH:-} ]]; then + base_epoch="$SOURCE_DATE_EPOCH" +else + base_epoch="$(stat -c %Y "$output/Makefile")" +fi +base_date="$(date -u -d "@$base_epoch" '+%Y-%m-%dT%H:%M:%S+00:00')" + +# --- 4. Base commit: pristine upstream, on its own --------------------------------------- +# Kept as its own commit so `git diff HEAD` is exactly the MiSTer delta and +# nothing else — the review question worth answering. + +cd "$output" +git init --quiet --initial-branch="$branch" +git config user.name "$EXPORT_NAME" +git config user.email "$EXPORT_EMAIL" +git config commit.gpgsign false + +git add --all +GIT_AUTHOR_DATE="$base_date" GIT_COMMITTER_DATE="$base_date" \ + git commit --quiet --file=- </dev/null 2>&1; then + git am --abort 2>/dev/null || true + die "git am failed. Run scripts/lint-kernel-patches.sh first — a malformed From: +line fails the whole series. If the headers are fine, a patch does not apply to +$version and the series needs rebasing onto it." +fi + +applied="$(git rev-list --count "$base_commit"..HEAD)" +((applied == ${#series[@]})) || + die "expected ${#series[@]} commits, got $applied" +say "Applied $applied/${#series[@]} patches cleanly" + +# --- 6. In-tree defconfig, so the tree is usable without Buildroot ------------------------ +# This is the step that makes the export worth shipping: `git clone && make` works, which +# is what a materialized tree is FOR and what `make linux` inside Buildroot cannot give. +# +# Buildroot consumes BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE by copying it to .config and +# running olddefconfig; `make MiSTer_defconfig` fills in defaults the same way, so the +# minimized file works unchanged as a defconfig. + +say "Generating arch/arm/configs/MiSTer_defconfig" +if [[ -n $fragments ]]; then + # Merge with the kernel's OWN merge_config.sh rather than reimplementing Buildroot's + # merge. -m merges without invoking a compiler; -r keeps later fragments winning. + read -r -a frag_list <<<"$fragments" + KCONFIG_CONFIG=arch/arm/configs/MiSTer_defconfig \ + ./scripts/kconfig/merge_config.sh -m -r -O arch/arm/configs \ + "$config_file" "${frag_list[@]}" >/dev/null 2>&1 || + die 'merge_config.sh failed merging the config fragments' + mv arch/arm/configs/.config arch/arm/configs/MiSTer_defconfig 2>/dev/null || true + config_note="merged from $(basename "$config_file") + $(printf '%s ' "${frag_list[@]##*/}")" +else + cp "$config_file" arch/arm/configs/MiSTer_defconfig + config_note="copied verbatim from $(basename "$config_file")" +fi + +git add arch/arm/configs/MiSTer_defconfig +GIT_AUTHOR_DATE="$base_date" GIT_COMMITTER_DATE="$base_date" \ + git commit --quiet --file=- <EXPORT.md < fetch $branch:$branch + git -C push origin $branch +$(if [[ -n $fork_sync ]]; then cat </dev/null + +# --- 8. Verify what we built, rather than assume it ---------------------------------------- + +say 'Verifying' +[[ -f arch/arm/configs/MiSTer_defconfig ]] || die 'defconfig missing from the tree' +git diff --quiet && git diff --cached --quiet || die 'tree is dirty after export' + +# The base must be untouched upstream: our delta may not reach outside the patches. +touched="$(git diff --name-only "$base_commit" "$tag" | wc -l)" + +printf '\n' +printf 'RESULT: PASS — exported Linux %s + %s patches\n' "$version" "$applied" +printf ' tree %s\n' "$output" +printf ' branch %s\n' "$branch" +printf ' tag %s\n' "$tag" +printf ' commits %s (1 base + %s patches + defconfig + EXPORT.md)\n' \ + "$(git rev-list --count HEAD)" "$applied" +printf ' files touched vs pristine upstream: %s\n' "$touched" +printf '\nPublish with:\n' +printf ' git -C fetch %s %s:%s\n' "$output" "$branch" "$branch" +printf ' git -C push origin %s\n' "$branch" From 1eeaf0f6de82dc19221b181f8be84100a3c2c219 Mon Sep 17 00:00:00 2001 From: "Michael C. Ferguson" Date: Thu, 16 Jul 2026 10:36:16 -0500 Subject: [PATCH 3/8] export-kernel-tree: extend the fork's spine instead of orphaning MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The orphan branch was wrong, and provably so: GitHub's compare API 404s with "No common ancestor", so an orphan branch cannot be PR'd anywhere at all. I had described that as the compare view being "meaningless"; it is stronger than that. The reason it was wrong is that I mis-read the fork's structure. Linux-Kernel_MiSTer is not one chain. Its tarball commits form a spine e12ed6c19 v5.13.12 -> 137491a75 v5.14 -> b6f2ca1c4 v5.14.5 -> aba1ef4c1 v5.15.1 and each MiSTer-vX.Y branch hangs off a spine point with the MiSTer series replayed on top (MiSTer-v5.15 = aba1ef4c1 + 112, MiSTer-v5.14 = 137491a75 + 63). They are siblings, not a line. Every spine commit is a pristine tarball with no MiSTer code. So --parent-repo/--parent parents the base commit on the newest spine point, which is simply what this project already does, four times. It gets all three properties at once, where each earlier design could only get two: - PR-able: merge-base with MiSTer-v5.15 is aba1ef4c1, so GitHub can compare. - Honest log: no MiSTer-5.15 commit appears, because they are siblings rather than ancestors. This was the entire objection the orphan existed to avoid, and parenting on the spine dissolves it — parenting on the branch TIP would have reintroduced it, listing ~112 commits whose changes this tree discards. - Meaningful base: its parent is a pristine tarball too, so the diff across it is the pure upstream 5.15.1 -> 6.18.38 delta. Verified: 80077 files, zero MiSTer-named paths on either side. Also fix `git add` -> `git add --force` for the base commit. The kernel ships .gitignore files matching paths it also tracks, so a plain add after a tarball extract silently drops them. This is not hypothetical — it is exactly why the fork's own v5.15.1 base is NOT byte-identical to kernel.org's v5.15.1 (11 files absent: Documentation/.yamllint, fs/*/.kunitconfig, selftests/bpf/test_progs.c, selftests/arm64/tags/* to the `tags` ctags pattern). Without --force we reproduced it, losing Documentation/.renames.txt. With --force the base tree hash is d13b0d25dbbc19af5884d0b780c74309c5d3fa1e — byte-identical to kernel.org's v6.18.38 tree, and independently checkable against their git. Our base is strictly more faithful than the one it extends. Verified: spine mode is deterministic (identical SHAs across runs), 38 commits (4 spine + base + 31 patches + defconfig + EXPORT.md), merge-base aba1ef4c1. Co-Authored-By: Claude Opus 4.8 --- scripts/export-kernel-tree.sh | 149 +++++++++++++++++++++++++++------- 1 file changed, 119 insertions(+), 30 deletions(-) diff --git a/scripts/export-kernel-tree.sh b/scripts/export-kernel-tree.sh index de24082..1380bfc 100755 --- a/scripts/export-kernel-tree.sh +++ b/scripts/export-kernel-tree.sh @@ -28,13 +28,42 @@ # the fork commit we last reconciled against. # - tag mister- # -# The branch is an ORPHAN lineage on purpose. It shares no ancestor with MiSTer-v5.15, -# because a merge whose tree ignores its first parent would make `git log` list ~113 -# commits whose changes are NOT in the tree (someone reads "xone: update driver" and -# concludes xone is in; it is a Buildroot package now). A log that lists absent changes -# is worse than an absent ancestor. What was dropped is recorded in -# MISTER-KERNEL-PATCH-RECON.md, which cites the superseding vanilla commit per fork -# commit — something no git command can produce. +# WHERE THE BRANCH HANGS (--parent-repo/--parent) +# ----------------------------------------------- +# Linux-Kernel_MiSTer is not one chain. Its tarball commits form a SPINE — +# +# e12ed6c19 v5.13.12 -> 137491a75 v5.14 -> b6f2ca1c4 v5.14.5 -> aba1ef4c1 v5.15.1 +# +# — and each MiSTer-vX.Y branch hangs off a spine point with the MiSTer series replayed +# on top (MiSTer-v5.15 = aba1ef4c1 + 112 commits). Every spine commit is a PRISTINE +# tarball with no MiSTer code in it. +# +# So the right shape for a new kernel is to extend the spine the same way, parenting the +# base commit on the newest spine point (aba1ef4c1) rather than on a branch tip: +# +# aba1ef4c1 v5.15.1 --+-- [112 MiSTer commits] --> MiSTer-v5.15 (theirs, untouched) +# | +# +-- v6.18.38 -- [our commits] -> MiSTer-v6.18 +# +# That buys three things at once: +# - shared ancestry with MiSTer-v5.15, so GitHub can compare and a PR is possible at +# all (across unrelated histories the compare API 404s: "No common ancestor"); +# - a log with NO MiSTer-5.15 commits in it — they are siblings, not ancestors — so +# nothing lists a change that is absent from the tree. Parenting on the branch TIP +# instead would list ~112 commits whose changes this tree discards, and a reader +# would see "xone: update driver" and conclude xone is present when it is a +# Buildroot package now; +# - a base commit whose diff against its parent is PURE upstream 5.15.1 -> 6.18.38, +# with zero MiSTer noise, because both trees are pristine. +# +# Their branch is never touched: it becomes a sibling, exactly as MiSTer-v5.14 already +# is. What each of its commits became — carried, superseded upstream, or dropped — is +# recorded in MISTER-KERNEL-PATCH-RECON.md, which cites the superseding vanilla commit. +# No git command can answer that: across this much context drift `git patch-id` matches +# nothing, so "is this commit in 6.18?" is semantic, not mechanical. +# +# Without --parent-repo the base commit is a root commit and the branch is an orphan — +# fine for a standalone tree, but it cannot be PR'd anywhere. # # This script NEVER touches a fork or a remote. To publish, fetch the orphan branch # into a fork and push from there (see EXPORT.md, which spells out the two commands). @@ -49,9 +78,13 @@ # across machines and meaningful, unlike download time. Override with # SOURCE_DATE_EPOCH. # -# Usage: scripts/export-kernel-tree.sh --output DIR [--fork-sync SHA] [--tarball FILE] +# Usage: scripts/export-kernel-tree.sh --output DIR [--parent-repo R --parent C] +# [--fork-sync SHA] [--tarball FILE] # # --output DIR where to build the tree (must not already exist) +# --parent-repo R clone R and parent the base commit inside it, extending that +# repo's tarball spine instead of starting a fresh root +# --parent C the spine commit to extend (requires --parent-repo) # --fork-sync SHA fork commit this export was reconciled against; recorded in # EXPORT.md as the backport-queue starting point # --tarball FILE use this tarball instead of the dl/ cache or a download @@ -78,10 +111,14 @@ say() { printf '\n=== %s\n' "$*"; } output='' fork_sync='' tarball_override='' +parent_repo='' +parent='' while (($#)); do case "$1" in --output) output="${2:-}"; shift 2 ;; + --parent-repo) parent_repo="${2:-}"; shift 2 ;; + --parent) parent="${2:-}"; shift 2 ;; --fork-sync) fork_sync="${2:-}"; shift 2 ;; --tarball) tarball_override="${2:-}"; shift 2 ;; -h | --help) sed -n '/^# Usage:/,/^# Exit:/p' "${BASH_SOURCE[0]}" | sed 's/^# \?//'; exit 0 ;; @@ -92,6 +129,11 @@ done [[ -n $output ]] || die 'missing --output DIR (try --help)' [[ ! -e $output ]] || die "--output already exists: $output" +# The two go together: a parent is meaningless without the repo it lives in, and cloning +# a repo without saying where to hang the branch would silently fall back to an orphan. +[[ -n $parent_repo && -z $parent ]] && die '--parent-repo requires --parent' +[[ -n $parent && -z $parent_repo ]] && die '--parent requires --parent-repo' + # --- 1. Read the pinned inputs out of the defconfig ----------------------------------- # The defconfig is the single source of truth for what we build; nothing here is # hardcoded, so a version bump is a one-line defconfig edit and this script follows. @@ -157,7 +199,21 @@ say "Tarball verified: sha256 $actual" # --- 3. Extract ------------------------------------------------------------------------ -mkdir -p "$output" +if [[ -n $parent_repo ]]; then + say "Cloning $parent_repo to extend its spine at $parent" + git clone --quiet --no-checkout "$parent_repo" "$output" || die "clone failed: $parent_repo" + git -C "$output" rev-parse --verify --quiet "$parent^{commit}" >/dev/null || + die "--parent $parent is not a commit in $parent_repo" + + # Detach at the spine point, then replace the worktree wholesale with the new + # tarball. `git add --all` stages the deletions and the additions together, so the + # resulting commit's tree is the pristine tarball and its parent is the spine. + git -C "$output" checkout --quiet --detach "$parent" + find "$output" -mindepth 1 -maxdepth 1 ! -name .git -exec rm -rf {} + +else + mkdir -p "$output" +fi + say "Extracting" tar -xf "$tarball" -C "$output" --strip-components=1 @@ -175,30 +231,46 @@ base_date="$(date -u -d "@$base_epoch" '+%Y-%m-%dT%H:%M:%S+00:00')" # nothing else — the review question worth answering. cd "$output" -git init --quiet --initial-branch="$branch" +[[ -n $parent_repo ]] || git init --quiet --initial-branch="$branch" git config user.name "$EXPORT_NAME" git config user.email "$EXPORT_EMAIL" git config commit.gpgsign false -git add --all +# Subject is bare "v6.18.38" to match the spine's existing convention (v5.13.12, v5.14, +# v5.14.5, v5.15.1) — the branch should read as the next entry, not a foreign import. +# +# --force is load-bearing, not defensive. The kernel ships .gitignore files that match +# paths it also tracks, so a plain `git add` after a tarball extract silently drops +# them. That is not hypothetical: it is exactly why this repo's own v5.15.1 base is NOT +# byte-identical to kernel.org's v5.15.1 — 11 files (Documentation/.yamllint, +# fs/*/.kunitconfig, selftests/bpf/test_progs.c, selftests/arm64/tags/* to the `tags` +# ctags pattern, ...) are simply absent from it. Without --force we would reproduce that +# bug here and lose Documentation/.renames.txt from 6.18.38. +git add --all --force GIT_AUTHOR_DATE="$base_date" GIT_COMMITTER_DATE="$base_date" \ git commit --quiet --file=- < 137491a75 v5.14 -> b6f2ca1c4 v5.14.5 -> aba1ef4c1 v5.15.1 + | + +----------------------------+ + | + [112 MiSTer commits] -> MiSTer-v5.15 (untouched) + | + v$version -> [$applied MiSTer commits] -> $branch + +\`MiSTer-v5.15\` is **not modified and not an ancestor** — it is a sibling, exactly as +\`MiSTer-v5.14\` already is. Nothing was lost. + +Two consequences worth knowing: + +- The base commit's parent is itself a pristine tarball commit, so + \`git diff aba1ef4c1 v$version\` is the **pure upstream 5.15.1 → $version delta**, + with no MiSTer code on either side. +- No MiSTer-5.15 commit appears in this branch's log, which is the point: this tree + does not contain most of them, and a log listing changes that are absent from the + tree would be worse than no log at all. + +What each 5.15 commit became — carried, superseded by an upstream commit (with the +vanilla commit cited), or deliberately dropped — is recorded per commit in +\`MISTER-KERNEL-PATCH-RECON.md\` in Buildroot_MiSTer. No git command can answer that: +across this much context drift \`git patch-id\` matches nothing, so "is this commit in +$version?" is a semantic question, not a mechanical one. ## Publishing From 6b90cfd4e1ab14f76cca729804d79e7248a11fe1 Mon Sep 17 00:00:00 2001 From: "Michael C. Ferguson" Date: Thu, 16 Jul 2026 11:22:58 -0500 Subject: [PATCH 4/8] export-kernel-tree: vendor the out-of-tree drivers, build them out-of-tree Without this the exported tree builds a kernel with no Xbox (xone) and no 11ac WiFi, while MiSTer-v5.15 vendors both in-tree -- a silent feature regression for anyone who builds the export expecting what MiSTer ships. Vendors all four enabled kernel-module packages (xone + rtl8812au/8814au/8821au, 1882 files) at the paths the 5.15 branch uses, and emits build-mister-modules.sh to build them. NOT wired into Kconfig, deliberately. The Realtek Makefiles do this, ABOVE their own `ifneq ($(KERNELRELEASE),)` guard: export TopDIR ?= $(shell pwd) $(shell cp $(TopDIR)/autoconf_..._linux.h $(TopDIR)/include/autoconf.h) Parse-time filesystem mutation keyed off `pwd`. In-tree, `pwd` is the kernel root rather than the module dir, so TopDIR points at the wrong tree and the driver's generated autoconf.h silently never appears -- $(shell ...) eats the error. These 2594-line Makefiles assume they are never in-tree, across ~1900 files. Wiring them in would mean inventing hooks no upstream tests, then maintaining patches to upstream Makefiles forever. So they are built through the exact out-of-tree invocation Buildroot already uses -- upstream's own supported path, proven by our image builds -- with each recipe read from its .mk rather than reinvented. A driver bump is a pin change in the .mk plus a re-run; no rewiring. Three bugs found by actually building the result, not by reading it: - SILENT DRIVER OMISSION. The defconfig annotates package lines with trailing comments ("BR2_PACKAGE_RTL8812AU=y # RTL8812AU 11ac -- ..."), so anchoring the match on `=y$` hit only BR2_PACKAGE_XONE. The export vendored xone alone and dropped all three WiFi drivers -- exactly the regression this code exists to prevent, with the hole sitting upstream of the fail-closed MODULE_PATH check. Fixed, plus the identical latent bug in defconfig_value(), plus a post-export assertion that every enabled driver is really in the committed tree. - WRONG BUILD PRECONDITION. `modules_prepare` is not enough: external modules link against Module.symvers, which modpost writes during `make modules`, which needs vmlinux from the zImage build. Without it every kernel symbol reads as undefined ("ERROR: modpost: \"skb_pull\" [8812au.ko] undefined!") -- which blames the driver when nothing is wrong with it. The script now checks and says so. - VERMAGIC MISMATCH. This tree is a git repo ~35 commits past the v6.18.38 base, so setlocalversion correctly appends "+" -> 6.18.38+. Buildroot builds the same source from a tarball with no git, so its equally-patched kernel says plain 6.18.38. That "+" lands in vermagic and modprobe rejects every module on it. Passing LOCALVERSION= (set, empty) suppresses it. Verified: rebuilt 8812au.ko now reports `vermagic=6.18.38 SMP mod_unload ARMv7 p2v8`, byte-identical to Buildroot's -- so the export's modules and the image's are interchangeable. Verified end to end with the real ARM toolchain: all 4 drivers build (8812au.ko, 8814au.ko, 8821au.ko, 9 xone modules), 43 commits, 1932 files, deterministic across runs. Committed file counts match the tarballs exactly. Co-Authored-By: Claude Opus 4.8 --- scripts/export-kernel-tree.sh | 334 +++++++++++++++++++++++++++++++++- 1 file changed, 332 insertions(+), 2 deletions(-) diff --git a/scripts/export-kernel-tree.sh b/scripts/export-kernel-tree.sh index 1380bfc..b6702ba 100755 --- a/scripts/export-kernel-tree.sh +++ b/scripts/export-kernel-tree.sh @@ -140,7 +140,11 @@ done defconfig_value() { # Values look like: BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="6.18.38" - sed -n "s/^$1=\"\(.*\)\"$/\1/p" "$DEFCONFIG" | tail -1 + # + # Do NOT anchor on the closing quote. This defconfig carries trailing comments on + # some lines, and anchoring silently yields an empty value rather than failing -- + # which for an optional setting (a config fragment) would mean quietly dropping it. + sed -n "s/^$1=\"\([^\"]*\)\".*$/\1/p" "$DEFCONFIG" | tail -1 } # Buildroot spells the external tree's own path as a make variable inside the defconfig; @@ -338,6 +342,262 @@ are actually a decision under ~4000 that are not. Generated by scripts/export-kernel-tree.sh in Buildroot_MiSTer. EOF +# --- 6b. Vendor the out-of-tree kernel modules ------------------------------------------- +# Without this the exported tree builds a kernel with no Xbox (xone) and no 11ac WiFi, +# while the 5.15 fork has both vendored in-tree — a silent feature regression for anyone +# who builds this tree expecting what MiSTer ships. +# +# WHY THE SOURCES ARE VENDORED BUT NOT WIRED INTO Kconfig +# ------------------------------------------------------- +# The obvious thing is in-tree integration (Kconfig symbol + `obj-$(CONFIG_X) += dir/`), +# the way the fork does it. It is not safe for these packages, and the reason is in the +# Realtek Makefiles, above their own `ifneq ($(KERNELRELEASE),)` guard: +# +# export TopDIR ?= $(shell pwd) +# $(shell cp $(TopDIR)/autoconf_..._linux.h $(TopDIR)/include/autoconf.h) +# +# That is parse-time filesystem mutation keyed off `pwd`. In an in-tree build `pwd` is the +# KERNEL ROOT, not the module directory, so TopDIR points at the wrong tree and the +# driver's generated autoconf.h silently never appears -- `$(shell ...)` swallows the +# error. These 2594-line Makefiles are built on the assumption that they are never +# in-tree, across ~1900 files of driver. Wiring them in-tree would mean inventing hooks no +# upstream tests, then patching upstream Makefiles we would have to maintain forever. +# +# So the sources go in at the paths the fork uses (the tree LOOKS like the fork's), and +# they are built through the exact out-of-tree invocation Buildroot already uses -- which +# is upstream's own supported path, and is proven daily by our own image builds. The +# recipe is read from each package's .mk rather than reinvented here, which is also what +# keeps a package bump cheap: change the pin in the .mk, re-run, done. +# +# xone is a partial exception -- 41 files, a clean Kbuild -- but its `obj-m :=` is declared +# UNCONDITIONALLY, never gated on CONFIG_XONE, so an in-tree Kconfig symbol for it would be +# decorative: present, and doing nothing. Two mechanisms in one tree is also harder to +# explain than one. It goes through the same path as the rest. + +# Package -> in-tree path. The only hand-maintained mapping here, kept declarative on +# purpose. Every kernel-module package gets an entry, not just the currently-enabled ones, +# so flipping one on in the defconfig needs no edit here. +declare -A MODULE_PATH=( + [xone]='drivers/hid/xone' + [rtl8812au]='drivers/net/wireless/realtek/rtl8812au' + [rtl8814au-morrownr]='drivers/net/wireless/realtek/rtl8814au' + [rtl8821au-morrownr]='drivers/net/wireless/realtek/rtl8821au' + [rtl8821cu-morrownr]='drivers/net/wireless/realtek/rtl8821cu' + [rtl88x2bu]='drivers/net/wireless/realtek/rtl88x2bu' + [rtl8188eu-aircrack-ng]='drivers/net/wireless/realtek/rtl8188eu' + [rtl8188fu]='drivers/net/wireless/realtek/rtl8188fu' +) + +# A package is a kernel module iff its .mk evals Buildroot's kernel-module infra. Detected +# rather than listed, so a new one cannot be missed by forgetting to update a list here. +# +# The `=y` is NOT anchored to end-of-line: this defconfig annotates most package lines +# with a trailing comment ("BR2_PACKAGE_RTL8812AU=y # RTL8812AU 11ac -- ..."), and +# anchoring matched only the one line without one, silently vendoring xone alone and +# dropping all three WiFi drivers. +mapfile -t enabled_kmods < <( + sed -n 's/^\(BR2_PACKAGE_[A-Z0-9_]*\)=y\([[:space:]].*\)\?$/\1/p' "$DEFCONFIG" | + while read -r sym; do + dir="$(tr 'A-Z_' 'a-z-' <<<"${sym#BR2_PACKAGE_}")" + mk="$REPO_ROOT/package/$dir/$dir.mk" + [[ -f $mk ]] || continue + grep -qF '$(eval $(kernel-module))' "$mk" || continue + printf '%s\n' "$dir" + done +) + +((${#enabled_kmods[@]})) || die 'detected zero kernel-module packages in the defconfig. +That is almost certainly a parsing bug in this script rather than the truth — the image +ships xone and the Realtek WiFi drivers. Refusing to export a tree missing them.' + +say "Vendoring ${#enabled_kmods[@]} out-of-tree kernel modules: ${enabled_kmods[*]}" +module_build_lines=() +module_doc_rows=() + +for pkg in "${enabled_kmods[@]}"; do + upper="$(tr 'a-z-' 'A-Z_' <<<"$pkg")" + mk="$REPO_ROOT/package/$pkg/$pkg.mk" + dest="${MODULE_PATH[$pkg]:-}" + + # Fail closed. Silently skipping an enabled driver is exactly the regression this + # whole section exists to prevent. + [[ -n $dest ]] || die "no in-tree path mapped for kernel-module package '$pkg'. +Add it to MODULE_PATH in $(basename "${BASH_SOURCE[0]}") — refusing to export a tree +that silently omits a driver the image ships." + + pkg_version="$(sed -n "s/^${upper}_VERSION = //p" "$mk" | tail -1)" + [[ -n $pkg_version ]] || die "no ${upper}_VERSION in $mk" + pkg_opts="$(sed -n "s/^${upper}_MODULE_MAKE_OPTS = //p" "$mk" | tail -1)" + + pkg_tar="$REPO_ROOT/dl/$pkg/$pkg-$pkg_version.tar.gz" + [[ -f $pkg_tar ]] || die "missing source tarball: $pkg_tar +Populate Buildroot's download cache first: make $pkg-source" + + # Same fail-closed rule as the kernel: the hash file is authority, no hash no export. + pkg_expected="$(sed -n "s|^sha256[[:space:]]\+\([0-9a-f]\{64\}\)[[:space:]]\+$pkg-$pkg_version\.tar\.gz$|\1|p" \ + "$REPO_ROOT/package/$pkg/$pkg.hash" | tail -1)" + [[ -n $pkg_expected ]] || die "no sha256 for $pkg-$pkg_version.tar.gz in package/$pkg/$pkg.hash" + pkg_actual="$(sha256sum "$pkg_tar" | cut -d' ' -f1)" + [[ $pkg_actual == "$pkg_expected" ]] || + die "hash mismatch for $pkg-$pkg_version.tar.gz + expected $pkg_expected + actual $pkg_actual" + + mkdir -p "$dest" + tar -xzf "$pkg_tar" -C "$dest" --strip-components=1 + + # --force again: these trees ship their own .gitignore files (build artifacts, + # *.mod.c, Module.symvers). Without it we would drop tracked sources that happen to + # match, the same way the fork's own v5.15.1 base lost 11 files. + git add --force "$dest" + GIT_AUTHOR_DATE="$base_date" GIT_COMMITTER_DATE="$base_date" \ + git commit --quiet --file=- <build-mister-modules.sh <<'MODEOF' +#!/usr/bin/env bash +# +# build-mister-modules.sh — build the out-of-tree drivers this tree vendors. +# +# The kernel builds with: +# make ARCH=arm MiSTer_defconfig +# make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage +# +# These drivers do NOT build from that, by design. They are vendored at the paths the +# 5.15 branch uses, but they are not wired into Kconfig, because their own Makefiles do +# parse-time work keyed off `$(shell pwd)`: +# +# export TopDIR ?= $(shell pwd) +# $(shell cp $(TopDIR)/autoconf_..._linux.h $(TopDIR)/include/autoconf.h) +# +# In an in-tree build `pwd` is the kernel root rather than the module directory, so that +# copy silently lands in the wrong place and the driver's generated autoconf.h never +# appears. These Makefiles assume they are always built out-of-tree. So that is how this +# builds them — which is upstream's own supported path, not a workaround. +# +# Usage: ./build-mister-modules.sh [ARCH] [CROSS_COMPILE] +# defaults: arm, arm-linux-gnueabihf- +# +# REQUIRES A FULLY BUILT KERNEL FIRST -- not just `modules_prepare`: +# +# make ARCH=arm MiSTer_defconfig +# make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- LOCALVERSION= zImage modules +# ./build-mister-modules.sh +# +# PASS `LOCALVERSION=` TO THE KERNEL BUILD, exactly as above. Empty, but SET. +# +# This tree is a git repo whose HEAD sits ~35 commits past the v base commit, so +# scripts/setlocalversion correctly concludes the source is modified and appends a "+", +# giving `6.18.38+`. Buildroot builds the same source from a tarball with no git around +# it, so its identical-but-also-patched kernel reports plain `6.18.38`. The two disagree +# only because one build can see its own history and the other cannot. +# +# That single "+" lands in vermagic, and vermagic is what the kernel matches on when +# loading a module: +# +# vermagic=6.18.38+ SMP mod_unload ARMv7 p2v8 <- built here, without LOCALVERSION= +# vermagic=6.18.38 SMP mod_unload ARMv7 p2v8 <- Buildroot, and this tree WITH it +# +# Mismatch that and modprobe rejects every module ("version magic ... should be ..."), +# which reads like a broken driver and is not. Setting LOCALVERSION= (even to empty) +# makes setlocalversion skip the "+" entirely, so this tree's kernel and modules are +# interchangeable with the shipped image's. +# +# `modules_prepare` is NOT enough, and the way it fails is worth knowing because the +# error blames the driver rather than the real cause. An external module is linked +# against the kernel's symbol table in Module.symvers, and that file is produced by +# modpost during `make modules`, which in turn needs vmlinux from the `zImage` build. +# Without it every kernel symbol the driver uses reads as undefined: +# +# ERROR: modpost: "skb_pull" [8812au.ko] undefined! +# +# Nothing is wrong with the driver there -- the kernel symbol table simply is not built +# yet. Hence the check below, which says so directly. + +set -o errexit +set -o nounset +set -o pipefail + +readonly KDIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +readonly ARCH="${1:-arm}" +readonly CROSS_COMPILE="${2:-arm-linux-gnueabihf-}" + +[[ -f $KDIR/.config ]] || { + printf 'No .config — run `make ARCH=%s MiSTer_defconfig` first.\n' "$ARCH" >&2 + exit 1 +} + +[[ -f $KDIR/Module.symvers ]] || { + cat >&2 <>build-mister-modules.sh +cat >>build-mister-modules.sh <<'MODEOF' + +printf '\nBuilt .ko files:\n' +find . -name '*.ko' -newer .config -printf ' %p\n' 2>/dev/null | sort +MODEOF +chmod +x build-mister-modules.sh + +git add --force build-mister-modules.sh +GIT_AUTHOR_DATE="$base_date" GIT_COMMITTER_DATE="$base_date" \ + git commit --quiet --file=- </.mk\` in Buildroot_MiSTer. Bumping a driver is a +pin change there plus a re-run of the export; nothing here needs rewiring. + +Unlike \`MiSTer-v5.15\`, which vendors these in-tree, that means a driver bump does not +touch this tree's history by hand — and the Realtek drivers here track upstreams that +build against 6.18 with **zero** compatibility patches. ## Where this branch hangs @@ -440,6 +756,20 @@ say 'Verifying' [[ -f arch/arm/configs/MiSTer_defconfig ]] || die 'defconfig missing from the tree' git diff --quiet && git diff --cached --quiet || die 'tree is dirty after export' +# Assert every enabled driver is actually IN the committed tree, rather than trusting that +# the loop above ran. A silent skip here ships a kernel missing WiFi, which is precisely +# the bug this section exists to prevent -- and which a too-strict defconfig parse already +# caused once, vendoring xone alone. +for pkg in "${enabled_kmods[@]}"; do + dest="${MODULE_PATH[$pkg]}" + git cat-file -e "$tag:$dest" 2>/dev/null || + die "$pkg is enabled but $dest is not in the exported tree" + n="$(git ls-tree -r --name-only "$tag" "$dest" | wc -l)" + ((n > 0)) || die "$pkg vendored at $dest but the directory is empty" + printf ' %-52s %s files\n' "$dest" "$n" +done +[[ -x build-mister-modules.sh ]] || die 'build-mister-modules.sh missing or not executable' + # The base must be untouched upstream: our delta may not reach outside the patches. touched="$(git diff --name-only "$base_commit" "$tag" | wc -l)" From 0944371245734ae687d73a1fd89af7eb6a9bcb7b Mon Sep 17 00:00:00 2001 From: "Michael C. Ferguson" Date: Thu, 16 Jul 2026 11:46:32 -0500 Subject: [PATCH 5/8] export-kernel-tree: add --onto, to replay on an upstream-published base Upstream created MiSTer-v6.18 (d9ac12a691) as its own vanilla 6.18.38 commit and asked for the changes as a PR against it. Notably it extends the spine at aba1ef4c1 (v5.15.1) -- independently the same structure --parent produces, which is a good sign the shape was right. But `--parent aba1ef4c1` builds OUR OWN base commit from the tarball, so our branch and theirs each carry a distinct "v6.18.38" commit over the same spine point. A PR would be diverged (ahead 39, behind 5) and merging would fuse two different bases. --onto replays straight onto an existing base instead: no base commit, no tarball needed for the kernel, and the result fast-forwards. Verified: 31/31 apply to d9ac12a691, 38 commits on top, their commit is an ancestor, so the PR is exactly our delta with nothing of theirs restated. Two guards, both from things that actually went wrong here: - The base's version is read back from its own Makefile and must equal the defconfig pin. Replaying a 6.18 series onto a 5.15 base otherwise fails deep in `git am` with conflicts that look like bad patches rather than a bad base. Verified: --onto aba1ef4c1 is refused with "is Linux 5.15.1, but this repo pins 6.18.38". - --onto is resolved to a SHA in the SOURCE repo before cloning. Ref names are ambiguous across a clone boundary, and this bit for real: `git clone` copies the source's LOCAL branches to origin/*, so `--onto origin/MiSTer-v6.18` resolved inside the clone to the source's own local MiSTer-v6.18 -- our tree, not theirs -- and replayed the series onto a tree that already had it. The version check could not catch it: both were 6.18.38. `git am` did, with "MiSTer_fb.c: already exists in index". Their base is missing Documentation/.renames.txt versus kernel.org's v6.18.38 (tree 4efcf6f42 vs d13b0d25d) -- the same .gitignore-eats-a-tracked-file bug that cost their v5.15.1 base 11 files, and that `git add --force` was added here to avoid. Cosmetic, no build impact, not ours to fix in a PR; raised upstream instead. Co-Authored-By: Claude Opus 4.8 --- scripts/export-kernel-tree.sh | 98 +++++++++++++++++++++++++++++------ 1 file changed, 81 insertions(+), 17 deletions(-) diff --git a/scripts/export-kernel-tree.sh b/scripts/export-kernel-tree.sh index b6702ba..451b43a 100755 --- a/scripts/export-kernel-tree.sh +++ b/scripts/export-kernel-tree.sh @@ -79,12 +79,17 @@ # SOURCE_DATE_EPOCH. # # Usage: scripts/export-kernel-tree.sh --output DIR [--parent-repo R --parent C] -# [--fork-sync SHA] [--tarball FILE] +# [--onto COMMIT] [--fork-sync SHA] [--tarball FILE] # # --output DIR where to build the tree (must not already exist) -# --parent-repo R clone R and parent the base commit inside it, extending that -# repo's tarball spine instead of starting a fresh root -# --parent C the spine commit to extend (requires --parent-repo) +# --parent-repo R clone R and work inside it, rather than starting a fresh root +# --parent C spine commit to extend; a base commit is created on top of it +# from the pinned tarball (requires --parent-repo) +# --onto COMMIT replay onto COMMIT, which must ALREADY BE the pinned kernel +# version -- no base commit is created and the tarball is not +# used for the kernel. Use when upstream has published its own +# vanilla base to PR against; the result fast-forwards onto it. +# Mutually exclusive with --parent (requires --parent-repo). # --fork-sync SHA fork commit this export was reconciled against; recorded in # EXPORT.md as the backport-queue starting point # --tarball FILE use this tarball instead of the dl/ cache or a download @@ -113,12 +118,14 @@ fork_sync='' tarball_override='' parent_repo='' parent='' +onto='' while (($#)); do case "$1" in --output) output="${2:-}"; shift 2 ;; --parent-repo) parent_repo="${2:-}"; shift 2 ;; --parent) parent="${2:-}"; shift 2 ;; + --onto) onto="${2:-}"; shift 2 ;; --fork-sync) fork_sync="${2:-}"; shift 2 ;; --tarball) tarball_override="${2:-}"; shift 2 ;; -h | --help) sed -n '/^# Usage:/,/^# Exit:/p' "${BASH_SOURCE[0]}" | sed 's/^# \?//'; exit 0 ;; @@ -129,10 +136,14 @@ done [[ -n $output ]] || die 'missing --output DIR (try --help)' [[ ! -e $output ]] || die "--output already exists: $output" -# The two go together: a parent is meaningless without the repo it lives in, and cloning -# a repo without saying where to hang the branch would silently fall back to an orphan. -[[ -n $parent_repo && -z $parent ]] && die '--parent-repo requires --parent' +# A parent is meaningless without the repo it lives in, and cloning a repo without saying +# where to hang the branch would silently fall back to an orphan. +[[ -n $parent_repo && -z $parent && -z $onto ]] && die '--parent-repo requires --parent or --onto' [[ -n $parent && -z $parent_repo ]] && die '--parent requires --parent-repo' +[[ -n $onto && -z $parent_repo ]] && die '--onto requires --parent-repo' +[[ -n $parent && -n $onto ]] && die '--parent and --onto are mutually exclusive: +--parent extends a spine and CREATES a base commit from the tarball; --onto replays onto +a base that already exists. Pick one.' # --- 1. Read the pinned inputs out of the defconfig ----------------------------------- # The defconfig is the single source of truth for what we build; nothing here is @@ -177,7 +188,13 @@ say "Exporting Linux $version + ${#series[@]} carried patches -> $output (branch # Fails closed: an unverified kernel tarball is the whole reason linux.hash exists. tarball="$tarball_override" -if [[ -z $tarball ]]; then +if [[ -n $onto ]]; then + # --onto: the base already exists upstream, so the kernel tarball is not needed and + # no base commit is created. The safety property still has to hold, though -- replaying + # a 6.18 series onto, say, a 5.15 base must not be attempted -- so the version is read + # back out of the target commit's own Makefile below rather than trusted. + say "Replaying onto existing base $onto (no base commit created)" +elif [[ -z $tarball ]]; then cached="$REPO_ROOT/dl/linux/linux-$version.tar.xz" if [[ -f $cached ]]; then tarball="$cached" @@ -190,20 +207,52 @@ if [[ -z $tarball ]]; then die "download failed: $url" fi fi -[[ -f $tarball ]] || die "no such tarball: $tarball" -expected="$(sed -n "s/^sha256[[:space:]]\+\([0-9a-f]\{64\}\)[[:space:]]\+linux-$version\.tar\.xz$/\1/p" "$HASH_FILE" | tail -1)" -[[ -n $expected ]] || die "no sha256 for linux-$version.tar.xz in $HASH_FILE — bump the hash from kernel.org's signed manifest" +expected='' +if [[ -z $onto ]]; then + [[ -f $tarball ]] || die "no such tarball: $tarball" -actual="$(sha256sum "$tarball" | cut -d' ' -f1)" -[[ $actual == "$expected" ]] || die "tarball hash mismatch for linux-$version.tar.xz + expected="$(sed -n "s/^sha256[[:space:]]\+\([0-9a-f]\{64\}\)[[:space:]]\+linux-$version\.tar\.xz$/\1/p" "$HASH_FILE" | tail -1)" + [[ -n $expected ]] || die "no sha256 for linux-$version.tar.xz in $HASH_FILE — bump the hash from kernel.org's signed manifest" + + actual="$(sha256sum "$tarball" | cut -d' ' -f1)" + [[ $actual == "$expected" ]] || die "tarball hash mismatch for linux-$version.tar.xz expected $expected (from $HASH_FILE) actual $actual" -say "Tarball verified: sha256 $actual" + say "Tarball verified: sha256 $actual" +fi # --- 3. Extract ------------------------------------------------------------------------ -if [[ -n $parent_repo ]]; then +if [[ -n $onto ]]; then + # Resolve the ref in the SOURCE repo and carry the SHA into the clone. Ref names are + # ambiguous across a clone boundary and it is not a theoretical problem: `git clone` + # copies the source's LOCAL branches to origin/*, so `--onto origin/MiSTer-v6.18` + # resolves inside the clone to the source's own local MiSTer-v6.18 -- a different + # commit from the origin/MiSTer-v6.18 the caller meant. That silently replayed a + # series onto a tree that already had it applied, and the version check could not + # catch it because both trees were the same Linux version. + onto="$(git -C "$parent_repo" rev-parse --verify --quiet "$onto^{commit}")" || + die "--onto is not a commit in $parent_repo" + say "Resolved --onto to $onto in $parent_repo" + + say "Cloning $parent_repo" + git clone --quiet --no-checkout "$parent_repo" "$output" || die "clone failed: $parent_repo" + git -C "$output" rev-parse --verify --quiet "$onto^{commit}" >/dev/null || + die "$onto is not reachable in the clone of $parent_repo" + git -C "$output" checkout --quiet --detach "$onto" + + # The base is someone else's, so verify it is the version we are about to patch + # rather than assuming. Read it from the target's own Makefile: replaying a 6.18 + # series onto a 5.15 base would otherwise fail deep in `git am` with conflicts that + # look like bad patches instead of a bad base. + onto_version="$(sed -nE 's/^VERSION = //p;s/^PATCHLEVEL = /./p;s/^SUBLEVEL = /./p' \ + "$output/Makefile" | head -3 | tr -d '\n')" + [[ $onto_version == "$version" ]] || die "--onto $onto is Linux $onto_version, but this +repo pins $version (BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE). Refusing to replay a $version +patch series onto a $onto_version base." + say "Base verified: $onto is Linux $onto_version" +elif [[ -n $parent_repo ]]; then say "Cloning $parent_repo to extend its spine at $parent" git clone --quiet --no-checkout "$parent_repo" "$output" || die "clone failed: $parent_repo" git -C "$output" rev-parse --verify --quiet "$parent^{commit}" >/dev/null || @@ -218,13 +267,19 @@ else mkdir -p "$output" fi -say "Extracting" -tar -xf "$tarball" -C "$output" --strip-components=1 +if [[ -z $onto ]]; then + say "Extracting" + tar -xf "$tarball" -C "$output" --strip-components=1 +fi # kernel.org tarballs come from `git archive`, so every file's mtime is the tag's commit # time. That makes this stable across machines, unlike the download time. if [[ -n ${SOURCE_DATE_EPOCH:-} ]]; then base_epoch="$SOURCE_DATE_EPOCH" +elif [[ -n $onto ]]; then + # No tarball here, so take the base's own commit date. Still deterministic: it is a + # property of the commit we were pointed at, not of when this script ran. + base_epoch="$(git -C "$output" log --format='%ct' -1 "$onto")" else base_epoch="$(stat -c %Y "$output/Makefile")" fi @@ -240,6 +295,14 @@ git config user.name "$EXPORT_NAME" git config user.email "$EXPORT_EMAIL" git config commit.gpgsign false +if [[ -n $onto ]]; then + # The base commit is upstream's; ours would be a duplicate. Branch and go straight to + # the series, so the result fast-forwards onto their branch and the PR is exactly our + # delta -- nothing of theirs restated. + base_commit="$(git rev-parse HEAD)" + git checkout --quiet -b "$branch" +else + # Subject is bare "v6.18.38" to match the spine's existing convention (v5.13.12, v5.14, # v5.14.5, v5.15.1) — the branch should read as the next entry, not a foreign import. # @@ -275,6 +338,7 @@ tree directly; see EXPORT.md. EOF base_commit="$(git rev-parse HEAD)" [[ -n $parent_repo ]] && git checkout --quiet -b "$branch" +fi # --- 5. Replay the carried series ------------------------------------------------------- # --committer-date-is-author-date keeps this reproducible: dates come from the patches, From 0322fdebb813ebcd78b4841963fbb96842a3f524 Mon Sep 17 00:00:00 2001 From: "Michael C. Ferguson" Date: Thu, 16 Jul 2026 12:13:38 -0500 Subject: [PATCH 6/8] export-kernel-tree: fix empty-series glob and the download temp leak Two real findings from the PR #26 review, both reproduced before fixing: - `series=("$patch_dir"/*.patch)` without nullglob leaves the literal pattern in the array when nothing matches, so `((${#series[@]}))` sees length 1 and the "no patches" guard never fires. `git am` then fails on a path that does not exist, blaming a patch rather than the empty directory. Verified: an empty dir gave length 1 with element "/*.patch". Now scoped `shopt -s nullglob` around the glob; an empty patch dir dies with "no patches in ". - The download path used `tarball="$(mktemp -d)/..."` with no cleanup, stranding a ~148MB kernel tarball per run, and bare `mktemp -d` is a GNU extension that errors on BSD/macOS. Now an explicit template plus an EXIT trap. --output is deliberately not cleaned: it is the deliverable and must survive a failure to be diagnosable. The review's third finding -- that `git add --all` respects .gitignore and needs -f -- was already fixed in this branch (the reviewer saw an earlier revision). That is the same bug that costs the upstream v5.15.1 base 11 files and its v6.18.38 base Documentation/.renames.txt; `git add --all --force` is why our tree hash matches kernel.org's exactly. The first cut of the trap made it worse, which is why this was tested rather than assumed: as an EXIT trap, cleanup()'s own return status becomes the script's exit status, so a bare `[[ -n $download_dir ]] && rm -rf ...` returned 1 on every run that used the dl/ cache -- the export printed PASS and exited 1, which would fail CI on success. Rewritten as an `if`. Verified after the change: export still exits 0, output SHA unchanged (9b485e0988), 31/31 patches apply. Co-Authored-By: Claude Opus 4.8 --- scripts/export-kernel-tree.sh | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/scripts/export-kernel-tree.sh b/scripts/export-kernel-tree.sh index 451b43a..8b5bb1e 100755 --- a/scripts/export-kernel-tree.sh +++ b/scripts/export-kernel-tree.sh @@ -113,6 +113,23 @@ readonly EXPORT_EMAIL="${EXPORT_COMMITTER_EMAIL:-export@mister-devel.invalid}" die() { printf 'export-kernel-tree: %s\n' "$*" >&2; exit 1; } say() { printf '\n=== %s\n' "$*"; } +# Only set when we download rather than use the dl/ cache. Cleaned on exit: it holds a +# ~150MB kernel tarball, so leaking it on every run is not a rounding error. --output is +# deliberately NOT touched here -- it is the deliverable, and it must survive a failure +# for the failure to be diagnosable. +# +# cleanup() uses `if` rather than `[[ ... ]] && rm`: as an EXIT trap, the function's own +# return status becomes the script's exit status, and a bare `[[ -n $download_dir ]]` +# returns 1 whenever nothing was downloaded -- making every successful cache-hit run exit +# 1 despite printing PASS. +download_dir='' +cleanup() { + if [[ -n $download_dir ]]; then + rm -rf "$download_dir" + fi +} +trap cleanup EXIT + output='' fork_sync='' tarball_override='' @@ -176,7 +193,12 @@ fragments="$(resolve_br_path "$(defconfig_value BR2_LINUX_KERNEL_CONFIG_FRAGMENT [[ -d $patch_dir ]] || die "patch dir not found: $patch_dir" [[ -f $config_file ]] || die "kernel config not found: $config_file" +# nullglob, or an empty patch dir yields an array holding the literal "*.patch" pattern +# -- length 1, so the guard below passes -- and `git am` then fails on a path that does +# not exist, blaming the patch rather than the empty directory. +shopt -s nullglob series=("$patch_dir"/*.patch) +shopt -u nullglob ((${#series[@]})) || die "no patches in $patch_dir" branch="MiSTer-v${version%.*}" # 6.18.38 -> MiSTer-v6.18, matching the fork's convention @@ -200,7 +222,10 @@ elif [[ -z $tarball ]]; then tarball="$cached" say "Using cached tarball: $tarball" else - tarball="$(mktemp -d)/linux-$version.tar.xz" + # Explicit template: bare `mktemp -d` is a GNU extension and errors on BSD/macOS. + download_dir="$(mktemp -d -t export-kernel-tree.XXXXXXXX)" || + die 'could not create a temporary download directory' + tarball="$download_dir/linux-$version.tar.xz" url="https://cdn.kernel.org/pub/linux/kernel/v${version%%.*}.x/linux-$version.tar.xz" say "Downloading $url" curl --fail --location --silent --show-error --output "$tarball" "$url" || From 7b16dd06814172cb25b364011d226e5835b85dce Mon Sep 17 00:00:00 2001 From: "Michael C. Ferguson" Date: Thu, 16 Jul 2026 12:34:56 -0500 Subject: [PATCH 7/8] scripts: use explicit mktemp templates, matching the rest of scripts/ Review finding on #25, valid and checked: bare `mktemp -d` is a GNU extension -- BSD/macOS mktemp requires a template and errors without one -- and this repo already established the pattern elsewhere: scripts/check-linux-img.sh:140 mktemp -d "${TMPDIR:-/tmp}/check-linux-img.XXXXXX" scripts/ci-tests.sh:156-157 mktemp "${TMPDIR:-/tmp}/ci-tests-*.XXXXXX" lint-kernel-patches.sh was the reported site. export-kernel-tree.sh was the same bug: the fix there used `mktemp -d -t