From be6456c04791b37fb4b4129e1b9cd75469aa98f1 Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Tue, 28 Jul 2026 00:46:20 +0900 Subject: [PATCH] fix: harden tap bump against layout changes and bad downloads This workflow rewrote each checksum by line position: gsed -i "/ARTIFACT\"/!b;n;c\ sha256 \"${sha}\"" It matched the url line, advanced one line, and replaced whatever it landed on. Correct only while sha256 sits directly beneath url inside an on_macos or on_linux block, which is how the three bssh formulae happen to be written today. Nine substitutions depend on that. lablup/mlxcel ran the same construct, copied from the same source. When its formula moved the url to the top level with a `version` stanza in between, the next bump overwrote `version` with the new checksum and left the previous release's sha256 below it. The formula ended up with no version and two sha256 lines, the second of which wins, so `brew install mlxcel` failed on checksum. Nothing validated the result and it sat broken in the tap for three days (lablup/homebrew-tap 9969ec4). bssh is not broken today, but it is one formula edit away from the same outcome, with three times the exposure. Stanzas are now located by name. `set_artifact` finds the url stanza whose value contains a given artifact, rewrites it, then rewrites the first sha256 stanza below it, preserving indentation. Anything in between is skipped rather than clobbered. Guards abort the run when a formula does not hold exactly one matching url, or when no sha256 follows it, instead of editing the wrong line. Two other faults surfaced while fixing this: Downloads used `curl -Ls` with no `-f`, so a missing or renamed asset produced a zero-byte file and its checksum went to the tap unnoticed. They now use `-f --retry 3`, reject empty files, and verify each archive is readable with `unzip -t` or `tar -tzf`, so a truncated download cannot become a valid-looking sha256 either. `git commit` ran unconditionally, so re-running for a version already in the tap failed on an empty index. It now exits cleanly. A validation step runs `ruby -c` and `brew style` on each formula, checks the version stanza, and requires the exact set of checksums downloaded this run, which catches a substitution that silently no-ops. `ruby -c` alone is not enough: the corrupted mlxcel formula was valid Ruby. Verified against the current tap formulae. Bumping all three to a synthetic version puts every url and checksum in the right place and `brew style` reports no offenses. Replaying the layout that broke mlxcel (url, then `version`, then sha256) now updates url and sha256 and leaves `version` untouched. The corrupted-file state trips the guards (version 0, sha256 2), and a missing artifact trips the url guard. The workflow YAML parses. --- .github/workflows/update_homebrew_formula.yml | 279 +++++++++++------- 1 file changed, 175 insertions(+), 104 deletions(-) diff --git a/.github/workflows/update_homebrew_formula.yml b/.github/workflows/update_homebrew_formula.yml index 7a4a2fba..c56073e8 100644 --- a/.github/workflows/update_homebrew_formula.yml +++ b/.github/workflows/update_homebrew_formula.yml @@ -57,133 +57,204 @@ jobs: - name: Download release artifacts and calculate SHA256 run: | cd homebrew-tap + set -euo pipefail + RAW_VERSION="${{ env.VERSION }}" VERSION="${RAW_VERSION#v}" # Remove leading 'v' echo "VERSION_NO_V=$VERSION" >> $GITHUB_ENV # Export version without 'v' to GitHub env - # bssh URLs - MAC_URL="https://github.com/lablup/bssh/releases/download/v${VERSION}/bssh-macos-aarch64.zip" - LINUX_ARM_URL="https://github.com/lablup/bssh/releases/download/v${VERSION}/bssh-linux-aarch64.tar.gz" - LINUX_X86_URL="https://github.com/lablup/bssh/releases/download/v${VERSION}/bssh-linux-x86_64.tar.gz" - - # bssh-server URLs - SERVER_MAC_URL="https://github.com/lablup/bssh/releases/download/v${VERSION}/bssh-server-macos-aarch64.zip" - SERVER_LINUX_ARM_URL="https://github.com/lablup/bssh/releases/download/v${VERSION}/bssh-server-linux-aarch64.tar.gz" - SERVER_LINUX_X86_URL="https://github.com/lablup/bssh/releases/download/v${VERSION}/bssh-server-linux-x86_64.tar.gz" - - # bssh-keygen URLs - KEYGEN_MAC_URL="https://github.com/lablup/bssh/releases/download/v${VERSION}/bssh-keygen-macos-aarch64.zip" - KEYGEN_LINUX_ARM_URL="https://github.com/lablup/bssh/releases/download/v${VERSION}/bssh-keygen-linux-aarch64.tar.gz" - KEYGEN_LINUX_X86_URL="https://github.com/lablup/bssh/releases/download/v${VERSION}/bssh-keygen-linux-x86_64.tar.gz" - + BASE="https://github.com/lablup/bssh/releases/download/v${VERSION}" mkdir -p tmp - # Download bssh artifacts - curl -Ls "$MAC_URL" -o tmp/mac.zip - curl -Ls "$LINUX_ARM_URL" -o tmp/linux-arm.tar.gz - curl -Ls "$LINUX_X86_URL" -o tmp/linux-x86.tar.gz - - # Download bssh-server artifacts - curl -Ls "$SERVER_MAC_URL" -o tmp/server-mac.zip - curl -Ls "$SERVER_LINUX_ARM_URL" -o tmp/server-linux-arm.tar.gz - curl -Ls "$SERVER_LINUX_X86_URL" -o tmp/server-linux-x86.tar.gz - - # Download bssh-keygen artifacts - curl -Ls "$KEYGEN_MAC_URL" -o tmp/keygen-mac.zip - curl -Ls "$KEYGEN_LINUX_ARM_URL" -o tmp/keygen-linux-arm.tar.gz - curl -Ls "$KEYGEN_LINUX_X86_URL" -o tmp/keygen-linux-x86.tar.gz - - # Export bssh URLs and checksums - echo "mac_url=$MAC_URL" >> $GITHUB_ENV - echo "linux_arm_url=$LINUX_ARM_URL" >> $GITHUB_ENV - echo "linux_x86_url=$LINUX_X86_URL" >> $GITHUB_ENV - echo "mac_sha=$(shasum -a 256 tmp/mac.zip | awk '{print $1}')" >> $GITHUB_ENV - echo "linux_arm_sha=$(shasum -a 256 tmp/linux-arm.tar.gz | awk '{print $1}')" >> $GITHUB_ENV - echo "linux_x86_sha=$(shasum -a 256 tmp/linux-x86.tar.gz | awk '{print $1}')" >> $GITHUB_ENV - - # Export bssh-server URLs and checksums - echo "server_mac_url=$SERVER_MAC_URL" >> $GITHUB_ENV - echo "server_linux_arm_url=$SERVER_LINUX_ARM_URL" >> $GITHUB_ENV - echo "server_linux_x86_url=$SERVER_LINUX_X86_URL" >> $GITHUB_ENV - echo "server_mac_sha=$(shasum -a 256 tmp/server-mac.zip | awk '{print $1}')" >> $GITHUB_ENV - echo "server_linux_arm_sha=$(shasum -a 256 tmp/server-linux-arm.tar.gz | awk '{print $1}')" >> $GITHUB_ENV - echo "server_linux_x86_sha=$(shasum -a 256 tmp/server-linux-x86.tar.gz | awk '{print $1}')" >> $GITHUB_ENV - - # Export bssh-keygen URLs and checksums - echo "keygen_mac_url=$KEYGEN_MAC_URL" >> $GITHUB_ENV - echo "keygen_linux_arm_url=$KEYGEN_LINUX_ARM_URL" >> $GITHUB_ENV - echo "keygen_linux_x86_url=$KEYGEN_LINUX_X86_URL" >> $GITHUB_ENV - echo "keygen_mac_sha=$(shasum -a 256 tmp/keygen-mac.zip | awk '{print $1}')" >> $GITHUB_ENV - echo "keygen_linux_arm_sha=$(shasum -a 256 tmp/keygen-linux-arm.tar.gz | awk '{print $1}')" >> $GITHUB_ENV - echo "keygen_linux_x86_sha=$(shasum -a 256 tmp/keygen-linux-x86.tar.gz | awk '{print $1}')" >> $GITHUB_ENV - - - name: Update bssh formula - run: | - cd homebrew-tap - VERSION="${{ env.VERSION_NO_V }}" # Use version without 'v' - - gsed -i "s/^ version .*/ version \"${VERSION}\"/" Formula/bssh.rb - - gsed -i "s|https://github.com/.*/bssh-macos-aarch64.zip|${mac_url}|" Formula/bssh.rb - gsed -i "/macos-aarch64.zip\"/!b;n;c\ sha256 \"${mac_sha}\"" Formula/bssh.rb - - gsed -i "s|https://github.com/.*/bssh-linux-aarch64.tar.gz|${linux_arm_url}|" Formula/bssh.rb - gsed -i "/linux-aarch64.tar.gz\"/!b;n;c\ sha256 \"${linux_arm_sha}\"" Formula/bssh.rb - - gsed -i "s|https://github.com/.*/bssh-linux-x86_64.tar.gz|${linux_x86_url}|" Formula/bssh.rb - gsed -i "/linux-x86_64.tar.gz\"/!b;n;c\ sha256 \"${linux_x86_sha}\"" Formula/bssh.rb - - - name: Update bssh-server formula + # `-f` so a missing or renamed asset fails here rather than leaving a + # zero-byte file whose checksum gets committed to the tap, and an + # archive integrity check so a truncated download cannot turn into a + # valid-looking sha256 either. + fetch() { # name url dest + local name="$1" url="$2" dest="$3" + curl -fLs --retry 3 "$url" -o "$dest" + if [ ! -s "$dest" ]; then + echo "::error::Empty download: ${url}" + exit 1 + fi + case "$dest" in + *.zip) unzip -qqt "$dest" > /dev/null ;; + *.tar.gz) tar -tzf "$dest" > /dev/null ;; + esac + { + echo "${name}_url=${url}" + echo "${name}_sha=$(shasum -a 256 "$dest" | awk '{print $1}')" + } >> "$GITHUB_ENV" + } + + fetch mac "${BASE}/bssh-macos-aarch64.zip" tmp/mac.zip + fetch linux_arm "${BASE}/bssh-linux-aarch64.tar.gz" tmp/linux-arm.tar.gz + fetch linux_x86 "${BASE}/bssh-linux-x86_64.tar.gz" tmp/linux-x86.tar.gz + + fetch server_mac "${BASE}/bssh-server-macos-aarch64.zip" tmp/server-mac.zip + fetch server_linux_arm "${BASE}/bssh-server-linux-aarch64.tar.gz" tmp/server-linux-arm.tar.gz + fetch server_linux_x86 "${BASE}/bssh-server-linux-x86_64.tar.gz" tmp/server-linux-x86.tar.gz + + fetch keygen_mac "${BASE}/bssh-keygen-macos-aarch64.zip" tmp/keygen-mac.zip + fetch keygen_linux_arm "${BASE}/bssh-keygen-linux-aarch64.tar.gz" tmp/keygen-linux-arm.tar.gz + fetch keygen_linux_x86 "${BASE}/bssh-keygen-linux-x86_64.tar.gz" tmp/keygen-linux-x86.tar.gz + + - name: Update formulae run: | cd homebrew-tap - VERSION="${{ env.VERSION_NO_V }}" - - # Check if bssh-server formula exists - if [ -f "Formula/bssh-server.rb" ]; then - gsed -i "s/^ version .*/ version \"${VERSION}\"/" Formula/bssh-server.rb + set -euo pipefail - gsed -i "s|https://github.com/.*/bssh-server-macos-aarch64.zip|${server_mac_url}|" Formula/bssh-server.rb - gsed -i "/bssh-server-macos-aarch64.zip\"/!b;n;c\ sha256 \"${server_mac_sha}\"" Formula/bssh-server.rb - - gsed -i "s|https://github.com/.*/bssh-server-linux-aarch64.tar.gz|${server_linux_arm_url}|" Formula/bssh-server.rb - gsed -i "/bssh-server-linux-aarch64.tar.gz\"/!b;n;c\ sha256 \"${server_linux_arm_sha}\"" Formula/bssh-server.rb + VERSION="${{ env.VERSION_NO_V }}" # Use version without 'v' - gsed -i "s|https://github.com/.*/bssh-server-linux-x86_64.tar.gz|${server_linux_x86_url}|" Formula/bssh-server.rb - gsed -i "/bssh-server-linux-x86_64.tar.gz\"/!b;n;c\ sha256 \"${server_linux_x86_sha}\"" Formula/bssh-server.rb + # Stanzas are located by name, never by line offset. This step used to + # rewrite each checksum with + # + # gsed -i "/ARTIFACT\"/!b;n;c\ sha256 \"...\"" + # + # which matched the url line, advanced one line, and replaced whatever + # it landed on. That holds only while sha256 sits directly beneath url + # inside an on_macos / on_linux block. lablup/mlxcel ran the same + # construct against a formula whose url had moved to the top level + # with a `version` stanza in between: it overwrote `version` with the + # new checksum and left the previous release's sha256 below, so the + # formula had no version, two sha256 lines, and failed `brew install` + # on checksum mismatch. It sat broken in the tap for three days + # (lablup/homebrew-tap 9969ec4). Nothing here depends on layout now, + # and the guards abort instead of editing the wrong line. + + set_version() { # file version + local file="$1" version="$2" n + n=$(grep -c "^[[:space:]]*version " "$file" || true) + if [ "$n" -ne 1 ]; then + echo "::error::${file}: expected exactly one version stanza, found ${n}." + exit 1 + fi + gsed -i "s|^\([[:space:]]*\)version .*|\1version \"${version}\"|" "$file" + } + + # Rewrites the url stanza whose value contains ARTIFACT, then the + # first sha256 stanza below it, preserving each line's indentation. + set_artifact() { # file artifact url sha + local file="$1" artifact="$2" url="$3" sha="$4" n + n=$(awk -v a="$artifact" '$0 ~ /^[[:space:]]*url / && index($0, a) { c++ } END { print c+0 }' "$file") + if [ "$n" -ne 1 ]; then + echo "::error::${file}: expected exactly one url stanza for ${artifact}, found ${n}." + exit 1 + fi + awk -v a="$artifact" -v u="$url" -v s="$sha" ' + !seen && $0 ~ /^[[:space:]]*url / && index($0, a) { + match($0, /^[[:space:]]*/) + printf "%surl \"%s\"\n", substr($0, 1, RLENGTH), u + seen = 1 + next + } + seen && !hit && $0 ~ /^[[:space:]]*sha256 / { + match($0, /^[[:space:]]*/) + printf "%ssha256 \"%s\"\n", substr($0, 1, RLENGTH), s + hit = 1 + next + } + { print } + END { exit(hit ? 0 : 1) } + ' "$file" > "${file}.new" || { + echo "::error::${file}: no sha256 stanza follows the url for ${artifact}." + rm -f "${file}.new" + exit 1 + } + mv "${file}.new" "$file" + } + + set_version Formula/bssh.rb "$VERSION" + set_artifact Formula/bssh.rb bssh-macos-aarch64.zip "$mac_url" "$mac_sha" + set_artifact Formula/bssh.rb bssh-linux-aarch64.tar.gz "$linux_arm_url" "$linux_arm_sha" + set_artifact Formula/bssh.rb bssh-linux-x86_64.tar.gz "$linux_x86_url" "$linux_x86_sha" + + if [ -f Formula/bssh-server.rb ]; then + set_version Formula/bssh-server.rb "$VERSION" + set_artifact Formula/bssh-server.rb bssh-server-macos-aarch64.zip "$server_mac_url" "$server_mac_sha" + set_artifact Formula/bssh-server.rb bssh-server-linux-aarch64.tar.gz "$server_linux_arm_url" "$server_linux_arm_sha" + set_artifact Formula/bssh-server.rb bssh-server-linux-x86_64.tar.gz "$server_linux_x86_url" "$server_linux_x86_sha" else echo "Warning: Formula/bssh-server.rb not found, skipping server formula update" fi - - name: Update bssh-keygen formula + if [ -f Formula/bssh-keygen.rb ]; then + set_version Formula/bssh-keygen.rb "$VERSION" + set_artifact Formula/bssh-keygen.rb bssh-keygen-macos-aarch64.zip "$keygen_mac_url" "$keygen_mac_sha" + set_artifact Formula/bssh-keygen.rb bssh-keygen-linux-aarch64.tar.gz "$keygen_linux_arm_url" "$keygen_linux_arm_sha" + set_artifact Formula/bssh-keygen.rb bssh-keygen-linux-x86_64.tar.gz "$keygen_linux_x86_url" "$keygen_linux_x86_sha" + else + echo "Warning: Formula/bssh-keygen.rb not found, skipping keygen formula update" + fi + + - name: Validate updated formulae run: | cd homebrew-tap - VERSION="${{ env.VERSION_NO_V }}" + set -euo pipefail - # Check if bssh-keygen formula exists - if [ -f "Formula/bssh-keygen.rb" ]; then - gsed -i "s/^ version .*/ version \"${VERSION}\"/" Formula/bssh-keygen.rb - - gsed -i "s|https://github.com/.*/bssh-keygen-macos-aarch64.zip|${keygen_mac_url}|" Formula/bssh-keygen.rb - gsed -i "/bssh-keygen-macos-aarch64.zip\"/!b;n;c\ sha256 \"${keygen_mac_sha}\"" Formula/bssh-keygen.rb + VERSION="${{ env.VERSION_NO_V }}" - gsed -i "s|https://github.com/.*/bssh-keygen-linux-aarch64.tar.gz|${keygen_linux_arm_url}|" Formula/bssh-keygen.rb - gsed -i "/bssh-keygen-linux-aarch64.tar.gz\"/!b;n;c\ sha256 \"${keygen_linux_arm_sha}\"" Formula/bssh-keygen.rb + # `brew style` is the check the tap itself is expected to pass, and it + # only applies formula cops to paths under Formula/. `ruby -c` is not + # sufficient on its own: the mlxcel corruption described above was + # valid Ruby and would have passed a syntax check. + check() { # file expected_sha... + local file="$1"; shift + local n sha + + echo "=== ${file} ===" + cat "$file" + + ruby -c "$file" + brew style "$file" + + if ! grep -q "^[[:space:]]*version \"${VERSION}\"$" "$file"; then + echo "::error::${file}: version stanza is not ${VERSION}." + exit 1 + fi + + # A substitution that silently no-ops would leave the previous + # release's checksum behind, so require the exact expected set. + n=$(grep -c "^[[:space:]]*sha256 " "$file" || true) + if [ "$n" -ne "$#" ]; then + echo "::error::${file}: expected $# sha256 stanzas, found ${n}." + exit 1 + fi + for sha in "$@"; do + if ! grep -q "^[[:space:]]*sha256 \"${sha}\"$" "$file"; then + echo "::error::${file}: sha256 ${sha} was not written." + exit 1 + fi + done + } + + check Formula/bssh.rb "$mac_sha" "$linux_arm_sha" "$linux_x86_sha" + + if [ -f Formula/bssh-server.rb ]; then + check Formula/bssh-server.rb "$server_mac_sha" "$server_linux_arm_sha" "$server_linux_x86_sha" + fi - gsed -i "s|https://github.com/.*/bssh-keygen-linux-x86_64.tar.gz|${keygen_linux_x86_url}|" Formula/bssh-keygen.rb - gsed -i "/bssh-keygen-linux-x86_64.tar.gz\"/!b;n;c\ sha256 \"${keygen_linux_x86_sha}\"" Formula/bssh-keygen.rb - else - echo "Warning: Formula/bssh-keygen.rb not found, skipping keygen formula update" + if [ -f Formula/bssh-keygen.rb ]; then + check Formula/bssh-keygen.rb "$keygen_mac_sha" "$keygen_linux_arm_sha" "$keygen_linux_x86_sha" fi - name: Commit and push changes to tap run: | cd homebrew-tap - git add Formula/bssh.rb - if [ -f "Formula/bssh-server.rb" ]; then - git add Formula/bssh-server.rb - fi - if [ -f "Formula/bssh-keygen.rb" ]; then - git add Formula/bssh-keygen.rb + set -euo pipefail + + for f in Formula/bssh.rb Formula/bssh-server.rb Formula/bssh-keygen.rb; do + if [ -f "$f" ]; then + git add "$f" + fi + done + + # Re-running the workflow for a version already in the tap is not an + # error; `git commit` would fail on an empty index otherwise. + if git diff --cached --quiet; then + echo "Formulae already up to date for v${{ env.VERSION_NO_V }}, nothing to push." + exit 0 fi + git commit -m "bump: bssh, bssh-server, and bssh-keygen to v${{ env.VERSION_NO_V }}" - git push origin main \ No newline at end of file + git push origin main