diff --git a/board/mister/de10nano/busybox.fragment b/board/mister/de10nano/busybox.fragment index 9df14c6..a3f56f5 100644 --- a/board/mister/de10nano/busybox.fragment +++ b/board/mister/de10nano/busybox.fragment @@ -211,3 +211,40 @@ CONFIG_NC_110_COMPAT=y # CONFIG_DEALLOCVT is not set # CONFIG_OPENVT is not set # CONFIG_SETKEYCODES is not set + +# GNU wget vs BusyBox's wget applet -- issue #130 (2026-09-01). Same +# "two packages install the same path, last one wins" hazard as the ifup/ifdown, +# util-linux and lsof/lsusb/mkdosfs/kbd blocks above, and the same fix. +# BR2_PACKAGE_WGET (added to the defconfig in the same change) installs a real +# GNU wget 1.25.0 at usr/bin/wget; BusyBox's CONFIG_WGET applet -- ON in the base +# config (output/build/busybox-1.38.0/.config:1003, read directly) -- claims the +# identical path via busybox.links:217. Turning the applet off makes the GNU +# binary the deterministic winner. +# +# This one is a functional fix, not just determinism. BusyBox's applet was built +# with BOTH of its TLS knobs off -- "# CONFIG_FEATURE_WGET_HTTPS is not set" +# (.config:1009) and "# CONFIG_FEATURE_WGET_OPENSSL is not set" (.config:1010) -- +# which makes SSL_SUPPORTED 0, so networking/wget.c's scheme dispatch has no +# https:// arm at all (the P_HTTPS branch at networking/wget.c:568-572 is inside +# `#if SSL_SUPPORTED`) and every https URL falls through to +# bb_error_msg_and_die("not an http or ftp url: %s") at networking/wget.c:578. +# That is the verbatim error in issue #130. +# +# Considered and rejected: flipping CONFIG_FEATURE_WGET_HTTPS=y here instead of +# adding the package. It is the cheaper edit but the wrong one. BusyBox's own +# Kconfig help for that symbol says its internal TLS "does *not* check that the +# peer is who it claims to be", does not verify signature hashes on incoming +# data, and does not check the peer's certificate -- i.e. it would give the +# MiSTer an https:// that silently accepts an impersonating server, on a box +# whose whole job is downloading cores and updates over the network. It would +# also still ignore /etc/wgetrc and still not match stock's option set. Stock +# solved this by shipping GNU wget; so do we. +# +# Divergence from stock, stated rather than glossed: stock kept CONFIG_WGET=y in +# its BusyBox *and* shipped the GNU ELF, so `busybox wget` remained reachable +# there as a hand-built-image accident. Buildroot cannot express that safely -- +# two packages owning usr/bin/wget is exactly the non-deterministic install-order +# race this file exists to prevent -- so the applet goes off and `busybox wget` +# is gone. Nothing calls it by that spelling; the `wget` name, which is what +# scripts use, is answered by a strictly more capable binary. +# CONFIG_WGET is not set diff --git a/configs/mister_de10nano_defconfig b/configs/mister_de10nano_defconfig index 20b75d2..14e2e35 100644 --- a/configs/mister_de10nano_defconfig +++ b/configs/mister_de10nano_defconfig @@ -480,6 +480,44 @@ BR2_PACKAGE_LIBCURL_CURL=y # installs the `curl` CLI binary -- of # default, stock ships it, community scripts use it BR2_PACKAGE_LIBCURL_OPENSSL=y # TLS backend parity: stock's curl links # libcrypto/libssl, not GnuTLS +# GNU wget -- stock parity restored (issue #130, 2026-09-01). Stock ships a real +# GNU wget ELF at usr/bin/wget linked against libgnutls.so.30, libnettle.so.8, +# libpcre.so.1, libuuid.so.1 and libz.so.1 (docs/stock-inventory/ +# binaries-needed-full.txt:351), plus GNU wget's own /etc/wgetrc -- 4945 bytes, +# see etc-configs.md:1097 -- a file BusyBox's applet never reads. Stock's BusyBox +# 1.33.1 ALSO had the wget applet compiled in (busybox-applets.md:278), but the +# GNU ELF owned the path, so the applet was unreachable as `wget` -- the same +# "two providers, one path" shape as ifup/util-linux/lsof below. +# This image previously shipped ONLY the BusyBox applet, with +# CONFIG_FEATURE_WGET_HTTPS and CONFIG_FEATURE_WGET_OPENSSL both off, so +# SSL_SUPPORTED was 0 and every https:// URL died at networking/wget.c:578 with +# "wget: not an http or ftp url:" -- while curl (above) worked, which is exactly +# how issue #130 was reported. +# The recorded reason for leaving wget out was the PCRE1 removal note further +# down this file ("the only stock consumers were wget/zsh, neither of which we +# build"). That premise is stale: this Buildroot's wget.mk:16 passes +# --disable-pcre UNCONDITIONALLY, so GNU wget does not want PCRE1 at all, and +# wget.mk:67 gives it --enable-pcre2 against the BR2_PACKAGE_PCRE2 we already +# ship. Nothing here resurrects PCRE1. +# TLS backend is GnuTLS, matching stock, and for free: wget.mk:26 prefers +# BR2_PACKAGE_GNUTLS over OpenSSL when both are present, and we set both. +# Built and readelf'd, not predicted (2026-09-01): the resolved DT_NEEDED is +# libgnutls.so.30, libnettle.so.8, libuuid.so.1, libz.so.1, libc.so.6 and +# ld-linux-armhf.so.3 -- identical to stock's -- plus libpcre2-8.so.0 where +# stock had libpcre.so.1, plus libunistring.so.5, which stock's older wget did +# not link. The last one is free: BR2_PACKAGE_LIBUNISTRING was already set and +# the .so was already in the image before this change. libpsl, libidn2 and +# c-ares stay out (wget.mk:18/40/60 take their --without/--disable branches), +# which is also what stock did -- none of the three is in stock's list either. +# The installed /etc/wgetrc is 4945 bytes, byte-for-byte stock's size, and the +# ARM binary's --version banner reports "+https ... +ssl/gnutls" under qemu-arm. +# Prerequisites were already satisfied, nothing else had to change: +# BR2_PACKAGE_BUSYBOX_SHOW_OTHERS=y (below), BR2_USE_WCHAR=y and BR2_USE_MMU=y. +# The colliding BusyBox applet is turned off in board/mister/de10nano/ +# busybox.fragment so this binary wins deterministically (same idiom as +# ifup/ifdown and the util-linux block). +BR2_PACKAGE_WGET=y # real GNU wget 1.25.0 w/ GnuTLS -- https + # works, /etc/wgetrc is read (stock parity) BR2_PACKAGE_DBUS=y BR2_PACKAGE_DBUS_CPP=y # dbusxx-introspect; low-value but zero-cost parity BR2_PACKAGE_DBUS_GLIB=y diff --git a/docs/package-manifest.md b/docs/package-manifest.md index ea025d8..1923446 100644 --- a/docs/package-manifest.md +++ b/docs/package-manifest.md @@ -259,7 +259,7 @@ Columns: **SONAME** | **stock realfile** (version hint, from `shared-libraries.m | `libtorrent.so.21` | `libtorrent.so.21.0.0` | `BR2_PACKAGE_LIBTORRENT` | 0.15.3 | **YES (likely)** | rakshasa's libtorrent (rtorrent's library, distinct from libtorrent-rasterbar); SONAME has moved past 21 in this version range per Debian's experimental packaging (`libtorrent27`). **Recommend dropping `rtorrent`/`libtorrent` entirely (see Drop list) — nothing MiSTer-side uses it**, so the exact soname doesn't matter | | `libxml2.so.2` | `libxml2.so.2.9.12` | `BR2_PACKAGE_LIBXML2` | 2.15.3 | no | | | `libmagic.so.1` | `libmagic.so.1.0.0` | `BR2_PACKAGE_FILE` | 5.46 | no | | -| `libpcre.so.1` | *(not provided)* | — | — | no | **Intentional parity deviation (Buildroot 2026.05).** PCRE1 was removed upstream (EOL/unmaintained; now a `Config.in.legacy` stub). Nothing in this image needs `libpcre.so.1`: the stock `MiSTer` binary does not link it (no `-lpcre`, verified against `origin/master`), Python uses its built-in `sre` engine, and 2026.05's `slang` dropped its pcre module. The only stock consumers were `wget`/`zsh`, neither of which we build. `libpcre2-8.so.0` (PCRE2) is provided as the modern replacement, pulled by `libglib2`/`libselinux` and listed explicitly in the defconfig. | +| `libpcre.so.1` | *(not provided)* | — | — | no | **Intentional parity deviation (Buildroot 2026.05).** PCRE1 was removed upstream (EOL/unmaintained; now a `Config.in.legacy` stub). Nothing in this image needs `libpcre.so.1`: the stock `MiSTer` binary does not link it (no `-lpcre`, verified against `origin/master`), Python uses its built-in `sre` engine, and 2026.05's `slang` dropped its pcre module. The only stock consumers were `wget`/`zsh`. **Corrected 2026-09-01 (issue #130):** the "neither of which we build" half of this sentence used to be true and is no longer — `BR2_PACKAGE_WGET=y` now ships GNU wget again (§4d below). That does **not** weaken this row: this Buildroot's `wget.mk:16` passes `--disable-pcre` unconditionally, so modern GNU wget does not link PCRE1 under any configuration, and `wget.mk:67` builds it against `BR2_PACKAGE_PCRE2` instead. `zsh` is still not built. `libpcre2-8.so.0` (PCRE2) is provided as the modern replacement, pulled by `libglib2`/`libselinux` and listed explicitly in the defconfig. | | `libpcreposix.so.0` | *(not provided)* | — | — | no | Dropped with PCRE1 (same row above) — it is PCRE1's POSIX wrapper. PCRE2 ships its own `libpcre2-posix.so.3`. | | `libudev.so.1` | `libudev.so.1.6.3` | `BR2_PACKAGE_EUDEV` | 3.2.14 | no | eudev, not systemd-udev — matches PLAN §3 ("hotplug is eudev, not mdev") | @@ -801,6 +801,93 @@ package won each one, not just that the path is present. --- +### 4d. GNU wget — an eighth BusyBox collision T5 missed (issue #130, 2026-09-01) + +T5's collision sweep (§4c) enumerated the applets that clash with packages *it +was adding*. It never looked at a package stock ships that this image had +dropped **and** BusyBox happened to cover by the same name — so `wget` fell +through every net, and shipped as a crippled BusyBox applet for as long as this +image has existed. A user found it, not a build check. + +**What stock ships.** A real GNU wget ELF at `usr/bin/wget`, linked +`libc.so.6, libgnutls.so.30, libnettle.so.8, libpcre.so.1, libuuid.so.1, +libz.so.1` (`docs/stock-inventory/binaries-needed-full.txt:351`), plus GNU +wget's own `/etc/wgetrc` (`etc-configs.md:1097`, 4945 bytes — a file the BusyBox +applet never reads). Stock's BusyBox 1.33.1 *also* had the applet compiled in +(`busybox-applets.md:278`, one of its 274), but the GNU binary owned the path, +so the applet was unreachable as `wget`. Same shape as §4c's `lsusb`/`chvt` +rows: two providers, one path, the real ELF wins. + +**What we shipped instead, and why it broke.** BusyBox was the only provider +(`busybox.links:217`), built with `CONFIG_FEATURE_WGET_HTTPS` **and** +`CONFIG_FEATURE_WGET_OPENSSL` both off (resolved `.config:1009-1010`). With +neither set, `SSL_SUPPORTED` is 0, the `P_HTTPS` arm of the scheme dispatch is +preprocessed away (`networking/wget.c:568-572` sits inside `#if SSL_SUPPORTED`), +and every `https://` URL reaches the fallthrough at `wget.c:578`: +`bb_error_msg_and_die("not an http or ftp url: %s")`. That is the verbatim text +in issue #130. `curl` was unaffected — we build it with OpenSSL — which is why +the report reads "but curl works!". + +**Why it had been left out, and why that reason no longer holds.** The only +recorded rationale is the PCRE1 row in §"Libraries" above: *"the only stock +consumers were `wget`/`zsh`, neither of which we build."* That was written to +justify dropping `libpcre.so.1` after Buildroot 2026.05 removed PCRE1, and it +quietly took wget down with it. The premise is stale — this Buildroot's +`wget.mk:16` passes `--disable-pcre` **unconditionally**, so GNU wget does not +link PCRE1 in any configuration, and `wget.mk:67` builds it against the +`BR2_PACKAGE_PCRE2` we already ship. Restoring wget resurrects nothing. + +**The fix.** `BR2_PACKAGE_WGET=y` in the defconfig, `# CONFIG_WGET is not set` +in `board/mister/de10nano/busybox.fragment`. No other symbol had to change: +`BR2_PACKAGE_BUSYBOX_SHOW_OTHERS=y`, `BR2_USE_WCHAR=y` and `BR2_USE_MMU=y` (the +package's three `Config.in` dependencies) were all already set. TLS backend is +GnuTLS — stock's — for free, because `wget.mk:26` prefers `BR2_PACKAGE_GNUTLS` +over OpenSSL when both are present and we set both. + +**Resulting link set vs stock.** This is a parity claim, so it was *built* and +`readelf -d`'d rather than predicted from the `.mk` — and doing so turned up one +difference the prediction missed (`libunistring`, last row). Each line is traced +to the `wget.mk` conditional that decides it: + +| stock DT_NEEDED | ours | how | +|---|---|---| +| `libgnutls.so.30`, `libnettle.so.8` | same | `wget.mk:26`, `--with-ssl=gnutls` (`BR2_PACKAGE_GNUTLS=y`) | +| `libuuid.so.1` | same | `wget.mk:46`, `--with-libuuid` (`BR2_PACKAGE_UTIL_LINUX_LIBUUID=y`) | +| `libz.so.1` | same | `wget.mk:53`, `--with-zlib` (`BR2_PACKAGE_ZLIB=y`) | +| `libc.so.6`, `ld-linux-armhf.so.3` | same | — | +| `libpcre.so.1` | `libpcre2-8.so.0` | intended substitution. `wget.mk:16` `--disable-pcre` is unconditional; `wget.mk:67` `--enable-pcre2` (`BR2_PACKAGE_PCRE2=y`) | +| *(absent)* | *(absent)* | `libpsl`, `libidn2`, `c-ares` — `wget.mk:18/40/60` all take their `--without`/`--disable` branches here, and none appears in stock's list either | +| *(absent)* | `libunistring.so.5` | **unpredicted, benign.** wget 1.25.0's bundled gnulib links `-lunistring`; stock's older wget did not, and `shared-libraries.md` has no libunistring row. Costs nothing: `BR2_PACKAGE_LIBUNISTRING=y` was *already* set and `usr/lib/libunistring.so.5.2.1` was already in the image before this change, pulled in by another package. No new runtime dependency, no new bytes. | + +**Verified, not assumed** (2026-09-01, `make wget` against the warm tree): +`./configure` resolved to `--with-ssl=gnutls --disable-pcre --enable-pcre2 +--with-libuuid --with-zlib --without-libpsl --disable-iri --without-cares`, +its summary reports `PCRE: yes, via libpcre2`, and the built ARM ELF's +`--version` banner under `qemu-arm` reads `+https ... +ssl/gnutls`. The +installed `/etc/wgetrc` is **4945 bytes — byte-for-byte stock's size** +(`etc-configs.md:1097`). Before the change, `output/target/usr/bin/wget` was +a symlink → `../../bin/busybox`; after it, a 564,760-byte GNU wget ELF. + +**Rejected alternative.** Flipping `CONFIG_FEATURE_WGET_HTTPS=y` in the +fragment is the one-line version and is the wrong call. BusyBox's own Kconfig +help for that symbol states its internal TLS "does *not* check that the peer is +who it claims to be", does not verify signature hashes on incoming data, and +does not check the peer's certificate — an `https://` that silently accepts an +impersonating server, on a device whose network use is downloading cores and +updates. It would also still ignore `/etc/wgetrc` and still not match stock's +option set. + +**Stated divergence.** Stock kept `CONFIG_WGET=y` *and* shipped the GNU ELF, so +`busybox wget` stayed reachable there. Buildroot cannot express that safely — +two packages owning `usr/bin/wget` is the install-order race +`busybox.fragment` exists to prevent — so the applet is off and that spelling is +gone. The `wget` name, which is what scripts actually use, is answered by a +strictly more capable binary. Guarded by `scripts/ci-tests.sh` §"T5" +(`not_busybox_symlink usr/bin/wget`, plus an `/etc/wgetrc` presence check that +distinguishes the two providers). + +--- + ## 5. Drop list — what stock ships that we should not The rootfs is **93% full** (297/347 MiB per PLAN §4.2, confirmed 13.6% free by diff --git a/scripts/ci-tests.sh b/scripts/ci-tests.sh index 056b739..fd4b8c5 100755 --- a/scripts/ci-tests.sh +++ b/scripts/ci-tests.sh @@ -1256,6 +1256,51 @@ not_busybox_symlink "usr/bin/openvt" "openvt (kbd)" not_busybox_symlink "usr/bin/deallocvt" "deallocvt (kbd)" not_busybox_symlink "usr/bin/setkeycodes" "setkeycodes (kbd)" +# -- The eighth collision, found by a user rather than by T5 (issue #130) ----- +# T5 swept for applets clashing with packages it was ADDING; it never looked at +# a package stock ships that this image had DROPPED and BusyBox covered by the +# same name. `wget` was exactly that, and shipped as a crippled applet with no +# https support at all until 2026-09-01. See docs/package-manifest.md §4d. +# Three checks, because each catches a different way of regressing: +# 1. provider -- BR2_PACKAGE_WGET off again (or CONFIG_WGET back on) would +# silently hand usr/bin/wget back to BusyBox. +not_busybox_symlink "usr/bin/wget" "wget (GNU wget -- issue #130, https support)" +# 2. /etc/wgetrc -- installed by the GNU wget package only; the BusyBox applet +# neither ships nor reads it, so this distinguishes the two providers by +# something other than the binary itself -- stock has it, see +# docs/stock-inventory/etc-configs.md:1097 +require_present "etc/wgetrc" "/etc/wgetrc (GNU wget's config -- BusyBox's applet never reads one)" +# 3. +https -- the actual bug. GNU wget's --version banner prints a feature +# line of +/-flags; a wget built --without-ssl still installs, still owns +# the path, and still ships wgetrc, so checks 1 and 2 would both pass while +# https stayed broken. This is the only one that would have caught #130. +# Deliberately NOT written as `qemu_target ... | grep -q`: a pipeline only +# reports grep's status, so a qemu/loader/sysroot problem (empty output) +# would be indistinguishable from the https regression and would send the +# reader after the wrong bug -- the one failure mode this check exists to +# name precisely. Run and test separately instead. +# `--version` exits 0 even when wget warns on stderr that it cannot read +# /etc/wgetrc -- which it DOES under qemu, because the file is not +# root-owned in output/target and wget's startup-file security check +# rejects it. Verified (exit 0, banner still on stdout), so the status +# test below is safe; do not "fix" that warning by dropping the check. +if [ -z "$QEMU_ARM" ]; then + skip "wget has https support (+https)" "qemu-arm not found on PATH" +elif [ ! -x "$TARGET/usr/bin/wget" ]; then + skip "wget has https support (+https)" "$TARGET/usr/bin/wget not present in output/target" +else + wget_ver="$WORKDIR/wget-version.txt" + if ! qemu_target "$TARGET/usr/bin/wget" --version >"$wget_ver" 2>/dev/null; then + fail "wget --version runs under qemu-arm" \ + "could not execute $TARGET/usr/bin/wget (loader/sysroot problem, not an https finding) -- +https NOT verified" + elif grep -q -- '+https' "$wget_ver"; then + pass "wget has https support (+https)" + else + fail "wget has https support (+https)" \ + "ran, but --version reports no +https -- built --without-ssl? banner: $(head -1 "$wget_ver")" + fi +fi + # ============================================================================= section "P3.5 — Bluetooth parity" # =============================================================================