Skip to content

Ship GNU wget — fixes https:// support (#130) - #131

Merged
mcfbytes merged 2 commits into
masterfrom
fix/wget-https-gnu-wget
Sep 1, 2026
Merged

Ship GNU wget — fixes https:// support (#130)#131
mcfbytes merged 2 commits into
masterfrom
fix/wget-https-gnu-wget

Conversation

@mcfbytes

@mcfbytes mcfbytes commented Sep 1, 2026

Copy link
Copy Markdown
Owner

Fixes #130. wget https://… failed with wget: not an http or ftp url: while curl worked.

This is not stock parity — it's a regression this image has carried since it existed.

What stock ships

A real GNU wget ELF at usr/bin/wget, linked libc.so.6, libgnutls.so.30, libnettle.so.8, libpcre.so.1, libuuid.so.1, libz.so.1 (binaries-needed-full.txt:351), plus GNU wget's own /etc/wgetrc (etc-configs.md:1097, 4945 bytes — a file the BusyBox applet never reads).

Stock's BusyBox 1.33.1 also had the wget applet compiled in (busybox-applets.md:278, one of its 274) — but the GNU binary owned the path, so the applet was unreachable as wget. Same shape as the lsusb/chvt rows in §4c: two providers, one path, the real ELF wins.

What we shipped, and why it broke

BusyBox was the only provider (busybox.links:217), built with CONFIG_FEATURE_WGET_HTTPS and CONFIG_FEATURE_WGET_OPENSSL both off (resolved .config:1009-1010). With neither set, SSL_SUPPORTED is 0, the P_HTTPS arm of the scheme dispatch is preprocessed away (networking/wget.c:568-572 sits inside #if SSL_SUPPORTED), and every https:// URL reaches the fallthrough at wget.c:578bb_error_msg_and_die("not an http or ftp url: %s"), the reporter's verbatim error. curl was unaffected because we build it with OpenSSL, which is why the report reads "but curl works!".

Why it was missed: T5's collision sweep enumerated applets clashing with packages it was adding. It never looked at a package stock ships that this image had dropped and BusyBox happened to cover by the same name. This is the eighth such collision; the other seven are already handled in busybox.fragment. A user found it, not a build check.

Why it had been left out: the only recorded rationale is the PCRE1 row in the manifest — "the only stock consumers were wget/zsh, neither of which we build" — written to justify dropping libpcre.so.1 after Buildroot 2026.05 removed PCRE1, which quietly took wget with it. That premise is stale: wget.mk:16 passes --disable-pcre unconditionally, so modern GNU wget does not link PCRE1 in any configuration, and wget.mk:67 builds it against the BR2_PACKAGE_PCRE2 we already ship. Nothing here resurrects PCRE1.

The fix

  • BR2_PACKAGE_WGET=y — kconfig confirms this is the only package delta; no new dependencies are pulled in. Its three Config.in deps (BUSYBOX_SHOW_OTHERS, USE_WCHAR, USE_MMU) were already satisfied. TLS backend is GnuTLS — stock's — for free, because wget.mk:26 prefers BR2_PACKAGE_GNUTLS over OpenSSL when both are set, and we set both.
  • # CONFIG_WGET is not set in busybox.fragment — so the GNU binary wins deterministically rather than by install order (same idiom as ifup/ifdown, util-linux, lsof/lsusb/mkdosfs/kbd).
  • Three regression guards in ci-tests.sh. The +https one is the only check that would have caught this bug: a wget built --without-ssl still installs, still owns the path, and still ships wgetrc, so a provider check and a wgetrc check both pass while https stays broken.
  • docs/package-manifest.md §4d documents the whole thing; the stale PCRE1 row is corrected in place.

Verified, not predicted

make wget + make busybox-reconfigure against the warm tree:

  • configure resolved to --with-ssl=gnutls --disable-pcre --enable-pcre2 --with-libuuid --with-zlib --without-libpsl --disable-iri --without-cares; its summary reports PCRE: yes, via libpcre2
  • --version under qemu-arm: -cares +digest -gpgme **+https** +ipv6 -iri +large-file -metalink -nls +ntlm +opie -psl **+ssl/gnutls**
  • /etc/wgetrc installs at 4945 bytes — byte-for-byte stock's size
  • usr/bin/wget went from a symlink → ../../bin/busybox to a 564,760-byte GNU wget ELF, and survived a subsequent busybox reinstall — the exact install-order race the fragment change closes. busybox.links no longer contains wget.

DT_NEEDED vs stock:

stock ours
libgnutls.so.30, libnettle.so.8, libuuid.so.1, libz.so.1, libc.so.6, ld-linux-armhf.so.3 same
libpcre.so.1 libpcre2-8.so.0 intended substitution
(absent) libunistring.so.5 unpredicted, benign — wget 1.25.0's gnulib links it. Costs nothing: BR2_PACKAGE_LIBUNISTRING=y was already set and the .so already in the image. Documented rather than glossed.
libpsl, libidn2, c-ares absent absent matches stock

Rejected alternative

Flipping CONFIG_FEATURE_WGET_HTTPS=y in the fragment is the one-line version and is the wrong call. BusyBox's own Kconfig help for that symbol states its internal TLS "does not check that the peer is who it claims to be", does not verify signature hashes on incoming data, and does not check the peer's certificate — an https:// that silently accepts an impersonating server, on a device whose network use is downloading cores and updates. It would also still ignore /etc/wgetrc and still not match stock's option set.

Stated divergence

Stock kept CONFIG_WGET=y and shipped the GNU ELF, so busybox wget stayed reachable there. Buildroot cannot express that safely — two packages owning usr/bin/wget is the install-order race busybox.fragment exists to prevent — so the applet is off and that spelling is gone. Nothing in the repo calls it by that spelling; the wget name, which is what scripts use, is answered by a strictly more capable binary.

Not covered

No full make all and no on-device test — only the targeted make wget / busybox-reconfigure above. The new ci-tests.sh checks need a rootfs.tar from a full build to run.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LUA7AfLnA285hUUKrzfHFf

`wget https://...` failed with "wget: not an http or ftp url:" while curl
worked. Not a stock-parity artifact: a regression this image has carried
since it existed.

Stock ships a real GNU wget ELF at usr/bin/wget linked against
libgnutls.so.30/libnettle.so.8/libpcre.so.1/libuuid.so.1/libz.so.1
(docs/stock-inventory/binaries-needed-full.txt:351), plus GNU wget's own
/etc/wgetrc (etc-configs.md:1097). Stock's BusyBox 1.33.1 also had the wget
applet compiled in (busybox-applets.md:278), but the GNU binary owned the
path, so the applet was unreachable as `wget`.

We shipped only the BusyBox applet, with CONFIG_FEATURE_WGET_HTTPS and
CONFIG_FEATURE_WGET_OPENSSL both off. That makes SSL_SUPPORTED 0, so the
P_HTTPS arm of networking/wget.c:568-572 is preprocessed away and every
https URL falls through to bb_error_msg_and_die("not an http or ftp url")
at wget.c:578 — the reporter's verbatim error.

T5's collision sweep missed it because it only looked at applets clashing
with packages it was *adding*, never at a package stock ships that this
image had dropped and BusyBox covered by the same name. This is the eighth
such collision; the other seven are already handled in busybox.fragment.

The recorded reason for dropping wget was the PCRE1 removal note
("the only stock consumers were wget/zsh, neither of which we build").
Stale: wget.mk:16 passes --disable-pcre unconditionally, so GNU wget does
not link PCRE1 in any configuration, and wget.mk:67 builds it against the
BR2_PACKAGE_PCRE2 we already ship. Nothing resurrects PCRE1.

  - BR2_PACKAGE_WGET=y — the only package delta; kconfig confirms no new
    dependencies are pulled in. Its three Config.in deps
    (BUSYBOX_SHOW_OTHERS, USE_WCHAR, USE_MMU) were already satisfied, and
    wget.mk:26 picks GnuTLS — stock's backend — because we set both it and
    OpenSSL.
  - "# CONFIG_WGET is not set" in busybox.fragment, so the GNU binary wins
    deterministically rather than by install order (same idiom as
    ifup/ifdown, util-linux, lsof/lsusb/mkdosfs/kbd).
  - Three regression guards in ci-tests.sh. The +https one is the only
    check that would have caught this bug: a wget built --without-ssl
    still installs, still owns the path, and still ships wgetrc.

Rejected the cheaper CONFIG_FEATURE_WGET_HTTPS=y instead: BusyBox's own
Kconfig help says its internal TLS does not check that the peer is who it
claims to be, does not verify signature hashes, and does not check the
peer's certificate — an https:// that silently accepts an impersonating
server, on a device whose network use is downloading cores and updates.

Verified with `make wget` + `busybox-reconfigure` against the warm tree,
not predicted from the .mk:
  - configure resolved to --with-ssl=gnutls --disable-pcre --enable-pcre2
    --with-libuuid --with-zlib --without-libpsl --disable-iri
    --without-cares; summary reports "PCRE: yes, via libpcre2"
  - --version under qemu-arm reports "+https ... +ssl/gnutls"
  - DT_NEEDED matches stock exactly but for libpcre.so.1 -> libpcre2-8.so.0
    and an unpredicted libunistring.so.5 (wget 1.25.0's gnulib links it;
    BR2_PACKAGE_LIBUNISTRING was already on and the .so already in the
    image, so it costs nothing). Documented rather than glossed.
  - /etc/wgetrc installs at 4945 bytes — byte-for-byte stock's size
  - usr/bin/wget went from a symlink -> ../../bin/busybox to a 564,760-byte
    GNU wget ELF, and survived a subsequent busybox reinstall (the exact
    install-order race the fragment change closes); busybox.links no longer
    contains wget

Nothing in the repo calls `busybox wget` by that spelling.

Closes #130

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LUA7AfLnA285hUUKrzfHFf
Signed-off-by: Michael C. Ferguson <michael.christopher.ferguson@gmail.com>
Copilot AI lite review requested due to automatic review settings September 1, 2026 14:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new CI check’s qemu_target | grep pipeline can misdiagnose execution failures as “missing +https,” and a couple of new references in comments are currently ambiguous/broken.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Restores stock-like wget behavior by shipping GNU wget (with HTTPS via GnuTLS) and disabling the BusyBox wget applet to avoid path-collision regressions, addressing issue #130 where wget https://… failed.

Changes:

  • Enable BR2_PACKAGE_WGET and disable BusyBox CONFIG_WGET to ensure usr/bin/wget is GNU wget deterministically.
  • Add CI regression guards to ensure wget is not a BusyBox symlink, wgetrc is present, and wget --version reports +https.
  • Update package manifest documentation to reflect the restored GNU wget and correct the PCRE1 rationale.
File summaries
File Description
scripts/ci-tests.sh Adds regression checks ensuring GNU wget owns usr/bin/wget, installs /etc/wgetrc, and has HTTPS enabled.
docs/package-manifest.md Documents the wget collision/regression and updates the PCRE1 row explanation accordingly.
configs/mister_de10nano_defconfig Enables GNU wget in the Buildroot defconfig with detailed parity/regression context.
board/mister/de10nano/busybox.fragment Disables BusyBox wget applet to prevent install-order collisions and restore HTTPS functionality via GNU wget.
Review details
  • Files reviewed: 4/4 changed files
  • Comments generated: 3
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread scripts/ci-tests.sh
Comment thread board/mister/de10nano/busybox.fragment Outdated
Comment thread configs/mister_de10nano_defconfig Outdated
All three findings were valid.

1. ci-tests.sh: the +https check was `qemu_target ... | grep -q`, which only
   reports grep's exit status. A qemu/loader/sysroot failure yields empty
   output and would have been reported as "built --without-ssl?" -- the
   wrong diagnosis, and precisely the failure mode this check exists to name
   precisely. Now runs and tests separately, with a distinct label and
   message for the could-not-execute case.

   Checked before applying rather than assuming, because wget does warn on
   stderr under qemu that it cannot read /etc/wgetrc (the file is not
   root-owned in output/target, so wget's startup-file security check
   rejects it): `--version` still exits 0 and still prints the banner to
   stdout, so gating on exit status cannot produce a false FAIL. Recorded in
   the comment so the next reader does not "fix" it by removing the check.

2. busybox.fragment: bare `wget.c:568-572` / `wget.c:578` citations are
   ambiguous (BusyBox has many *.c files) and inconsistent with the
   surrounding text, which already says networking/wget.c -- and with this
   file's own style elsewhere (networking/nc.c:17-21,
   console-tools/loadfont.c:53). Fully qualified both; rewrapped the line
   that grew past the file's width.

3. defconfig: the citation was split as `etc-configs.md` / `:1097` across
   two comment lines, breaking grep for the file:line token. Rejoined.
   Fixed the same class of split in ci-tests.sh (`docs/` /
   `stock-inventory/...`), which Copilot did not flag.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LUA7AfLnA285hUUKrzfHFf
Signed-off-by: Michael C. Ferguson <michael.christopher.ferguson@gmail.com>
@mcfbytes

mcfbytes commented Sep 1, 2026

Copy link
Copy Markdown
Owner Author

Full local rebuild: green, and the new checks fire correctly

make all completed clean — linux.img written, check-zimage-dtb.sh and check-linux-img.sh all assertions passed, no errors in 7,445 log lines. This closes the "Not covered" caveat in the PR description.

All three new guards pass against a real rootfs.tar:

PASS  wget (GNU wget -- issue #130, https support) is the real package, not BusyBox
PASS  /etc/wgetrc (GNU wget's config -- BusyBox's applet never reads one) present
PASS  wget has https support (+https)

In the shipped image usr/bin/wget is a real 492,184-byte stripped ELF (not a busybox symlink) and /etc/wgetrc is 4,945 bytes — stock's exact size. Total footprint 0.47 MiB, no man pages or locale data (--disable-doc/--disable-nls). Size budget passes at 36.4% free.

Three unrelated FAILs in the same suite — none from this PR

Recording these so nobody mistakes them for fallout from this change. Neither is fixed here.

1. Stale out-of-tree modules (real shipping bug, pre-existing).

FAIL  xone: all 9 .ko.xz modules present
FAIL  out-of-tree WiFi: 8852cu.ko.xz present under updates/

The image carries four module trees and the OOT modules are stranded on the oldest:

tree size contents
6.18.46 4.8 MiB all 9 xone + 8852cu ← stale
6.18.47 3.0 MiB orphaned
6.18.48 3.0 MiB ← the running kernel, no OOT modules
7.2.1 3.2 MiB RT variant (intentional)

On 6.18.48 the Xbox dongle and RTL8852CU would not load. Cause is three renovate kernel bumps (b14ab84ce172c10fec80e) landing without the kmod packages being rebuilt — silent, exit 0, the documented Buildroot stamping trap. git diff master...HEAD on this branch touches the kernel pin not at all; the rebuild only surfaced it. Also wastes 7.8 MiB on two dead trees. Worth its own issue.

2. test-initramfs.sh — harness bug, not an image defect. 5/8 pass; the three exFAT cases fail because the harness's throwaway QEMU kernel has no exFAT driver (mount -t exfat → "No such device" in-guest). scripts/test-initramfs/qemu-test-kernel.config:25 does set CONFIG_EXFAT_FS=y and merge_config.sh logged Previous value: # CONFIG_EXFAT_FS is not set, yet the resolved .config ends up with it unset — and EXFAT_FS has no depends on at all, only selects (BUFFER_HEAD, NLS, LEGACY_DIRECT_IO), every one of which is satisfied there. So nothing legitimately dropped it. Untested lead: merge_config.sh is invoked without ARCH=arm (test-initramfs.sh:193), so its internal alldefconfig may resolve for the host arch before the later ARCH=arm olddefconfig.

The product kernel is unaffected — output/build/linux-6.18.48/.config has CONFIG_EXFAT_FS=y and carries board patch 0031. exFAT cards still work; only the test's own kernel can't mount them.

@mcfbytes
mcfbytes merged commit d9a8256 into master Sep 1, 2026
6 checks passed
@mcfbytes
mcfbytes deleted the fix/wget-https-gnu-wget branch September 1, 2026 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] wget doesn't support https

2 participants