Skip to content

Bundle relocated interpreter's libdir into the dist toolchain package#2747

Closed
jetm wants to merge 1 commit into
mozilla:mainfrom
jetm:fix/uninative-toolchain-libdir
Closed

Bundle relocated interpreter's libdir into the dist toolchain package#2747
jetm wants to merge 1 commit into
mozilla:mainfrom
jetm:fix/uninative-toolchain-libdir

Conversation

@jetm

@jetm jetm commented Jun 20, 2026

Copy link
Copy Markdown

Problem

sccache-dist packages a toolchain's shared libraries by parsing ldd
output, which resolves a binary's NEEDED libraries against the host's
dynamic loader. Yocto/OpenEmbedded "uninative" cross toolchains ship a
relocated glibc whose loader has a built-in search path pointing at its
own sysroot. For those binaries ldd reports host paths (e.g.
/usr/lib/libm.so.6), but inside the build sandbox the relocated loader
searches its own sysroot lib dir — where those libraries were never
packaged. The remote compile then dies with libm.so.6: cannot open shared object file and silently falls back to local compilation, so
distribution never actually runs.

Solution

When a packaged executable's PT_INTERP lives outside the standard host
loader directories (/lib, /lib64, /usr/lib, /usr/lib64), also
bundle the interpreter's own directory. That directory holds the
libc/libm the relocated loader resolves against, so they land at the
absolute path the loader searches inside the sandbox. Standard host
toolchains are unaffected: their interpreter is under a standard loader
directory, so the existing ldd-only path is preserved.

Key changes

  • Add read_elf_interpreter() to read a binary's PT_INTERP via the
    already-vendored object crate.
  • Add relocated_interpreter_libdir() to classify an interpreter as
    relocated and return the directory to bundle.
  • In add_executable_and_deps(), bundle that directory's contents when a
    relocated interpreter is detected.
  • Unit test covering standard vs. relocated interpreter classification.

Reviewer notes

  • No behavior change for standard host toolchains — their interpreter is
    under /lib*, so nothing extra is bundled.
  • Verified against a real OpenEmbedded aarch64 cross toolchain: a full
    Yocto image build distributed 634 compiles to the build server with 0
    dist errors and no libm.so.6 failures, and the packaged toolchain tar
    now contains the loader's libc/libm at the uninative sysroot path.

sccache-dist packages a toolchain's shared libraries by parsing `ldd`
output, which resolves NEEDED libraries against the host's dynamic
loader. Yocto/OpenEmbedded "uninative" cross toolchains ship a relocated
glibc whose loader has a built-in search path pointing at its own
sysroot. For those binaries `ldd` reports host paths (e.g.
/usr/lib/libm.so.6), yet inside the build sandbox the relocated loader
searches its own sysroot lib dir, where those libraries were never
packaged. The remote compile then dies with "libm.so.6: cannot open
shared object file" and silently falls back to local compilation, so
distribution never actually runs.

When a packaged executable's PT_INTERP lives outside the standard host
loader directories, also bundle the interpreter's own directory. That
directory holds the libc/libm the relocated loader resolves against, so
they land at the absolute path the loader searches inside the sandbox.
Standard host toolchains are untouched: their interpreter is under /lib,
/lib64, or /usr/lib, so the existing ldd-only path is preserved.

Signed-off-by: Javier Tia <javier@peridio.com>
@codecov-commenter

codecov-commenter commented Jun 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.97872% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.64%. Comparing base (b799af2) to head (082e7a0).

Files with missing lines Patch % Lines
src/dist/pkg.rs 82.97% 8 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2747   +/-   ##
=======================================
  Coverage   74.63%   74.64%           
=======================================
  Files          70       70           
  Lines       39898    39945   +47     
=======================================
+ Hits        29778    29817   +39     
- Misses      10120    10128    +8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jetm

jetm commented Jun 23, 2026

Copy link
Copy Markdown
Author

Superseded by #2750, which combines this with the rest of the OpenEmbedded/Yocto distributed-compile fixes into a single series. Closing in favor of that PR.

@jetm jetm closed this Jun 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants