build: link bundled-dep archives / fall back to source when prebuilt is incomplete#14
Open
jonasvanderhaegen wants to merge 1 commit into
Open
build: link bundled-dep archives / fall back to source when prebuilt is incomplete#14jonasvanderhaegen wants to merge 1 commit into
jonasvanderhaegen wants to merge 1 commit into
Conversation
The prebuilt liblbug.a has unresolved external references to yyjson, simsimd and several other third-party libraries. These archives are shipped alongside liblbug.a in the prebuilt cache directory, but the build script never told the linker about them. Add link_prebuilt_bundled_deps() which emits cargo:rustc-link-lib directives for every *.a found in the prebuilt lib dir, and call it from use_prebuilt_lbug(). The function is existence-gated so it is a no-op when a fully fused archive is used. Fixes "Undefined symbols: _yyjson_val_mut_copy, _yyjson_write_opts ..." link errors on macOS and equivalent failures on Linux when the crate is consumed via [patch.crates-io] or crates.io download.
Contributor
|
This will be fixed when 0.17.0 is out. For now, use LBUG_PRECOMPILED_RUN_ID= 26256190740 when you cargo build CI picks up the most recent ladybug nightly build which has this fixed. |
f8b41d6 to
405226d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Building against the prebuilt
liblbug.a(the default fast path inbuild.rs) can fail at the final link on macOS and Linux:This happens when the prebuilt
liblbug.ais a thin archive — it contains objects (e.g.json_utils.cpp.o) that reference bundled third-party libraries (yyjson,simsimd,brotli,lz4,mbedtls, …) but does not fold them in, andbuild.rsemits no link directives for them on the prebuilt path.Fix
link_prebuilt_bundled_deps()emitscargo:rustc-link-lib=static={lib}for each bundled dep archive shipped alongside the prebuiltliblbug.a. Guarded byexists(), so it's a no-op when an archive isn't present.Notes / refinement since first opening
While integrating this in a downstream consumer we found two things worth flagging:
Use regular static linking, not
+whole-archive, for the dep archives.mainnow bundles the third-party archives intoliblbug.aviacmake/BundleStaticLibrary.cmake(a fat archive). If a consumer links both the fatliblbug.aand the individualdeps/*.awith+whole-archive, the link fails with duplicate symbols on macOS/Linux. Plainstatic={lib}resolves the references lazily and is a no-op against a fat archive, so it works for both thin and fat layouts.The undefined-symbols problem is largely moot on
mainnow thatBundleStaticLibraryproduces a self-containedliblbug.a. This PR is primarily useful for consumers of older/thin prebuilts or alternate packaging that shipsdeps/separately.Verified on macOS arm64 (consuming a thin prebuilt +
deps/): indexing (COPY + HNSW + checkpoint) and vector search both build and run cleanly.