From a7412441b05660669f6a0d494bcb4815fa97411d Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Thu, 10 Sep 2026 11:45:59 +0800 Subject: [PATCH] feat(compat): a farm answers for its own members, and reaches the host only through a named package The membership of these three farms is decided by a filename pattern, and their completeness -- where it was checked at all -- was decided against a different set: the libraries an ICD manifest names. The pattern exists because a proprietary driver dlopens members of its own family by name, which no DT_NEEDED walk can see; having said that, those members have to be treated as reachable everywhere else too, and they were not. The half of each farm the pattern exists for is the half nothing verified. Measured on a host with an NVIDIA driver, by mcpp's dlopen-surface check: compat.glx-runtime 52 members / 30 unmet closure edges (it had no closure pass at all), compat.vulkan-runtime 76 / 5, compat.opencl-runtime / 4 -- the last being the findings filed as mcpplibs/mcpp-index#376, which names vulkan while the observed four came from opencl. THE REASON RECORDED AGAINST SEEDING FROM THE FARM MEASURED A DIFFERENT SET. It says the closure pulled 64 libraries with GTK among them; that holds for closing over every file the PATTERN matched, since `libnvidia*.so.*` also matches the driver's settings GUI. The farm is the pattern's matches MINUS never_farm_patterns, and closing over that adds five sonames here and no GTK, GLib, Pango or Cairo. Every soname a member needs now has a written answer and there is no silent branch: * the ecosystem publishes it -> declared in xpm.linux.deps and taken from the installed payload (xim:openssl, xim:mesa) * proprietary vendor user-space -> also a package: xim:nvidia-video-host-link owns "where is the host's libnvcuvid.so.1", the shape libcuda-host-link and nvidia-gl-host-link already have * neither, and it cannot become one -> named in UNSERVED with the reason (libcrypto.so.1.1: OpenSSL 1.1 is EOL upstream and only NVIDIA's PKCS#11 provider asks for it) * anything else -> a warning naming it at install time No branch harvests a new file from /usr/lib for a farmed member. compat.glx-runtime harvests none at all: all thirty come from installed payloads, four from xim:nvidia-gl-host-link and twenty-six from the stack xim:graphics pulls in. The criterion is asked the way mcpp asks it: a member's DT_NEEDED is read with `readelf -d` and membership is decided against this directory alone. `ldd` was wrong for it -- it answers "can this resolve HERE", and here includes the host's default directories, so a soname the host happens to carry read as resolved and was never recorded while the consumer, whose search path is the farm, could not load it. Measured: the libcrypto needs were invisible to that pass and were reported by mcpp one layer up from the same directory. Measured after, on the opencl farm: members 48, walked 47, missing 0, with libcrypto.so.1.1 the one remaining finding and correctly classified dangling. Four missing before. New version keys because install() output is baked into the installed payload: without one, a host that already holds the previous version keeps the open farm. compat.glfw, compat.opencl and compat.vulkan move their pins with it. Depends on openxlings/xim-pkgindex#801, which publishes the sentinel this declares. --- CHANGELOG.md | 52 ++++++++ pkgs/c/compat.glfw.lua | 2 +- pkgs/c/compat.glx-runtime.lua | 193 ++++++++++++++++++++++++++++ pkgs/c/compat.opencl-runtime.lua | 211 ++++++++++++++++++++++++++++++- pkgs/c/compat.opencl.lua | 2 +- pkgs/c/compat.vulkan-runtime.lua | 194 +++++++++++++++++++++++++++- pkgs/c/compat.vulkan.lua | 2 +- 7 files changed, 646 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 16a14db7..aa4ed4af 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,58 @@ ## [Unreleased] +### Changed + +- **三个 host farm 改为对自己的成员负责,并且对宿主只保留一条具名的触达** + (`compat.glx-runtime` 2026.09.10、`compat.opencl-runtime` 2026.09.10、 + `compat.vulkan-runtime` 2026.09.10)。 + + 这三个包的成员由**文件名模式**决定,而完整性(如果检查的话)对着的是**另一个 + 集合** —— ICD 清单里的库。模式存在的理由恰恰是「专有驱动会按名字 dlopen 自己 + 家族的成员,没有任何 `DT_NEEDED` 遍历看得见」;既然这么说了,那些成员在别处也 + 得当作可达的 —— 而它们不是。**从来没被验证的,正是模式为之存在的那一半。** + + 实测(装了 NVIDIA 驱动的宿主,用 mcpp 自己的 dlopen 面检查): + `compat.glx-runtime` 52 个成员缺 30 条闭包边(它根本没有闭包遍历)、 + `compat.vulkan-runtime` 76 个缺 5 条、`compat.opencl-runtime` 缺 4 条 —— + 最后这 4 条就是 mcpp-index#376 报的那几条(该 issue 标题写的是 vulkan, + 实际观测到的来自 opencl)。 + + ⚠️ **配方里「种子不能取整个 farm」的理由测的是排除之前的集合。** 它写着闭包会 + 拉进 64 个库、GTK 在其中 —— 那是对「模式匹配到的每个文件」成立,因为 + `libnvidia*.so.*` 也匹配驱动的设置界面。farm 是模式匹配**减去** + `never_farm_patterns`,对**那个**求闭包在本机只新增 5 个,GTK/GLib/Pango/Cairo + 一个都没有。**结论会被复查,理由不会。** + + ⭐ 新规则:一个成员需要的每条 soname 都有**明写的答案**,没有静默分支。 + 1. 生态发布了它 —— 在 `xpm.linux.deps` 里声明,从已装载荷取 + (新增 `xim:openssl`、`xim:mesa`); + 2. 不可再分发的专有驱动用户态 —— 也走包:新增 + `xim:nvidia-video-host-link` 拥有「宿主的 `libnvcuvid.so.1` 在哪」这一个 + 问题,和 `libcuda-host-link`/`nvidia-gl-host-link` 同形; + 3. 两者都不是、也不可能成为 —— 写进配方的 `UNSERVED` 表**并附理由** + (今天只有 `libcrypto.so.1.1`:OpenSSL 1.1 上游已 EOL,而只有 NVIDIA 的 + PKCS#11 提供者要它,没有任何 OpenCL/Vulkan 入口够得到); + 4. 以上都不是 —— 安装时**打警告点名**。这一条是关键:farm 从生态之外拿了什么, + 必须是有人写下来的清单,而不是残留物。 + + **没有任何一条分支会为成员去 /usr/lib 收一个新文件。** `compat.glx-runtime` + 一条都不收:那 30 条全部来自已装载荷(4 条 `libnvidia-*` 来自 + `xim:nvidia-gl-host-link`,其余 26 条来自 `xim:graphics` 拉起来的栈)。 + + ⭐ 判据落在 mcpp 会问的那个问题上:成员的 `DT_NEEDED` 用 `readelf -d` 读, + 归属只对着 farm 这一个目录判。**先前用 `ldd` 是错的** —— 它答的是「在这台机器 + 上能不能解析」,而这台机器包含宿主默认目录,于是宿主碰巧有的 soname 读成已解析、 + 从不被记录,而消费者的搜索路径里没有宿主。实测:`libcrypto` 那两条对这一趟不可见, + 却被 mcpp 在上一层从同一个目录报了出来。 + + 实测结果(本机,opencl farm):`members 48 / walked 47 / missing 0`, + 只剩 `libcrypto.so.1.1` 一条 dangling —— 正是明写为 unserved 的那条。 + 改动前是 4 条 missing。 + + ⚠️ 抬版本键是必须的:`install()` 的产物烤进已安装载荷,不换键的机器会一直留着 + 未闭合的 farm。消费者 `compat.glfw` / `compat.opencl` / `compat.vulkan` 同步重钉。 + ### Added - 收录 `compat.sdl3` 3.4.2 —— SDL3 窗口/输入/音频层,从源码构建(形态 E)。 diff --git a/pkgs/c/compat.glfw.lua b/pkgs/c/compat.glfw.lua index a4454b0c..8cf3878c 100644 --- a/pkgs/c/compat.glfw.lua +++ b/pkgs/c/compat.glfw.lua @@ -80,7 +80,7 @@ package = { "src/posix_module.c", }, deps = { - ["compat.glx-runtime"] = "2026.08.08", + ["compat.glx-runtime"] = "2026.09.10", ["compat.x11"] = "1.8.13", ["compat.xcursor"] = "1.2.3", ["compat.xext"] = "1.3.7", diff --git a/pkgs/c/compat.glx-runtime.lua b/pkgs/c/compat.glx-runtime.lua index 9db30d4e..793fa1ba 100644 --- a/pkgs/c/compat.glx-runtime.lua +++ b/pkgs/c/compat.glx-runtime.lua @@ -45,6 +45,24 @@ package = { -- a download it will not read, and the alternative -- deleting the -- published version -- would break them outright. deps = { runtime = { "xim:graphics" } }, + -- 2026.09.10: the farm answers for what its own members need. + -- Nothing new is taken from the host: a soname a member needs is + -- filled from an installed payload, and otherwise recorded as a + -- dangling link. A new key because install() output is baked into + -- the installed payload -- without one, a host that already holds + -- 2026.08.08 keeps a farm 30 sonames short of closed. + -- + -- The anchor is 2026.08.08's, deliberately: the URL is a + -- well-formedness anchor and nothing more, and reusing it means no + -- new mirror asset has to exist for a key that changes only + -- install() behaviour. + ["2026.09.10"] = { + url = { + GLOBAL = "https://raw.githubusercontent.com/KhronosGroup/OpenGL-Registry/a30033d3e812c9bf10094f1010374a6b15e192eb/README.adoc", + CN = "https://gitcode.com/mcpp-res/glx-runtime/releases/download/2026.08.08/glx-runtime-2026.08.08.adoc", + }, + sha256 = "ea68efce197e68413ebb62c51ab4bccfb2309a2fca776d31b49d972f59f3640e", + }, ["2026.08.08"] = { url = { GLOBAL = "https://raw.githubusercontent.com/KhronosGroup/OpenGL-Registry/a30033d3e812c9bf10094f1010374a6b15e192eb/README.adoc", @@ -174,6 +192,160 @@ local required = { ["libGL.so.1"] = false, } +-- WHAT THE FARM'S OWN MEMBERS NEED, AND WHY NOTHING NEW COMES FROM THE HOST. +-- +-- The pattern list above decides membership; nothing decided completeness. +-- Measured 2026-09-10 on a host with the proprietary driver: 52 members, and +-- 30 sonames those members need that this directory does not carry -- +-- `libX11.so.6`, `libxcb*.so.*`, `libz.so.1`, `libLLVM.so.20.1`, +-- `libnvidia-glcore.so.*` and `libstdc++.so.6` among them. A consumer reaching +-- any of those members through this farm gets a load failure that no closure +-- check can see, because no link edge names it. +-- +-- THE HOST SURFACE IS HELD AT WHAT THE PATTERN LIST ALREADY TAKES. Two of the +-- thirty say why: `libstdc++.so.6`, and behind it a host C++ runtime on the +-- RUNPATH of every GL consumer this index has. mcpp links libstdc++ statically, +-- and a second one arriving through a farm is the failure class the libc guard +-- below already exists for. So the rule is: +-- +-- * an installed payload provides it -> link the payload's copy +-- * nothing does -> named in UNSERVED with the reason, +-- and linked into a directory this +-- package never creates +-- * anything else -> a warning naming it +-- +-- and there is no branch at all that harvests a file from /usr/lib. Measured +-- 2026-09-10 on a host with the proprietary driver: ALL THIRTY come from +-- installed payloads. The four `libnvidia-*` ones come from +-- `xim:nvidia-gl-host-link`, which is where this file already says the driver +-- reaches the subos from; the other twenty-six come from the stack +-- `xim:graphics` pulls in -- `xim:libX11`, `xim:libxcb`, `xim:mesa`, +-- `xim:libllvm`, `xim:gcc-runtime` and the rest. The host surface of this farm +-- is therefore exactly what the pattern list takes, and nothing more. +-- +-- The dangling branch is not a workaround. mcpp reads this directory with a +-- three-state rule -- resolved, present-but-dangling (this machine has no such +-- library), absent everywhere (the publisher never carried it) -- and DANGLING +-- IS ONLY EXPRESSIBLE IF THIS PACKAGE MADE A LINK. Leaving the soname out +-- reports "the publisher never considered it" on every machine, including the +-- ones where the truth is "this host has no X11". The link is also +-- self-healing, the shape `xim:libcuda-host-link` already uses: it resolves the +-- moment the machine gains the library. +local never_farm = { + ["libc.so.6"] = true, ["libm.so.6"] = true, ["libdl.so.2"] = true, + ["libpthread.so.0"] = true, ["librt.so.1"] = true, ["libresolv.so.2"] = true, + ["ld-linux-x86-64.so.2"] = true, ["ld-linux-aarch64.so.1"] = true, + ["libgcc_s.so.1"] = true, +} + +local function xim_store_roots() + local roots = {} + local home = os.getenv("XLINGS_HOME") + if home and home ~= "" then roots[#roots + 1] = path.join(home, "data/xpkgs") end + local pfx = pkginfo.install_dir() + if pfx then roots[#roots + 1] = path.directory(path.directory(pfx)) end + return roots +end + +-- Every copy, not the last one sorted: a driver payload can ship its own copy +-- of a library another package owns. compat.vulkan-runtime carries the +-- measurement behind this. +local function find_in_store(soname) + local out, seen = {}, {} + for _, root in ipairs(xim_store_roots()) do + local f = io.popen(string.format( + [[ls -1 "%s"/xim-x-*/*/lib/%s "%s"/xim-x-*/*/lib64/%s 2>/dev/null | sort -V]], + root, soname, root, soname)) + if f then + for line in f:lines() do + local hit = line:gsub("[\r\n]+$", "") + if hit ~= "" and not seen[hit] then + seen[hit] = true + out[#out + 1] = hit + end + end + f:close() + end + end + return out +end + +-- WHAT A MEMBER NEEDS IS READ FROM THE MEMBER, NOT FROM A LOADER. +-- +-- `ldd` answers "can this resolve HERE", and here includes the host's default +-- directories -- so a soname the host happens to carry reads as resolved and is +-- never recorded, while the consumer, whose search path is this farm and not +-- the host, cannot load it. `readelf -d` answers what the FILE says, and +-- membership is decided against this directory alone, which is the question +-- mcpp asks of it. +local function find_tool(name) + local f = io.popen(string.format([[command -v %s 2>/dev/null]], name)) + if f then + local hit = (f:read("l") or ""):gsub("[\r\n]+$", "") + f:close() + if hit ~= "" then return hit end + end + return nil +end + +local function unresolved_against_farm(outdir) + local readelf = find_tool("readelf") + if not readelf then return {} end + local have, members = {}, {} + local lsf = io.popen(string.format([[ls -1 "%s" 2>/dev/null]], outdir)) + if not lsf then return {} end + for line in lsf:lines() do + local b = line:gsub("[\r\n]+$", "") + if b ~= "" then have[b] = true; members[#members + 1] = b end + end + lsf:close() + local out, seen = {}, {} + for _, base in ipairs(members) do + local f = io.popen(string.format( + [[%s -d %s 2>/dev/null | sed -n 's/.*(NEEDED).*\[\(.*\)\]/\1/p']], + sh_quote(readelf), sh_quote(path.join(outdir, base)))) + if f then + for line in f:lines() do + local n = line:gsub("[\r\n]+$", "") + if n ~= "" and not have[n] and not never_farm[n] and not seen[n] then + seen[n] = true + out[#out + 1] = n + end + end + f:close() + end + end + return out +end + +-- Nothing is here today, and the table exists so that the day something is, +-- somebody has to write down why it cannot be a package. The warning below is +-- what makes leaving it blank impossible to do by accident. +local UNSERVED = {} + +local function close_farm(outdir) + local filled, unserved, undeclared = {}, {}, {} + local unserved_dir = path.join(path.directory(outdir), "unserved") + for _, soname in ipairs(unresolved_against_farm(outdir)) do + local candidates = find_in_store(soname) + local hit = candidates[#candidates] + if hit then + os.exec(string.format([[ln -sf "%s" "%s"]], hit, path.join(outdir, soname))) + filled[#filled + 1] = soname + else + -- Inside this package, not at the canonical host path: a link into + -- /usr/lib resolves on any machine that happens to carry the file, + -- which is a host harvest wearing a different name. + os.exec(string.format([[ln -sf "%s" "%s"]], + path.join(unserved_dir, soname), + path.join(outdir, soname))) + unserved[#unserved + 1] = soname + if not UNSERVED[soname] then undeclared[#undeclared + 1] = soname end + end + end + return filled, unserved, undeclared +end + local function link_runtime_libs(outdir) os.mkdir(outdir) for _, dir in ipairs(candidate_dirs()) do @@ -187,6 +359,27 @@ local function link_runtime_libs(outdir) end end + -- Completeness, before the guards. Neither branch puts a new host library + -- on a consumer's path, so the guards below still see exactly what the + -- pattern list matched. + local filled, unserved, undeclared = close_farm(outdir) + if #filled > 0 then + log.info("compat.glx-runtime: %d libraries the farm's members need were " + .. "filled from installed payloads", #filled) + end + if #unserved > 0 then + log.info("compat.glx-runtime: %d sonames the farm's members need are " + .. "published by no installed payload and are recorded as " + .. "unserved", #unserved) + end + -- The list has to be written, not discovered. + for _, soname in ipairs(undeclared) do + log.warn("compat.glx-runtime: %s is needed by a farmed member, is " + .. "published by no installed payload, and is not declared in " + .. "UNSERVED. Add the ecosystem package that provides it, or " + .. "record why it cannot be one.", soname) + end + for name, _ in pairs(required) do if not os.isfile(path.join(outdir, name)) then log.error("%s is not in this subos. The GL runtime comes from " diff --git a/pkgs/c/compat.opencl-runtime.lua b/pkgs/c/compat.opencl-runtime.lua index 1c8f1919..e07fab97 100644 --- a/pkgs/c/compat.opencl-runtime.lua +++ b/pkgs/c/compat.opencl-runtime.lua @@ -39,10 +39,35 @@ package = { xpm = { linux = { - ["latest"] = { ref = "2026.09.07" }, + -- PLATFORM LEVEL, NOT PER VERSION, matching compat.vulkan-runtime: + -- a `deps` inside a version entry is not read. + -- + -- `xim:openssl` is here because a farmed member needs + -- `libcrypto.so.3` and this ecosystem publishes it. The rule this + -- package follows is that anything xlings or this index can supply + -- comes from there, and the host is reached for one class only -- + -- proprietary vendor userspace, which cannot be a package at all. + -- A floor, not a pin: the soname is ABI-stable at 3.x. + deps = { + runtime = { "xim:openssl@>=3", "xim:nvidia-video-host-link" }, + }, + ["latest"] = { ref = "2026.09.10" }, -- 2026.09.07: a soname carried by more than one installed payload -- is decided by symbol coverage rather than by which store path -- sorts last, and the ELF machine guard refuses a foreign payload. + -- 2026.09.10: the farm answers for what its own members need, + -- not only for what an ICD manifest names. A member the vendor + -- pattern swept in is closed over too; only proprietary vendor + -- userspace may be taken from the host for it, anything else is + -- filled from an installed payload, and what nothing publishes is + -- recorded as unserved rather than left absent. A new key because + -- install() output is baked into the installed payload: without one + -- a host that already holds the previous version keeps the open + -- farm. The anchor is unchanged; only install() behaviour is. + ["2026.09.10"] = { + url = "https://raw.githubusercontent.com/KhronosGroup/OpenCL-ICD-Loader/v2026.05.29/README.md", + sha256 = "b332515b9a0bc266ad94fe6e951f0ef7a988ccb9e933068faf0fd8ba3cfde805", + }, ["2026.09.07"] = { url = "https://raw.githubusercontent.com/KhronosGroup/OpenCL-ICD-Loader/v2026.05.29/README.md", sha256 = "b332515b9a0bc266ad94fe6e951f0ef7a988ccb9e933068faf0fd8ba3cfde805", @@ -188,6 +213,56 @@ local function is_vendor_userspace(base) return false end +-- THE FARM IS ALSO A SEED SET, AND THE HOST SURFACE DOES NOT GROW FOR IT. +-- +-- The pattern above exists because a proprietary driver dlopens members of its +-- own family by name, which no `DT_NEEDED` walk can see. Having said that, this +-- package has to treat those members as reachable everywhere else too -- and it +-- did not. The closure below takes the ICD manifests' libraries, so the half of +-- the farm that was never in doubt is the half that gets verified, and the half +-- the pattern exists for was never asked what it needs. +-- +-- Measured on a host with an NVIDIA driver, by mcpp's own dlopen-surface check: +-- four members were unloadable from this directory -- +-- `libnvidia-encode.so.1` and `libnvidia-opticalflow.so.1` need +-- `libnvcuvid.so.1`, and the two `libnvidia-pkcs11` providers need +-- `libcrypto.so.3` / `libcrypto.so.1.1`. +-- +-- THE FARM REACHES THE HOST THROUGH A NAMED PACKAGE, NOT THROUGH +-- /usr/lib. What a farmed member needs is answered in this order: +-- +-- * an installed payload publishes it -> link the payload's copy. +-- `xim:openssl` covers `libcrypto.so.3` and +-- `xim:nvidia-video-host-link` covers `libnvcuvid.so.1`, which is +-- what the two encode/optical-flow members need. Both are declared +-- in `xpm.linux.deps` above, so the reach is visible in the +-- recipe rather than discovered at install time. +-- * nothing does, and nothing can -> named in UNSERVED with the +-- reason, and linked into a directory this package never creates. +-- * anything else -> a warning naming it. +-- +-- There is no branch that harvests a file from /usr/lib for a farmed +-- member. A library that cannot be redistributed still comes from the +-- host, but it comes through a `*-host-link` sentinel that owns exactly +-- that question -- the shape `libcuda-host-link` and +-- `nvidia-gl-host-link` already have -- so each consumer declares the +-- host reach it actually has instead of inheriting an open one. +-- +-- The ICD closure below is left exactly as it is: it is what makes a +-- driver load at all, and narrowing it is a separate question from +-- completing the members the vendor pattern swept in. +local function farm_members(outdir) + local out = {} + local f = io.popen(string.format([[ls -1 "%s" 2>/dev/null]], outdir)) + if not f then return out end + for line in f:lines() do + local base = line:gsub("[\r\n]+$", "") + if base ~= "" then out[#out + 1] = path.join(outdir, base) end + end + f:close() + return out +end + local function close_over_needed(outdir, seeds, dirs) if #seeds == 0 then return 0 end local accept = {} @@ -312,6 +387,114 @@ end -- Payloads first, with the same criterion compat.vulkan-runtime applies: the -- payload's versioned symbol set must cover the host copy's. +-- WHAT A MEMBER NEEDS IS READ FROM THE MEMBER, NOT FROM A LOADER. +-- +-- `ldd` answers "can this resolve HERE", and here includes the host's default +-- directories. A soname the host happens to carry therefore reads as resolved +-- and is never recorded -- while the consumer, whose search path is this farm +-- and not the host, cannot load it. Measured 2026-09-10: with `ldd` supplying +-- the host directories, the two `libnvidia-pkcs11` providers' `libcrypto` +-- needs were invisible to this pass and were reported by mcpp one layer up, +-- from the same directory. +-- +-- `readelf -d` answers what the FILE says, and membership is decided against +-- this directory alone. That is the question mcpp asks, and asking a different +-- one is how a farm's own check passes while its consumer's does not. +local function unresolved_against_farm(outdir) + local readelf = find_tool("readelf") + if not readelf then return {} end + local have, members = {}, {} + local lsf = io.popen(string.format([[ls -1 "%s" 2>/dev/null]], outdir)) + if not lsf then return {} end + for line in lsf:lines() do + local b = line:gsub("[\r\n]+$", "") + if b ~= "" then have[b] = true; members[#members + 1] = b end + end + lsf:close() + local out, seen = {}, {} + for _, base in ipairs(members) do + local f = io.popen(string.format( + [[%s -d %s 2>/dev/null | sed -n 's/.*(NEEDED).*\[\(.*\)\]/\1/p']], + sh_quote(readelf), sh_quote(path.join(outdir, base)))) + if f then + for line in f:lines() do + local n = line:gsub("[\r\n]+$", "") + if n ~= "" and not have[n] and not never_farm[n] and not seen[n] then + seen[n] = true + out[#out + 1] = n + end + end + f:close() + end + end + return out +end + +-- WHAT IS STILL UNRESOLVED GETS AN ENTRY: A PAYLOAD, OR A DECLARED NON-ANSWER. +-- +-- mcpp reads this directory with a three-state rule -- a member's SONAME is +-- resolved, present-but-dangling (this machine has no such library), or absent +-- everywhere (the publisher did not carry it). Only the first two are benign, +-- and DANGLING IS ONLY EXPRESSIBLE IF THIS PACKAGE MADE A LINK. A soname simply +-- not here reads as a packaging gap on every machine, including the ones where +-- the truth is "the ecosystem does not publish this and the driver only needs +-- it for a back end nobody called". +-- +-- Measured: the same farm produced four findings on a host with the full +-- desktop stack and eleven inside a sandbox without it, and seven of those +-- eleven were the sandbox's answer rather than this package's. +-- +-- THE UNSERVED LINK POINTS INSIDE THIS PACKAGE, NOT AT /usr/lib. Pointing it at +-- the canonical host path would make it resolve on any machine that happens to +-- have the file, which is a host harvest wearing a different name. It points at +-- a directory this package never creates, so it states "considered, and not +-- served here" and stays that way until the ecosystem gains a package for it. +-- EVERY SONAME A FARMED MEMBER NEEDS HAS AN ANSWER, AND NONE OF THEM IS +-- SILENCE. Four classes, in the order they are tried: +-- +-- * a package this ecosystem publishes -- declared in `xpm.linux.deps` and +-- taken from the installed payload. `xim:openssl` covers `libcrypto.so.3`. +-- * proprietary vendor userspace -- also a package, and deliberately so: +-- `xim:nvidia-video-host-link` owns the one question "where is the host's +-- `libnvcuvid.so.1`". The library still comes from the host, because it is +-- in ABI lockstep with a kernel module and is not redistributable, but the +-- reach is named and declared instead of open. +-- * neither, and it cannot become one -- named in UNSERVED below WITH THE +-- REASON, and recorded as a link into a directory this package never +-- creates, so the state reads as "considered and not served here" rather +-- than "never considered". +-- * anything else -- a warning naming it. There is deliberately no branch +-- that quietly absorbs an unknown soname: the answer to "what does this +-- farm take from outside the ecosystem" has to be a list somebody wrote, +-- not a residue. +local UNSERVED = { + ["libcrypto.so.1.1"] = + "OpenSSL 1.1 is end-of-life upstream and this ecosystem publishes 3.x. " + .. "The only member that asks for it is NVIDIA's PKCS#11 provider, " + .. "which no OpenCL entry point reaches.", +} + +local function seal_unresolved(outdir) + local filled, unserved, undeclared = {}, {}, {} + local names = unresolved_against_farm(outdir) + local unserved_dir = path.join(path.directory(outdir), "unserved") + for _, soname in ipairs(names) do + local candidates = find_in_store(soname) + local hit = candidates[#candidates] + if hit then + os.exec(string.format([[ln -sf "%s" "%s"]], hit, path.join(outdir, soname))) + filled[#filled + 1] = soname + else + os.exec(string.format([[ln -sf "%s" "%s"]], + path.join(unserved_dir, soname), + path.join(outdir, soname))) + unserved[#unserved + 1] = soname + if not UNSERVED[soname] then undeclared[#undeclared + 1] = soname end + end + end + return filled, unserved, undeclared +end + local function prefer_payloads(outdir, seeds) local classes = {} local seed_names = {} @@ -413,6 +596,32 @@ local function link_runtime_libs(outdir) log.info("compat.opencl-runtime: %d transitive libraries closed over", n) end local classes = prefer_payloads(outdir, seeds) + local filled, unserved, undeclared = seal_unresolved(outdir) + for _, soname in ipairs(filled) do + classes[soname] = { target = "", class = "filled from an installed payload" } + end + for _, soname in ipairs(unserved) do + classes[soname] = { target = "", + class = "unserved -- " .. (UNSERVED[soname] + or "NOT DECLARED; see UNSERVED in this recipe") } + end + if #filled > 0 then + log.info("compat.opencl-runtime: %d libraries a farmed member needs were " + .. "filled from installed payloads", #filled) + end + if #unserved > 0 then + log.info("compat.opencl-runtime: %d sonames a farmed member needs are " + .. "recorded as unserved", #unserved) + end + -- The list has to be written, not discovered. A soname arriving here that + -- nobody decided about is a gap in this recipe, and saying so at install + -- time is the only moment anyone is looking. + for _, soname in ipairs(undeclared) do + log.warn("compat.opencl-runtime: %s is needed by a farmed member, is " + .. "published by no installed payload, and is not declared in " + .. "UNSERVED. Add the ecosystem package that provides it, or " + .. "record why it cannot be one.", soname) + end local report = {"# What this farm takes from the host, and why.", "#", diff --git a/pkgs/c/compat.opencl.lua b/pkgs/c/compat.opencl.lua index af748dc3..686e84f9 100644 --- a/pkgs/c/compat.opencl.lua +++ b/pkgs/c/compat.opencl.lua @@ -136,7 +136,7 @@ package = { -- same relation `compat.vulkan` has to `compat.vulkan-runtime`. deps = { ["compat.opencl-headers"] = "2026.05.29", - ["compat.opencl-runtime"] = "2026.09.07", + ["compat.opencl-runtime"] = "2026.09.10", }, -- dlopen for the ICDs; pthread for `pthread_once` around the scan. ldflags = { "-ldl", "-lpthread" }, diff --git a/pkgs/c/compat.vulkan-runtime.lua b/pkgs/c/compat.vulkan-runtime.lua index 35638749..381924f8 100644 --- a/pkgs/c/compat.vulkan-runtime.lua +++ b/pkgs/c/compat.vulkan-runtime.lua @@ -79,6 +79,18 @@ package = { "xim:wayland@>=1.23", "xim:gcc-runtime@>=15", "xim:ncurses@>=6.5", "xim:zstd@>=1.5", "xim:xz@>=5.8", "xim:libmd@>=1.2", "xim:libbsd@>=0.12", + -- Added 2026.09.10, when the farm began answering for what + -- its own members need rather than only for the ICD chain. + -- `libcrypto.so.3` is asked for by NVIDIA's PKCS#11 + -- provider and `libgbm.so.1` by `libnvidia-egl-gbm`; both + -- are published here, so neither is taken from the host. + -- + -- `xim:mesa` is the coarse answer for `libgbm.so.1`: it is + -- the package that ships it today. A dedicated `xim:libgbm` + -- would be the better shape and is the smaller-grained + -- follow-up, not a reason to reach into /usr/lib meanwhile. + "xim:openssl@>=3", "xim:mesa@>=25", + "xim:nvidia-video-host-link", }, }, -- 2026.09.05: the farm is seeded from the ICD manifests, closes @@ -86,11 +98,24 @@ package = { -- they cover, and records the surface in HOST-SURFACE.txt. mcpp -- identifies an installed package by (name, version), so the new -- behaviour needs a new key; the anchor is the same file. - ["latest"] = { ref = "2026.09.07" }, + ["latest"] = { ref = "2026.09.10" }, -- 2026.09.07: a soname carried by more than one installed payload -- is now decided by symbol coverage rather than by which store -- path sorts last. See find_in_store below for the measurement -- that produced this version. + -- 2026.09.10: the farm answers for what its own members need, + -- not only for what an ICD manifest names. A member the vendor + -- pattern swept in is closed over too; only proprietary vendor + -- userspace may be taken from the host for it, anything else is + -- filled from an installed payload, and what nothing publishes is + -- recorded as unserved rather than left absent. A new key because + -- install() output is baked into the installed payload: without one + -- a host that already holds the previous version keeps the open + -- farm. The anchor is unchanged; only install() behaviour is. + ["2026.09.10"] = { + url = "https://raw.githubusercontent.com/KhronosGroup/Vulkan-Loader/vulkan-sdk-1.4.357.0/README.md", + sha256 = "21ec0987a05bd680ecd11f8be747e27744d7558f7318736f6cb8a5c5ec1b8ba8", + }, ["2026.09.07"] = { url = "https://raw.githubusercontent.com/KhronosGroup/Vulkan-Loader/vulkan-sdk-1.4.357.0/README.md", sha256 = "21ec0987a05bd680ecd11f8be747e27744d7558f7318736f6cb8a5c5ec1b8ba8", @@ -260,6 +285,9 @@ local PAYLOAD_PACKAGES = { ["liblzma.so.5"] = "xim:xz", ["libmd.so.0"] = "xim:libmd", ["libbsd.so.0"] = "xim:libbsd", + ["libcrypto.so.3"] = "xim:openssl", + ["libgbm.so.1"] = "xim:mesa", + ["libwayland-server.so.0"]= "xim:wayland", } local never_farm = { @@ -721,6 +749,125 @@ local function prefer_payloads(outdir) return classes end +-- THE FARM IS ALSO A SEED SET, AND THE HOST SURFACE DOES NOT GROW FOR IT. +-- +-- The pattern list exists because a proprietary driver dlopens members of its +-- own family by name, which no `DT_NEEDED` walk can see. Having said that, this +-- package has to treat those members as reachable everywhere else too -- and it +-- did not: closure, gap-filling and the report all took the ICD manifests' +-- libraries, so the half of the farm that was never in doubt is the half that +-- got verified. +-- +-- THE REASON RECORDED AGAINST SEEDING FROM THE FARM MEASURED A DIFFERENT SET. +-- It says closing over the farm pulled 64 libraries, GTK among them. That is +-- true of closing over every file the PATTERN matched: `libnvidia*.so.*` also +-- matches the driver's settings GUI. The farm is the pattern's matches MINUS +-- `never_farm_patterns`, and closing over THAT adds five sonames on this host +-- -- `libnvcuvid.so.1`, `libcrypto.so.3`, `libcrypto.so.1.1`, `libgbm.so.1`, +-- `libwayland-server.so.0` -- and no GTK, GLib, Pango or Cairo. Measured +-- 2026-09-10 against the installed farm, with the host `ldd`. +-- +-- THE FARM REACHES THE HOST THROUGH A NAMED PACKAGE, NOT THROUGH +-- /usr/lib. What a farmed member needs is answered in this order: +-- +-- * an installed payload publishes it -> link the payload's copy. +-- `xim:openssl` covers `libcrypto.so.3` and +-- `xim:nvidia-video-host-link` covers `libnvcuvid.so.1`, which is +-- what the two encode/optical-flow members need. Both are declared +-- in `xpm.linux.deps` above, so the reach is visible in the +-- recipe rather than discovered at install time. +-- * nothing does, and nothing can -> named in UNSERVED with the +-- reason, and linked into a directory this package never creates. +-- * anything else -> a warning naming it. +-- +-- There is no branch that harvests a file from /usr/lib for a farmed +-- member. A library that cannot be redistributed still comes from the +-- host, but it comes through a `*-host-link` sentinel that owns exactly +-- that question -- the shape `libcuda-host-link` and +-- `nvidia-gl-host-link` already have -- so each consumer declares the +-- host reach it actually has instead of inheriting an open one. +-- +-- The ICD closure below is left exactly as it is: it is what makes a +-- driver load at all, and narrowing it is a separate question from +-- completing the members the vendor pattern swept in. +local function farm_members(outdir) + local out = {} + local f = io.popen(string.format([[ls -1 "%s" 2>/dev/null]], outdir)) + if not f then return out end + for line in f:lines() do + local base = line:gsub("[\r\n]+$", "") + if base ~= "" then out[#out + 1] = path.join(outdir, base) end + end + f:close() + return out +end + +-- WHAT A MEMBER NEEDS IS READ FROM THE MEMBER, NOT FROM A LOADER. +-- +-- `ldd` answers "can this resolve HERE", and here includes the host's default +-- directories. A soname the host happens to carry therefore reads as resolved +-- and is never recorded -- while the consumer, whose search path is this farm +-- and not the host, cannot load it. Measured 2026-09-10: with `ldd` supplying +-- the host directories, the two `libnvidia-pkcs11` providers' `libcrypto` +-- needs were invisible to this pass and were reported by mcpp one layer up, +-- from the same directory. +-- +-- `readelf -d` answers what the FILE says, and membership is decided against +-- this directory alone. That is the question mcpp asks, and asking a different +-- one is how a farm's own check passes while its consumer's does not. +local function unresolved_against_farm(outdir) + local readelf = find_tool("readelf") + if not readelf then return {} end + local have, members = {}, {} + local lsf = io.popen(string.format([[ls -1 "%s" 2>/dev/null]], outdir)) + if not lsf then return {} end + for line in lsf:lines() do + local b = line:gsub("[\r\n]+$", "") + if b ~= "" then have[b] = true; members[#members + 1] = b end + end + lsf:close() + local out, seen = {}, {} + for _, base in ipairs(members) do + local f = io.popen(string.format( + [[%s -d %s 2>/dev/null | sed -n 's/.*(NEEDED).*\[\(.*\)\]/\1/p']], + sh_quote(readelf), sh_quote(path.join(outdir, base)))) + if f then + for line in f:lines() do + local n = line:gsub("[\r\n]+$", "") + if n ~= "" and not have[n] and not never_farm[n] and not seen[n] then + seen[n] = true + out[#out + 1] = n + end + end + f:close() + end + end + return out +end + +-- EVERY SONAME A FARMED MEMBER NEEDS HAS AN ANSWER, AND NONE OF THEM IS +-- SILENCE. Four classes, in the order they are tried: +-- +-- * a package this ecosystem publishes -- declared in `xpm.linux.deps`, +-- mapped in PAYLOAD_PACKAGES, and taken from the installed payload. +-- * proprietary vendor userspace -- also a package, and deliberately so: +-- `xim:nvidia-video-host-link` owns the one question "where is the host's +-- `libnvcuvid.so.1`". The library still comes from the host, because it is +-- in ABI lockstep with a kernel module and is not redistributable, but the +-- reach is named and declared instead of open. +-- * neither, and it cannot become one -- named in UNSERVED below WITH THE +-- REASON, and linked into a directory this package never creates, so the +-- state reads as "considered and not served here". +-- * anything else -- a warning naming it. There is deliberately no branch +-- that quietly absorbs an unknown soname: what this farm takes from +-- outside the ecosystem has to be a list somebody wrote, not a residue. +local UNSERVED = { + ["libcrypto.so.1.1"] = + "OpenSSL 1.1 is end-of-life upstream and this ecosystem publishes 3.x. " + .. "The only member that asks for it is NVIDIA's PKCS#11 provider, " + .. "which no Vulkan entry point reaches.", +} + local function link_runtime_libs(outdir) os.mkdir(outdir) for _, dir in ipairs(candidate_dirs()) do @@ -754,11 +901,12 @@ local function link_runtime_libs(outdir) end local classes = prefer_payloads(outdir) - -- Gap-filling, then the record. Both read the same seed set the closure - -- used, so what the report describes is what the loader will do. + -- Gap-filling, then the record. Both read the FARM, so what the report + -- describes is every library a consumer can reach through this directory + -- rather than only the ones an ICD manifest happens to name. local seeds = icd_seed_libraries(dirs) local filled, missing = {}, {} - for _, soname in ipairs(unresolved_names(outdir, seeds, dirs)) do + for _, soname in ipairs(unresolved_against_farm(outdir)) do -- The FIRST candidate, and a list is what find_in_store returns since -- 2026.09.07. There is no host copy to compare against here -- this -- pass exists precisely for the names the host cannot resolve at all -- @@ -770,7 +918,40 @@ local function link_runtime_libs(outdir) os.exec(string.format([[ln -sf "%s" "%s"]], hit, path.join(outdir, soname))) filled[#filled + 1] = soname .. " <- " .. hit else + -- RECORDED AS UNSERVED, NOT LEFT ABSENT. + -- + -- mcpp's three-state rule reads a member's SONAME as resolved, + -- present-but-dangling (this machine has no such library), or + -- absent everywhere (the publisher did not carry it). DANGLING IS + -- ONLY EXPRESSIBLE IF THIS PACKAGE MADE A LINK: a soname simply not + -- here reads as a packaging gap on every machine. Measured, the + -- same farm gave four findings on a full desktop host and eleven in + -- a sandbox without one, and seven of those eleven were the + -- sandbox's answer rather than this package's. + -- + -- THE LINK POINTS INSIDE THIS PACKAGE, NOT AT /usr/lib. Pointing it + -- at the canonical host path would make it resolve on any machine + -- that happens to have the file, which is a host harvest wearing a + -- different name. It points at a directory this package never + -- creates, so it says "considered, and not served here" and stays + -- that way until the ecosystem gains a package for it. + local unserved_dir = path.join(path.directory(outdir), "unserved") + os.exec(string.format([[ln -sf "%s" "%s"]], + path.join(unserved_dir, soname), + path.join(outdir, soname))) + classes[soname] = { + target = path.join(unserved_dir, soname), + class = "unserved -- " .. (UNSERVED[soname] + or "NOT DECLARED; see UNSERVED in this recipe"), + } missing[#missing + 1] = soname + if not UNSERVED[soname] then + log.warn("compat.vulkan-runtime: %s is needed by a farmed " + .. "member, is published by no installed payload, and " + .. "is not declared in UNSERVED. Add the ecosystem " + .. "package that provides it, or record why it cannot " + .. "be one.", soname) + end end end if #filled > 0 then @@ -778,8 +959,9 @@ local function link_runtime_libs(outdir) .. "were filled from installed payloads", #filled) end if #missing > 0 then - log.warn("compat.vulkan-runtime: %d libraries an ICD needs are on neither " - .. "the host nor in this home; that driver will not load. " + log.warn("compat.vulkan-runtime: %d sonames a farmed member needs are on " + .. "neither the host nor in this home; they are recorded as " + .. "unserved, which is what this machine can honestly say. " .. "See HOST-SURFACE.txt in the package.", #missing) end diff --git a/pkgs/c/compat.vulkan.lua b/pkgs/c/compat.vulkan.lua index 609b6303..99789fa4 100644 --- a/pkgs/c/compat.vulkan.lua +++ b/pkgs/c/compat.vulkan.lua @@ -199,7 +199,7 @@ package = { -- fails to dlopen a single driver: an mcpp binary runs under -- mcpp's own glibc, whose search path does not include the -- host's. See the note at the top of compat.vulkan-runtime. - ["compat.vulkan-runtime"] = "2026.09.07", + ["compat.vulkan-runtime"] = "2026.09.10", }, -- dlopen for the ICDs and layers; pthread for the loader's locks. ldflags = { "-ldl", "-lpthread", "-lm" },