Skip to content

kernel: GKI-wide compile compat + dcache fixes - #31

Merged
maxsteeel merged 9 commits into
maxsteeel:experimental/hooklessfrom
Bouteillepleine:pr/gki-compat-and-dcache
Jul 28, 2026
Merged

kernel: GKI-wide compile compat + dcache fixes#31
maxsteeel merged 9 commits into
maxsteeel:experimental/hooklessfrom
Bouteillepleine:pr/gki-compat-and-dcache

Conversation

@Bouteillepleine

@Bouteillepleine Bouteillepleine commented Jul 27, 2026

Copy link
Copy Markdown

Summary

Kernel changes only: makes nomount.c compile clean across the whole AOSP/GKI range, and fixes two real dcache bugs found on-device.

CI workflow removed as requested (@maxsteeel) — the compile matrix that was originally the second commit here has been reverted out, so this PR now touches kernel/src/nomount.{c,h} and nothing else. I'll open it as its own PR once #23 is merged, per your plan.

Verified: 10/10 green across 4.9, 4.14, 4.19, 5.4, 5.10, 5.15, 6.1, 6.6, 6.12 and 6.18, and the engine is device-tested on android16-6.12 (OnePlus 15). That matrix still runs in my fork, so these commits stay validated across the range without adding any workflow here.

Updated after @maxsteeel's commits on this branch. The dentry-ops helper was refactored from my NM_SET_DOPS() macro into nm_install_dentry_ops() in nomount.h; the flag semantics are unchanged (clear every DCACHE_OP_*, set d_op, then set only DCACHE_OP_REVALIDATE), and no raw d_op = assignment bypasses it. The wording below uses the new name. The matrix re-ran on your 17f2ab9 and was 10/10 green, so the refactor is validated across the whole range too.

Kernel fixes

Version-boundary compile breaks (each broke a specific kernel that the current unconditional calls don't handle):

  • 4.9: getattr is 3-arg (vfsmount, dentry, kstat) and vfs_getattr_nosec is 2-arg → guarded to <4.11.
  • d_revalidate: the 4-arg form is guarded at >=6.13, not >=6.11android16-6.12 and 6.6 are still 2-arg; only 6.18 is 4-arg. The old >=6.11 guard broke 6.12.
  • 6.18: vm_area_desc.file is const in nm_mmap_prepare → assign via a cast.
  • pre-4.11: nm_alloc_rule used 4-arg statx vfs_getattr + STATX_* → 2-arg form.
  • pre-4.11: idr_remove() returns void → probe with idr_find() first.

dcache consistency:

  • nm_install_dentry_ops() (was NM_SET_DOPS()) — when d_op = &nm_dops is set on a dentry whose superblock is an overlayfs (its s_d_op carries DCACHE_OP_* flags), the stale op-flags made the VFS call NULL ops → OOPS on readdir / ... Now every d_op site clears all DCACHE_OP_* and sets only DCACHE_OP_REVALIDATE.
  • nm_d_revalidate — drop a stale injected child when its parent is no longer hijacked (ghost dentry that survived drop_caches until reboot), and make the verdict UID-aware so a blocked reader's negative dentry can't pollute an unblocked reader.
  • nm_dir_lookup — resolve a virtual-dir parent's dir_node from i_private, so a new multi-child subtree (e.g. app native libs under a synthesized dir) is served correctly.

How the range was verified (no CI added here)

The matrix lives in my fork now, not in this PR. It builds only fs/nomount.o (not a full kernel) per version, which is what makes 10 kernels cheap to check:

  • 4.x use distro gcc-aarch64-linux-gnu (gcc-13); the AOSP gcc-4.9 prebuilt is GCC 4.9, which android-common 4.x rejects (needs ≥5.1).
  • 5.4+ use clang with LLVM=1 + unversioned packages (the versioned LLVM=-NN suffix isn't understood by pre-6.1 kbuild) and CROSS_COMPILE for the target triple.
  • 6.18 from git.kernel.org linux-6.18.y; the rest from android.googlesource.com.

Happy to bring it over as a standalone PR after #23, shaped however you prefer.

Note (not in this PR): hookless RRO

While validating on-device, I confirmed RRO overlays work with pure hookless injection — no overlayfs mount: inject the overlay APKs into /product/overlay in post-fs-data; OverlayManagerService scans later in system_server and registers them, idmap2 reads the injected APK (via the mmap_prepare redirect) and builds a valid idmap, and cmd overlay enable works. That makes an RRO-using setup fully mountless. It's a downstream module-side change, but it exercises this engine's mmap/getattr path — happy to share details if useful upstream.

Note (not in this PR): readdir merge validated at scale, and a downstream walker pitfall

Scaling that RRO work up produced some measurements worth recording for the dcache/readdir paths this PR touches, plus one platform hazard that cost me a lot of bootloops to diagnose.

Corrected after @maxsteeel's review: the partition-root behaviour below is intended — NoMount is bind-mount semantics, injected file by file, so nm add /product <backing> replacing all of /product is exactly right for the rule as written. The bug was entirely in my walker for creating that rule by accident; the engine did what it was told. I've withdrawn my earlier suggestion that nm add should reject partition-root targets — it would forbid a legitimate deliberate whole-partition replacement to paper over a downstream mistake. (Directory-level merging is a separate feature @maxsteeel is exploring as an explicit --merge mode; nothing here asks for it.)

The engine's readdir merge is correct. Injecting into a populated stock directory (/product/overlay, 78 real entries) and listing it:

injected entries visible stock preserved
1 79 yes
5 83 yes
10 88 yes
20 98 yes
139 217 yes

Real entries and injected children are both emitted, deduped by name, across the multi-getdents sequence in nomount_hijacked_iterate_dir — no masking at any scale. (This is the per-file bind-mount behaviour working as designed, not directory merging.)

The pitfall is a walker that emits a bare partition root as a target. nm add /product <backing> redirects the whole partition, so every stock entry under it disappears — correct for the rule, but not what my module meant to ask for. On OnePlus/Oppo the consequence is specifically fatal at boot: with the stock /product/overlay/OplusGmsConfigOverlayCommon replaced, zygote's OverlayConfig preloads the stock twin at /my_product/cust/<region>/overlay/… instead, and /my_product is not in zygote's FD allowlist — so FileDescriptorInfo::CreateFromFd raises

JNI FatalError: (system_server) Not allowlisted (64):
  /my_product/cust/ROW/overlay/OplusGmsConfigOverlayCommon/OplusGmsConfigOverlayCommon.apk

at the first forkSystemServer (before system_server or OMS runs) → SIGABRT → bootloop, independent of how many overlays were injected.

Upstream's walker is not affected, for a reason worth keeping: module/metamount.sh uses find -L, so a module's system/<partition> symlink is followed and descended rather than emitted as an entry, and every emitted path is at least two components deep. Two things reintroduce the hazard downstream — mapping system/product/product (SAR aliases), and any walker whose file-type check does not follow symlinks (e.g. Rust's fs::FileType, which sees system/product -> ../product as "not a directory" and injects it as a leaf). Both applied to my module, and the fix was simply to skip targets that resolve to a single-component path.

So this is a note for module authors rather than a request for an engine change: keep partition-root paths out of what you feed nm add, and avoid /my_product-style non-allowlisted partitions as injection targets for anything zygote may preload — that one is a platform constraint no walker can work around.


Fixes were developed against experimental/hookless and rebased onto bf285ba. No upstream feature is dropped — the diff is our additions plus the before side of the lines they modify.

Verified compiling clean across the full GKI/AOSP range (4.9, 4.14, 4.19, 5.4,
5.10, 5.15, 6.1, 6.6, 6.12, 6.18) via a per-version compile matrix, and
device-tested on android16-6.12 (OnePlus 15).

Version-boundary compile fixes (were breaking specific kernels):
- 4.9: getattr is 3-arg (vfsmount,dentry,kstat) and vfs_getattr_nosec is 2-arg;
  guarded to <4.11 (upstream called the 4-arg statx forms unconditionally).
- d_revalidate: 4-arg form guarded at >=6.13, not >=6.11 — android16-6.12 (and
  6.6) are still 2-arg; only 6.18 is 4-arg, so the old >=6.11 guard broke 6.12.
- 6.18: vm_area_desc.file is const in nm_mmap_prepare -> assign via a cast.
- pre-4.11: nm_alloc_rule used the 4-arg statx vfs_getattr + STATX_* -> 2-arg.
- pre-4.11: idr_remove() returns void -> probe with idr_find() first.

dcache consistency:
- NM_SET_DOPS(): when we set d_op=&nm_dops on a dentry whose superblock is an
  overlayfs (its s_d_op carries DCACHE_OP_* flags), the stale op-flags made the
  VFS call NULL ops -> OOPS on readdir/`..`. Clear all DCACHE_OP_* and set only
  DCACHE_OP_REVALIDATE at every d_op assignment site.
- nm_d_revalidate: drop a stale injected child when its parent is no longer
  hijacked (ghost dentry survived drop_caches until reboot), and make the result
  UID-aware so a blocked reader's negative dentry can't pollute an unblocked one.
- nm_dir_lookup: resolve a virtual-dir parent's dir_node from i_private so a new
  multi-child subtree (e.g. app libs under a synthesized dir) is served correctly.
Compile-tests fs/nomount.o against the canonical AOSP/GKI tree for every supported
kernel (4.9, 4.14, 4.19, 5.4, 5.10, 5.15, 6.1, 6.6, 6.12, 6.18) on push and on PRs
touching kernel/src or kernel/patches. Catches version-boundary API breakage (the
class of bug the dcache/compat commit fixes) before it ships.

- 4.x use the distro gcc-aarch64-linux-gnu (gcc-13); the AOSP gcc-4.9 prebuilt is
  GCC 4.9 which android-common 4.x rejects (needs >=5.1).
- 5.4+ use clang with LLVM=1 + unversioned packages (the versioned LLVM=-NN suffix
  isn't understood by pre-6.1 kbuild) and CROSS_COMPILE for the target triple.
- 6.18 source is git.kernel.org linux-6.18.y; the rest are android.googlesource.com.
- ccache + kernel-clone caching keep re-runs fast; a summary job renders a pass grid.
@maxsteeel

Copy link
Copy Markdown
Owner

Hi, personally I don't like these workflow, you can remove or revert it?
About the changes in nomount.c, i'll make some changes

@maxsteeel

Copy link
Copy Markdown
Owner

Could you remove the workflow from this PR and add it to another PR? Since I don't want to add it for now, but later when the PR #23 is merged, I have plans to add certain workflows, and I could merge your workflow.

@Bouteillepleine
Bouteillepleine marked this pull request as ready for review July 27, 2026 23:08
@maxsteeel

Copy link
Copy Markdown
Owner

Note (not in this PR): readdir merge validated at scale, and a userspace trap worth knowing

Scaling that RRO work up surfaced a platform hazard that is easy to hit from userspace and looks exactly like a kernel bug. Documenting it because the diagnosis cost a lot of bootloops, and because the measurements are a useful datapoint for the dcache/readdir paths this PR touches.

The engine's readdir merge is correct. Injecting into a populated stock directory (/product/overlay, 78 real entries) and listing it:

injected entries visible stock preserved
1 79 yes
5 83 yes
10 88 yes
20 98 yes
139 217 yes
Real entries and injected children are both emitted, deduped by name, across the multi-getdents sequence in nomount_hijacked_iterate_dir — no masking at any scale.

The trap is a rule whose target is a bare partition root. nm add /product <backing> redirects the whole partition, so every stock entry under it disappears from the merged view — correct behaviour for the rule as written, but never what a module means. On OnePlus/Oppo that is specifically fatal at boot: with the stock /product/overlay/OplusGmsConfigOverlayCommon masked, zygote's OverlayConfig preloads the stock twin at /my_product/cust/<region>/overlay/… instead, and /my_product is not in zygote's FD allowlist — so FileDescriptorInfo::CreateFromFd raises

JNI FatalError: (system_server) Not allowlisted (64):
  /my_product/cust/ROW/overlay/OplusGmsConfigOverlayCommon/OplusGmsConfigOverlayCommon.apk

at the first forkSystemServer (before system_server or OMS runs) → SIGABRT → bootloop, independent of how many overlays were injected.

Upstream's walker is not affected, for a reason worth keeping: module/metamount.sh uses find -L, so a module's system/<partition> symlink is followed and descended rather than emitted as an entry, and every emitted path is at least two components deep. Two things reintroduce the hazard downstream — mapping system/product/product (SAR aliases), and any walker whose file-type check does not follow symlinks (e.g. Rust's fs::FileType, which sees system/product -> ../product as "not a directory" and injects it as a leaf). Both applied to my module, and the fix was simply to skip targets that resolve to a single-component path.

Two suggestions, take or leave: rejecting partition-root targets in nm add itself would make the failure impossible from any walker, and /my_product-style non-allowlisted partitions are worth avoiding as injection targets for anything zygote may preload.

I just saw the change you made in the PR description, and well, it's the expected change. NoMount currently works just like bind mount, so that observation is completely normal. I am also working on an OverlayFS-style system where I can merge different folders (like nm add /system /data/adb/modules/example/system --merge), but it is still in a very early stage of development, and I don't know if I can carry it out without complicating the current architecture. So for now, it will remain as is, file-by-file injections in the style of bind mount.

@Bouteillepleine

Copy link
Copy Markdown
Author

Agreed on all counts — bind-mount semantics make the partition-root behaviour correct, and the fault was my walker emitting that target. I've corrected the note in the description and dropped the nm add suggestion.

On --merge, one data point in case it helps you decide whether it's worth the architectural cost: RRO doesn't need it. Per-file injection already produces a correctly merged directory, and the whole OMS path accepts the result.

Concretely, on a OnePlus 15 (android16-6.12) with a 139-overlay RRO module (OxygenCustomizer) injected file-by-file into /product/overlay:

  • listing shows 217 entries (78 stock + 139 injected), stock entries intact
  • idmap2 reads the injected APKs through the mmap_prepare redirect and builds valid idmaps
  • OMS registers all 139 and cmd overlay enable works
  • 0 mounts — the module is fully mountless, which is the whole reason I wanted this path

Cost of doing it per-file at that scale: the full pass (clear + re-inject 171 rules) takes ~70 ms, and the rules occupy roughly 60 KB of slab (nm_inodes 448 + nm_dirs 512 objects at 64 B). So the O(files) rule count isn't a practical problem anywhere near this size — the case for --merge would be modules an order of magnitude larger, not RRO.

If you do build it, two things I'd pre-decide, both from the same ambiguity that bit me: keeping replace as the default for a directory target (an explicit --merge flag, as you wrote it, does exactly that), and defining precedence when two modules merge into the same directory — per-file injection resolves that implicitly by last-writer-wins per path, whereas a directory merge needs a documented order and dedup across backings, not just against injected names in one dir_node.

Not asking for the feature — the current design covers my use case completely.

@maxsteeel maxsteeel changed the title kernel: GKI-wide compile compat + dcache fixes; add compile matrix CI kernel: GKI-wide compile compat + dcache fixes Jul 28, 2026
@maxsteeel
maxsteeel merged commit e4bd08c into maxsteeel:experimental/hookless Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants