From 34126ab8867b32b8a63bc21869a0c99e496b3451 Mon Sep 17 00:00:00 2001 From: WangYuli Date: Thu, 6 Aug 2026 23:39:26 +0800 Subject: [PATCH 01/13] arm64/iee: mark __create_pgd_mapping_for_iee as __init modpost reports a section mismatch: WARNING: modpost: vmlinux: section mismatch in reference: __create_pgd_mapping_for_iee+0x54 (section: .text.unlikely.) -> __create_pgd_mapping_for_iee_locked (section: .init.text) __create_pgd_mapping_for_iee() lacks a __init annotation, so the compiler places it in a non-init section (.text.unlikely) while it calls __create_pgd_mapping_for_iee_locked(), which lives in .init.text. After init memory is freed, that reference would dangle, hence the mismatch warning. The only caller is __map_memblock_for_iee(), itself __init, reached exclusively from iee_init_mappings() during paging_init(). The call can never happen at runtime, so the wrapper belongs in .init.text too. Annotate it __init instead of __ref: the reference is genuinely init-only, and this also lets the wrapper's code be freed by free_initmem(), slightly reducing the runtime memory footprint. Fixes: 76baf5e2e1ad ("HAOC: Add support for AArch64 Isolated Execution Environment(IEE).") Assisted-by: Kimi Code:K3 Signed-off-by: WangYuli --- arch/arm64/kernel/haoc/iee/iee-mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/kernel/haoc/iee/iee-mmu.c b/arch/arm64/kernel/haoc/iee/iee-mmu.c index c7b227907c3f0..b3e73381d3461 100644 --- a/arch/arm64/kernel/haoc/iee/iee-mmu.c +++ b/arch/arm64/kernel/haoc/iee/iee-mmu.c @@ -487,7 +487,7 @@ static void __init __create_pgd_mapping_for_iee_locked(pgd_t *pgdir, phys_addr_t } while (pgdp++, addr = next, addr != end); } -static void __create_pgd_mapping_for_iee(pgd_t *pgdir, phys_addr_t phys, +static void __init __create_pgd_mapping_for_iee(pgd_t *pgdir, phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, phys_addr_t (*pgtable_alloc)(int), From 64589da9f81cfc059de572c50a05b78b2732b921 Mon Sep 17 00:00:00 2001 From: WangYuli Date: Thu, 6 Aug 2026 23:39:34 +0800 Subject: [PATCH 02/13] arm64/iee: add __init to iee_init_mappings() declaration in iee.h The declaration of iee_init_mappings() in asm/haoc/iee.h lacks the __init annotation present on its definition in iee-mmu.c and on the duplicate declaration in asm/haoc/iee-mmu.h. Keep the annotations consistent so the init-only nature of the function is visible at every declaration site. No functional change. Fixes: 76baf5e2e1ad ("HAOC: Add support for AArch64 Isolated Execution Environment(IEE).") Assisted-by: Kimi Code:K3 Signed-off-by: WangYuli --- arch/arm64/include/asm/haoc/iee.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/haoc/iee.h b/arch/arm64/include/asm/haoc/iee.h index 7137aa186d29a..9e653dd34c92e 100644 --- a/arch/arm64/include/asm/haoc/iee.h +++ b/arch/arm64/include/asm/haoc/iee.h @@ -66,7 +66,7 @@ extern bool haoc_enabled; #define TCR_HPD1 (UL(1) << 42) -void iee_init_mappings(pgd_t *pgdp); +void __init iee_init_mappings(pgd_t *pgdp); void iee_init_post(void); void iee_stack_init(void); void iee_init_tcr(void); From 4a10185a2ea7331b15f4780d289ef589fe7f1d74 Mon Sep 17 00:00:00 2001 From: WangYuli Date: Mon, 10 Aug 2026 17:59:13 +0800 Subject: [PATCH 03/13] HAOC: IEE: Replace indirect function table with a C dispatcher The x86 IEE access gate dispatches IEE operations through iee_funcs[], a plain writable .data function pointer table, using a raw indirect call in hand-written assembly: vmlinux.o: warning: objtool: iee_rw_gate+0x2a: indirect call found in RETPOLINE build In a CONFIG_RETPOLINE=y build every indirect branch must go through the retpoline thunks; the gate's "call *(%rax,%rdi,8)" bypasses them, leaving an unprotected indirect branch (Spectre v2/BTI) inside the entry point of a security mechanism. The table makes this worse: under HAOC's own threat model an attacker with arbitrary kernel write (or ROP into the gate with a controlled %rdi) can redirect or index off the writable table and turn the gate into a call-anywhere gadget that runs with CR0.WP cleared. Replace the table with iee_dispatch(), a C function that switches on the op flag and calls each _iee_*() implementation directly. All callees take at most three register arguments and absorb the flag in their first (unused) parameter, so the existing gate ABI is preserved bit-for-bit while the compiler generates retpoline/objtool/IBT compliant code. This also removes the writable dispatch table from the kernel image entirely. An unknown flag used to dereference past the table (a NULL entry or adjacent .data) and jump to a wild pointer. Fail closed instead: panic() unconditionally. BUG() was considered and rejected: with CONFIG_PANIC_ON_OOPS unset and nearly all gate callers in process context, die() would merely kill the current task, leaving the CPU running with CR0.WP cleared, interrupts disabled and the IEE stack leaked -- exactly the state IEE_SIP exists to prevent. Fixes: a8edec335283 ("HAOC: Add support for x86 Isolated Execution Environment") Assisted-by: Kimi Code:K3 Signed-off-by: WangYuli --- arch/x86/include/asm/haoc/haoc.h | 4 + arch/x86/kernel/haoc/haoc.c | 221 ++++++++++++++++++++++------ arch/x86/kernel/haoc/iee/iee-gate.S | 5 +- 3 files changed, 183 insertions(+), 47 deletions(-) diff --git a/arch/x86/include/asm/haoc/haoc.h b/arch/x86/include/asm/haoc/haoc.h index ef665161259de..53526217cb5f6 100644 --- a/arch/x86/include/asm/haoc/haoc.h +++ b/arch/x86/include/asm/haoc/haoc.h @@ -75,4 +75,8 @@ void _iee_set_cred_security(unsigned long __unused, struct cred *cred,void *secu void _iee_set_cred_rcu(unsigned long __unused, struct cred *cred, struct rcu_head *rcu); void _iee_set_cred_ucounts(unsigned long __unused, struct cred *cred, struct ucounts *ucounts); #endif + +/* Called from iee_rw_gate (asm) to dispatch IEE ops to the _iee_*() functions. */ +unsigned long iee_dispatch(int flag, unsigned long arg1, + unsigned long arg2, unsigned long arg3); #endif diff --git a/arch/x86/kernel/haoc/haoc.c b/arch/x86/kernel/haoc/haoc.c index b9576af3fc913..8fbc3268b4884 100644 --- a/arch/x86/kernel/haoc/haoc.c +++ b/arch/x86/kernel/haoc/haoc.c @@ -7,53 +7,186 @@ * Hu Bing */ +#include +#include #include +#include -typedef void (*iee_func)(void); -iee_func iee_funcs[] = { - (iee_func)_iee_memcpy, - (iee_func)_iee_memset, - (iee_func)_iee_set_freeptr, - (iee_func)_iee_test_and_clear_bit, +/* + * IEE function dispatcher. + * + * Called from iee_rw_gate (asm) on the per-cpu IEE stack with interrupts + * disabled and CR0.WP cleared. The first argument of every _iee_*() + * function absorbs @flag, keeping the register convention of the old + * indirect function table. + * + * Direct calls only: no indirect branch (retpoline/objtool clean) and no + * dispatch table in writable memory. + */ +notrace unsigned long iee_dispatch(int flag, unsigned long arg1, + unsigned long arg2, unsigned long arg3) +{ + switch (flag) { + case IEE_OP_MEMCPY: + _iee_memcpy(flag, (void *)arg1, (void *)arg2, arg3); + break; + case IEE_OP_MEMSET: + _iee_memset(flag, (void *)arg1, (int)arg2, arg3); + break; + case IEE_OP_SET_FREEPTR: + _iee_set_freeptr(flag, (void **)arg1, (void *)arg2); + break; + case IEE_OP_TEST_CLEAR_BIT: + return _iee_test_and_clear_bit(flag, (long)arg1, + (unsigned long *)arg2); #ifdef CONFIG_IEE_PTRP - (iee_func)_iee_set_token_pgd, - (iee_func)_iee_invalidate_token, - (iee_func)_iee_validate_token, + case IEE_OP_SET_TOKEN_PGD: + _iee_set_token_pgd(flag, (struct task_struct *)arg1, + (pgd_t *)arg2); + break; + case IEE_OP_INVALIDATE_TOKEN: + _iee_invalidate_token(flag, (struct task_struct *)arg1); + break; + case IEE_OP_VALIDATE_TOKEN: + _iee_validate_token(flag, (struct task_struct *)arg1); + break; #endif #ifdef CONFIG_CREDP - (iee_func)_iee_copy_cred, - (iee_func)_iee_copy_kernel_cred, - (iee_func)_iee_init_copied_cred, - (iee_func)_iee_commit_creds, - (iee_func)_iee_abort_cred, - (iee_func)_iee_set_cred_uid, - (iee_func)_iee_set_cred_gid, - (iee_func)_iee_set_cred_suid, - (iee_func)_iee_set_cred_sgid, - (iee_func)_iee_set_cred_euid, - (iee_func)_iee_set_cred_egid, - (iee_func)_iee_set_cred_fsuid, - (iee_func)_iee_set_cred_fsgid, - (iee_func)_iee_set_cred_user, - (iee_func)_iee_set_cred_user_ns, - (iee_func)_iee_set_cred_group_info, - (iee_func)_iee_set_cred_securebits, - (iee_func)_iee_set_cred_cap_inheritable, - (iee_func)_iee_set_cred_cap_permitted, - (iee_func)_iee_set_cred_cap_effective, - (iee_func)_iee_set_cred_cap_bset, - (iee_func)_iee_set_cred_cap_ambient, - (iee_func)_iee_set_cred_jit_keyring, - (iee_func)_iee_set_cred_session_keyring, - (iee_func)_iee_set_cred_process_keyring, - (iee_func)_iee_set_cred_thread_keyring, - (iee_func)_iee_set_cred_request_key_auth, - (iee_func)_iee_set_cred_non_rcu, - (iee_func)_iee_set_cred_atomic_set_usage, - (iee_func)_iee_set_cred_atomic_op_usage, - (iee_func)_iee_set_cred_security, - (iee_func)_iee_set_cred_rcu, - (iee_func)_iee_set_cred_ucounts, + case IEE_OP_COPY_CRED: + _iee_copy_cred(flag, (struct cred *)arg1); + break; + case IEE_OP_COPY_KERNEL_CRED: + _iee_copy_kernel_cred(flag, (const struct cred *)arg1, + (struct cred *)arg2); + break; + case IEE_OP_INIT_COPIED_CRED: + _iee_init_copied_cred(flag, (struct task_struct *)arg1, + (struct cred *)arg2); + break; + case IEE_OP_COMMIT_CRED: + _iee_commit_creds(flag, (const struct cred *)arg1); + break; + case IEE_OP_ABORT_CRED: + _iee_abort_cred(flag, (const struct cred *)arg1); + break; + case IEE_OP_SET_CRED_UID: + _iee_set_cred_uid(flag, (struct cred *)arg1, + KUIDT_INIT((uid_t)arg2)); + break; + case IEE_OP_SET_CRED_GID: + _iee_set_cred_gid(flag, (struct cred *)arg1, + KGIDT_INIT((gid_t)arg2)); + break; + case IEE_OP_SET_CRED_SUID: + _iee_set_cred_suid(flag, (struct cred *)arg1, + KUIDT_INIT((uid_t)arg2)); + break; + case IEE_OP_SET_CRED_SGID: + _iee_set_cred_sgid(flag, (struct cred *)arg1, + KGIDT_INIT((gid_t)arg2)); + break; + case IEE_OP_SET_CRED_EUID: + _iee_set_cred_euid(flag, (struct cred *)arg1, + KUIDT_INIT((uid_t)arg2)); + break; + case IEE_OP_SET_CRED_EGID: + _iee_set_cred_egid(flag, (struct cred *)arg1, + KGIDT_INIT((gid_t)arg2)); + break; + case IEE_OP_SET_CRED_FSUID: + _iee_set_cred_fsuid(flag, (struct cred *)arg1, + KUIDT_INIT((uid_t)arg2)); + break; + case IEE_OP_SET_CRED_FSGID: + _iee_set_cred_fsgid(flag, (struct cred *)arg1, + KGIDT_INIT((gid_t)arg2)); + break; + case IEE_OP_SET_CRED_USER: + _iee_set_cred_user(flag, (struct cred *)arg1, + (struct user_struct *)arg2); + break; + case IEE_OP_SET_CRED_USER_NS: + _iee_set_cred_user_ns(flag, (struct cred *)arg1, + (struct user_namespace *)arg2); + break; + case IEE_OP_SET_CRED_GROUP_INFO: + _iee_set_cred_group_info(flag, (struct cred *)arg1, + (struct group_info *)arg2); + break; + case IEE_OP_SET_CRED_SECUREBITS: + _iee_set_cred_securebits(flag, (struct cred *)arg1, + (unsigned int)arg2); + break; + case IEE_OP_SET_CRED_CAP_INHER: + _iee_set_cred_cap_inheritable(flag, (struct cred *)arg1, + (kernel_cap_t){ .val = arg2 }); + break; + case IEE_OP_SET_CRED_CAP_PERM: + _iee_set_cred_cap_permitted(flag, (struct cred *)arg1, + (kernel_cap_t){ .val = arg2 }); + break; + case IEE_OP_SET_CRED_CAP_EFFECT: + _iee_set_cred_cap_effective(flag, (struct cred *)arg1, + (kernel_cap_t){ .val = arg2 }); + break; + case IEE_OP_SET_CRED_CAP_BSET: + _iee_set_cred_cap_bset(flag, (struct cred *)arg1, + (kernel_cap_t){ .val = arg2 }); + break; + case IEE_OP_SET_CRED_CAP_AMBIENT: + _iee_set_cred_cap_ambient(flag, (struct cred *)arg1, + (kernel_cap_t){ .val = arg2 }); + break; + case IEE_OP_SET_CRED_JIT_KEYRING: + _iee_set_cred_jit_keyring(flag, (struct cred *)arg1, + (unsigned char)arg2); + break; + case IEE_OP_SET_CRED_SESS_KEYRING: + _iee_set_cred_session_keyring(flag, (struct cred *)arg1, + (struct key *)arg2); + break; + case IEE_OP_SET_CRED_PROC_KEYRING: + _iee_set_cred_process_keyring(flag, (struct cred *)arg1, + (struct key *)arg2); + break; + case IEE_OP_SET_CRED_THREAD_KEYRING: + _iee_set_cred_thread_keyring(flag, (struct cred *)arg1, + (struct key *)arg2); + break; + case IEE_OP_SET_CRED_REQ_KEYRING: + _iee_set_cred_request_key_auth(flag, (struct cred *)arg1, + (struct key *)arg2); + break; + case IEE_OP_SET_CRED_NON_RCU: + _iee_set_cred_non_rcu(flag, (struct cred *)arg1, (int)arg2); + break; + case IEE_OP_SET_CRED_ATSET_USAGE: + _iee_set_cred_atomic_set_usage(flag, (struct cred *)arg1, + (int)arg2); + break; + case IEE_OP_SET_CRED_ATOP_USAGE: + return _iee_set_cred_atomic_op_usage(flag, (struct cred *)arg1, + (int)arg2, (int)arg3); + case IEE_OP_SET_CRED_SECURITY: + _iee_set_cred_security(flag, (struct cred *)arg1, (void *)arg2); + break; + case IEE_OP_SET_CRED_RCU: + _iee_set_cred_rcu(flag, (struct cred *)arg1, + (struct rcu_head *)arg2); + break; + case IEE_OP_SET_CRED_UCOUNTS: + _iee_set_cred_ucounts(flag, (struct cred *)arg1, + (struct ucounts *)arg2); + break; #endif - NULL -}; + default: + /* + * Unreachable in practice: all flags are compile-time + * constants in the iee_*() wrappers. An unknown flag means + * memory corruption or ROP into the gate; halt the machine + * instead of continuing with CR0.WP cleared on this CPU. + */ + panic("iee_dispatch: unknown flag %d\n", flag); + } + return 0; +} diff --git a/arch/x86/kernel/haoc/iee/iee-gate.S b/arch/x86/kernel/haoc/iee/iee-gate.S index 5497c3788b026..02142848bc520 100644 --- a/arch/x86/kernel/haoc/iee/iee-gate.S +++ b/arch/x86/kernel/haoc/iee/iee-gate.S @@ -55,9 +55,8 @@ SYM_FUNC_START(iee_rw_gate) movq %rsp, %r12 movq PER_CPU_VAR(iee_stacks) + IEE_STACK, %rsp - /* call iee func */ - leaq iee_funcs(%rip), %rax - call *(%rax, %rdi, 8) + /* direct dispatch: no indirect call, no writable func table */ + call iee_dispatch /* switch to kernel stack */ movq %r12, %rsp From bdca584f936f9f1d7993fd3d8565cc421491b9b2 Mon Sep 17 00:00:00 2001 From: WangYuli Date: Mon, 10 Aug 2026 18:00:25 +0800 Subject: [PATCH 04/13] HAOC: IEE: Make x86 gates frame-pointer and objtool clean The hand-written x86 IEE gates are declared as SYM_FUNC but never set up a frame pointer, and they switch to the per-cpu IEE stack in the middle of the function before calling into C code: vmlinux.o: warning: objtool: iee_rwx_gate+0x38: call without frame pointer save/setup With CONFIG_STACK_VALIDATION=y and CONFIG_FRAME_POINTER=y, objtool requires a valid stack frame before any call instruction. The same defect exists in iee_rw_gate and surfaces as soon as its dispatch call becomes a direct one. Establish a real %rbp frame chain in both gates so the frame-pointer unwinder can walk across the stack switch at runtime: the saved %rbp value links the callee frame back to the caller frame regardless of which stack each frame lives on. objtool itself provably cannot model a mid-function switch to a per-cpu stack: once the CFA is %rbp-anchored, "mov %reg, %rsp" degrades it to CFI_UNDEFINED, and the only modelled stack-switch idiom (the "stack swizzle") yields CFI_SP_INDIRECT, which also fails has_valid_stack_frame(). Follow the x86 entry code convention for such code: declare the gates SYM_CODE_* (STT_NOTYPE) to exempt them from function stack validation while keeping the manually maintained frame chain. SYM_CODE_START does not emit ENDBR64. Add it explicitly to both gates; iee_rw_gate is exported, and CONFIG_X86_KERNEL_IBT=y makes objtool flag data relocations (from the __ksymtab entry) to non-ENDBR targets: vmlinux.o: warning: objtool: .export_symbol+0x1348: data relocation to !ENDBR: iee_rw_gate+0x0 Fixes: a8edec335283 ("HAOC: Add support for x86 Isolated Execution Environment") Fixes: 7d1832b64006 ("Haoc: Add support for x86 Sensitive Instruction Protection(IEE_SIP)") Assisted-by: Kimi Code:K3 Signed-off-by: WangYuli --- arch/x86/kernel/haoc/iee/iee-gate.S | 44 +++++++++++++++++++++++------ 1 file changed, 36 insertions(+), 8 deletions(-) diff --git a/arch/x86/kernel/haoc/iee/iee-gate.S b/arch/x86/kernel/haoc/iee/iee-gate.S index 02142848bc520..62a016555bdd2 100644 --- a/arch/x86/kernel/haoc/iee/iee-gate.S +++ b/arch/x86/kernel/haoc/iee/iee-gate.S @@ -9,6 +9,7 @@ #include #include +#include #include #include #include @@ -39,9 +40,21 @@ /* * IEE memory access gate. * Kernel calls the gate to modify IEE-protected memory. + * + * The gates switch to the per-cpu IEE stack in the middle of the + * function. objtool cannot model that: %rsp is untraceable after the + * per-cpu stack load, so no provable frame pointer state exists at the + * call sites (the only modelled stack-switch idiom yields CFI_SP_INDIRECT, + * which also fails has_valid_stack_frame()). Declare the gates as code + * (not function) symbols to exempt them from stack validation, the same + * way x86 entry code handles mid-function stack switches, and maintain a + * real %rbp frame chain manually so the frame pointer unwinder still + * works at runtime. */ -SYM_FUNC_START(iee_rw_gate) +SYM_CODE_START(iee_rw_gate) + ENDBR + /* save Interrupt flag */ pushfq /* close irq*/ @@ -51,6 +64,10 @@ SYM_FUNC_START(iee_rw_gate) DISABLE_WP r12 + /* set up a real frame for the frame pointer unwinder */ + pushq %rbp + movq %rsp, %rbp + /* switch to iee stack */ movq %rsp, %r12 movq PER_CPU_VAR(iee_stacks) + IEE_STACK, %rsp @@ -61,6 +78,8 @@ SYM_FUNC_START(iee_rw_gate) /* switch to kernel stack */ movq %r12, %rsp + popq %rbp + ENABLE_WP r12 popq %r12 @@ -69,11 +88,13 @@ SYM_FUNC_START(iee_rw_gate) popfq jmp __x86_return_thunk /* ret */ -SYM_FUNC_END(iee_rw_gate) +SYM_CODE_END(iee_rw_gate) EXPORT_SYMBOL(iee_rw_gate) #ifdef CONFIG_IEE_SIP -SYM_FUNC_START(iee_rwx_gate) +SYM_CODE_START(iee_rwx_gate) + ENDBR + pushq %r12 /* save Interrupt flag*/ @@ -87,19 +108,26 @@ SYM_FUNC_START(iee_rwx_gate) movq %rax, %cr4 DISABLE_WP r12 - + + /* set up a real frame for the frame pointer unwinder */ + pushq %rbp + movq %rsp, %rbp + movq %rsp, %r12 /* If iee hasn't been initialized, skip stack switch. */ cmpb $0, iee_init_done(%rip) jz 2f - + /* switch to iee stack */ movq PER_CPU_VAR(iee_stacks) + IEE_STACK, %rsp - + 2: call _iee_si_handler + /* switch to kernel stack. If iee hasn't been initialized, skip switch*/ movq %r12, %rsp - + + popq %rbp + ENABLE_WP r12 /* set SMEP=1 to disable supervisor-mode exec user-mode insn */ @@ -115,5 +143,5 @@ SYM_FUNC_START(iee_rwx_gate) popq %r12 jmp __x86_return_thunk /* ret */ -SYM_FUNC_END(iee_rwx_gate) +SYM_CODE_END(iee_rwx_gate) #endif \ No newline at end of file From 63d556aef7793e38d694b2471de9cf56ddfd289e Mon Sep 17 00:00:00 2001 From: WangYuli Date: Tue, 11 Aug 2026 13:43:59 +0800 Subject: [PATCH 05/13] HAOC: IEE: Fix rwx gate CR4 restore on CPUs without SMEP/SMAP The exit path of iee_rwx_gate unconditionally ORs X86_CR4_SMEP_SMAP into CR4 after running the sensitive-instruction handler: movq %cr4, %rax 1: orq $X86_CR4_SMEP_SMAP, %rax movq %rax, %cr4 andq $X86_CR4_SMEP_SMAP, %rax cmpq $X86_CR4_SMEP_SMAP, %rax jnz 1 With CONFIG_IEE_SIP=y and haoc=1, every native_write_cr4() goes through this gate with no CPU capability check at all. The first call happens early in setup_arch() (init_mem_mapping() -> cr4_set_bits_and_update_boot(X86_CR4_PSE)), so on any CPU without SMAP (or SMEP) the exit write sets an unsupported CR4 bit, which raises #GP(0) on bare metal with interrupts off and kills the boot this early. Under QEMU TCG the unsupported write does not fault but is silently dropped, leaving CR4 in a state the kernel does not expect; the boot then dies slightly later in early_ioremap: [ 0.008249] last_pfn = 0x3ffe0 max_arch_pfn = 0x400000000 [ 0.008743] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs [ 0.008956] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT QEMU exception log from the failing boot (-cpu qemu64, haoc=1, nokaslr; note CR4 holding neither SMEP nor SMAP): 0: v=0e e=0000 cpl=0 IP=ffffffff81fd4fbe CR2=ffff888000014750 memcpy_orig 1: v=0e e=0000 cpl=0 IP=ffffffff8439a7c9 CR2=ffff888004436ff8 early_ioremap_pmd 2: v=0e e=0000 cpl=0 IP=ffffffff8439a7ea CR2=ffff888003a3bff8 early_ioremap_pmd CR4=00000000000000a0 (PAE|PGE only) The same exit code also misbehaves on SMAP-capable systems: a deliberate clearcpuid=smap is silently overridden, because the gate forces SMAP back on at every exit, leaving hardware CR4 inconsistent with the cpu_tlbstate.cr4 shadow and defeating the administrator's choice. Root cause is twofold: 1. The gate exit forces a constant SMEP|SMAP mask instead of the bits the platform actually enabled. Saving CR4 on entry and restoring it on exit would not be a valid fix either: the gate proxies every CR4 write (the handler applies the requested value with SMEP stripped, as it executes from user-mapped .iee.si_text pages), so restoring the entry value would silently discard all runtime CR4 updates and, since setup_smep()/setup_smap() themselves go through the gate, the two bits could never be set at all. 2. The "read-back verification" loop is dead code: it tests %rax against itself right after the OR, so the comparison always succeeds and the branch is never taken; CR4 is never re-read. There is nothing to retry anyway -- a CR4 write either takes effect or faults. The ENABLE_WP macro carries the same vacuous testq/je pattern for CR0.WP. Introduce iee_cr4_set_mask, the SMEP/SMAP bits the boot CPU genuinely has, initialized in identify_cpu() right before setup_smep()/setup_smap() and after forced capabilities are applied, so clearcpuid= is honoured. The gate exit now re-reads the handler-written CR4 and ORs in only those supported bits, preserving the hardening intent (SMEP/SMAP cannot be cleared through the gate) without ever writing unsupported bits. Early gate users before identify_boot_cpu() (PSE/PGE/PCIDE setup) run with mask == 0, which is correct since neither bit is involved there. Delete the dead retry loops in the gate exit and in ENABLE_WP. Boot-tested with QEMU 10.2.1 (TCG), clang-built kernel: qemu64 and qemu64,+smep with haoc=1, which previously died at the point above, now boot through IEE initialization and the si_test self-test up to the expected missing-rootfs panic; qemu64,+smep,+smap with and without haoc=1 shows no regression; haoc=1 clearcpuid=smap now keeps SMAP cleared (CR4=0x001006f0 at runtime). Fixes: 7d1832b64006 ("Haoc: Add support for x86 Sensitive Instruction Protection(IEE_SIP)") Assisted-by: Kimi Code:K3 Signed-off-by: WangYuli --- arch/x86/include/asm/haoc/iee-si.h | 1 + arch/x86/kernel/cpu/common.c | 9 +++++++++ arch/x86/kernel/haoc/iee/iee-gate.S | 26 ++++++++++++++------------ arch/x86/kernel/haoc/iee/iee-si.c | 10 ++++++++++ 4 files changed, 34 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/haoc/iee-si.h b/arch/x86/include/asm/haoc/iee-si.h index 3baf1813f19d3..3792532cc432b 100644 --- a/arch/x86/include/asm/haoc/iee-si.h +++ b/arch/x86/include/asm/haoc/iee-si.h @@ -10,6 +10,7 @@ extern unsigned long cr4_pinned_mask; extern struct static_key_false cr_pinning; extern unsigned long cr4_pinned_bits; +extern unsigned long iee_cr4_set_mask; extern unsigned long __iee_si_text_start[]; extern unsigned long __iee_si_text_end[]; diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index d9e596eae225c..01f41bdc42591 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2088,6 +2088,15 @@ static void identify_cpu(struct cpuinfo_x86 *c) /* Disable the PN if appropriate */ squash_the_stupid_serial_number(c); +#ifdef CONFIG_IEE_SIP + /* + * The IEE rwx gate re-enables SMEP/SMAP from this mask; keep it in + * sync with what setup_smep()/setup_smap() are about to enable. + */ + iee_cr4_set_mask = (cpu_has(c, X86_FEATURE_SMEP) ? X86_CR4_SMEP : 0) | + (cpu_has(c, X86_FEATURE_SMAP) ? X86_CR4_SMAP : 0); +#endif /* CONFIG_IEE_SIP */ + /* Set up SMEP/SMAP/UMIP */ setup_smep(c); setup_smap(c); diff --git a/arch/x86/kernel/haoc/iee/iee-gate.S b/arch/x86/kernel/haoc/iee/iee-gate.S index 62a016555bdd2..fc81d79f80503 100644 --- a/arch/x86/kernel/haoc/iee/iee-gate.S +++ b/arch/x86/kernel/haoc/iee/iee-gate.S @@ -14,8 +14,6 @@ #include #include -#define X86_CR4_SMEP_SMAP (X86_CR4_SMEP | X86_CR4_SMAP) - /* * scratch_reg would be changed, * caller should dertimine if scratch_reg should be saved and restored. @@ -28,13 +26,14 @@ .endm .macro ENABLE_WP scratch_reg:req - /* Enable write protection */ + /* + * Enable write protection. A CR0 write either takes effect or + * faults, so there is nothing to retry; keep this a single write. + * (Pinning enforcement stays in native_write_cr0()/_iee_si_handler.) + */ movq %cr0, %\scratch_reg -1: orq $X86_CR0_WP, %\scratch_reg movq %\scratch_reg, %cr0 - testq $X86_CR0_WP, %\scratch_reg - je 1b .endm /* @@ -130,13 +129,16 @@ SYM_CODE_START(iee_rwx_gate) ENABLE_WP r12 - /* set SMEP=1 to disable supervisor-mode exec user-mode insn */ - movq %cr4, %rax /* rax -> cr4 */ -1: orq $X86_CR4_SMEP_SMAP, %rax + /* + * Re-enable SMEP/SMAP after running the handler. Only the bits the + * CPU actually supports may be set (setting an unsupported CR4 bit + * #GPs); iee_cr4_set_mask is initialized in identify_cpu() before + * setup_smep()/setup_smap(). No read-back retry: a CR4 write either + * takes effect or faults, there is nothing to retry. + */ + movq %cr4, %rax /* value written by the handler */ + orq iee_cr4_set_mask(%rip), %rax movq %rax, %cr4 - andq $X86_CR4_SMEP_SMAP, %rax - cmpq $X86_CR4_SMEP_SMAP, %rax - jnz 1 /* restore irq*/ popfq diff --git a/arch/x86/kernel/haoc/iee/iee-si.c b/arch/x86/kernel/haoc/iee/iee-si.c index 214c7617cfed0..2633ec9b8fe0b 100644 --- a/arch/x86/kernel/haoc/iee/iee-si.c +++ b/arch/x86/kernel/haoc/iee/iee-si.c @@ -1,10 +1,20 @@ // SPDX-License-Identifier: GPL-2.0 +#include #include + #include #include #include #include +/* + * SMEP/SMAP bits the boot CPU actually enabled. The rwx gate exit + * re-enables these after running the handler; it must never set bits + * the CPU does not support (writing them to CR4 would #GP). + * Initialized in identify_cpu() before setup_smep()/setup_smap(). + */ +unsigned long iee_cr4_set_mask __read_mostly; + unsigned long __iee_si_code notrace _iee_si_handler(int flag, ...) { va_list pArgs; From a39347357c908088f225355ee854dd679d54cfd8 Mon Sep 17 00:00:00 2001 From: WangYuli Date: Tue, 11 Aug 2026 17:16:49 +0800 Subject: [PATCH 06/13] HAOC: IEE: Give init_task a dedicated token page on x86 With CREDP enabled, every credential operation performed by swapper/0 corrupts init_task. The CREDP token accessors locate a task's struct task_token through __addr_to_iee(tsk): for slab-allocated tasks the IEE alias was remapped to a dedicated token page by iee_alloc_task_token_slab(), but init_task is a kernel image symbol, so its IEE alias (via __kimg_to_iee()) still points at its own physical page, and the token writes the gate performs with CR0.WP cleared land on init_task itself. struct task_token overlays the start of task_struct: token.new_cred (+40) covers task_struct.usage (+40) and task_struct.flags (+44). While swapper forks PID 1/2, _iee_copy_cred() stores the new cred pointer into token.new_cred, i.e. init_task.usage/flags transiently hold the two halves of a direct-mapped pointer until _iee_init_copied_cred() clears them; afterwards init_task.flags is left as 0 (even PF_KTHREAD is lost). The window between the two gate calls is wide (LSM hooks, key management, ...), and if the timer tick lands in it, scheduler_tick() reads that garbage flags word on current. Whenever the randomized direct-map base makes the PF_WQ_WORKER bit (0x20) of the transient value set -- which happens on roughly half of the boots since CONFIG_RANDOMIZE_MEMORY randomizes page_offset_base -- wq_worker_tick() runs on swapper and dereferences kthread_data(&init_task), which is legitimately NULL: [ 0.578469] Oops: 0000 [#1] PREEMPT SMP NOPTI [ 0.578469] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G S [ 0.578469] RIP: 0010:wq_worker_tick+0x19/0x180 [ 0.578469] RAX: 0000000000000000 RBX: ffff88a8bec52a40 [ 0.578469] CR2: 0000000000000020 CR4: 00000000001006f0 [ 0.578469] Call Trace: [ 0.578469] [ 0.578469] scheduler_tick+0x13d/0x340 [ 0.578469] update_process_times+0x7b/0x90 [ 0.578469] tick_periodic+0x6b/0x80 [ 0.578469] tick_handle_periodic+0x29/0x90 [ 0.578469] timer_interrupt+0x1d/0x30 [ 0.578469] __common_interrupt+0x49/0xc0 [ 0.578469] common_interrupt+0x92/0xb0 [ 0.578469] [ 0.578469] [ 0.578469] asm_common_interrupt+0x2b/0x40 [ 0.578469] RIP: 0010:iee_rw_gate+0x42/0x50 [ 0.578469] ? prepare_creds+0x87/0x300 [ 0.578469] copy_creds+0x85/0x3a0 [ 0.578469] copy_process+0x32a/0x1070 [ 0.578469] Kernel panic - not syncing: Fatal exception in interrupt The arm64 port already handles this with iee_prepare_init_task_token(), which allocates a dedicated token page for init_task and remaps its IEE alias onto it. Port that to x86, reusing the existing iee_set_token_page_valid() helper, and call it from iee_init(): iee_init() runs in mem_init(), which is before swapper's first credential operation in rest_init(). Note that CREDP's token accesses depend on the token-page remapping machinery, which is only built with IEE_PTRP; without it every token write would write through the 1:1 alias. Make that dependency explicit in Kconfig. Fixes: 416542ddedfbe ("HAOC: Add support for x86 CRED Protection (CREDP).") Assisted-by: Kimi Code:K3 Signed-off-by: WangYuli --- arch/x86/Kconfig | 1 + arch/x86/include/asm/haoc/iee-token.h | 1 + arch/x86/kernel/haoc/iee/iee-init.c | 6 +++++ arch/x86/kernel/haoc/iee/iee-token.c | 33 +++++++++++++++++++++++++++ 4 files changed, 41 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 91685de954096..a8b1123c2b142 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1576,6 +1576,7 @@ config IEE_SIP config CREDP bool "Struct cred protection(CREDP)" depends on IEE + select IEE_PTRP help Protects kernel struct cred. All modifications of cred must be made and verified by IEE APIs, and critical dereferences of cred would be monitored diff --git a/arch/x86/include/asm/haoc/iee-token.h b/arch/x86/include/asm/haoc/iee-token.h index 53e75396577d3..ff2e18822b6fa 100644 --- a/arch/x86/include/asm/haoc/iee-token.h +++ b/arch/x86/include/asm/haoc/iee-token.h @@ -10,6 +10,7 @@ extern unsigned long long iee_rw_gate(int flag, ...); extern struct kmem_cache *task_struct_cachep; +extern void iee_prepare_init_task_token(void); extern void iee_set_token_page_valid(unsigned long token, unsigned long new, unsigned int order); extern void iee_set_token_page_invalid(unsigned long token_addr, diff --git a/arch/x86/kernel/haoc/iee/iee-init.c b/arch/x86/kernel/haoc/iee/iee-init.c index 7ac4704769f4d..f567cbffdb5cb 100644 --- a/arch/x86/kernel/haoc/iee/iee-init.c +++ b/arch/x86/kernel/haoc/iee/iee-init.c @@ -17,6 +17,9 @@ #ifdef CONFIG_IEE_SIP #include #endif +#ifdef CONFIG_IEE_PTRP +#include +#endif /* IEE_OFFSET = pgtable_l5_enabled() ? 0x40000000000000 : 0x200000000000; */ unsigned long IEE_OFFSET = 0x200000000000; @@ -162,6 +165,9 @@ void __init iee_init(void) _iee_offset_init(); _iee_mapping_init(); _iee_stack_init(); +#ifdef CONFIG_IEE_PTRP + iee_prepare_init_task_token(); +#endif } bool __ro_after_init haoc_enabled; diff --git a/arch/x86/kernel/haoc/iee/iee-token.c b/arch/x86/kernel/haoc/iee/iee-token.c index b39ba03d9c565..8b665ea836afb 100644 --- a/arch/x86/kernel/haoc/iee/iee-token.c +++ b/arch/x86/kernel/haoc/iee/iee-token.c @@ -1,8 +1,41 @@ // SPDX-License-Identifier: GPL-2.0 +#include +#include +#include #include #include #include "slab.h" +/* + * The token of a slab-allocated task_struct lives on a dedicated page + * (see iee_alloc_task_token_slab()). init_task is a kernel image + * symbol, so its IEE alias still points at its own physical page and + * token writes (e.g. token->new_cred in CREDP) would land on + * init_task itself. Give it a dedicated token page too. + */ +void __init iee_prepare_init_task_token(void) +{ + unsigned long token = (unsigned long)__kimg_to_iee(&init_task); + unsigned long token_page; + unsigned int order = 0; + + /* Allocate one more page if the token crosses a page boundary. */ + if (ALIGN(token + sizeof(struct task_token), PAGE_SIZE) != + ALIGN(token + 1, PAGE_SIZE)) + order = 1; + + token_page = __get_free_pages(GFP_KERNEL | __GFP_ZERO, order); + if (!token_page) + panic("IEE: failed to allocate token page for init_task\n"); + + /* The remap works on whole pages; align the token address down. */ + iee_set_token_page_valid(ALIGN_DOWN(token, PAGE_SIZE), token_page, + order); + + /* init_task is already running, validate its token directly. */ + iee_validate_token(&init_task); +} + void iee_set_token_page_valid(unsigned long token, unsigned long token_page, unsigned int order) { From feeecf3b7e769cc43929bf2d81a00e08955ee8b0 Mon Sep 17 00:00:00 2001 From: WangYuli Date: Tue, 11 Aug 2026 17:17:50 +0800 Subject: [PATCH 07/13] HAOC: IEE: Fix inverted haoc_enabled check in iee_free_slab() iee_free_slab() returns early when haoc_enabled is set, which is exactly the case its caller runs in: with haoc=1 the deferred slab free is never scheduled, so task_struct and cred slabs are leaked together with their IEE token pages instead of being released. Drop the inverted check, and bail out if the work item allocation fails instead of dereferencing NULL. Fixes: 7806e1cc71a7 ("HAOC: Support pointer protection for x86 IEE (IEE_PTRP)") Assisted-by: Kimi Code:K3 Signed-off-by: WangYuli --- arch/x86/kernel/haoc/iee/iee-func.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/arch/x86/kernel/haoc/iee/iee-func.c b/arch/x86/kernel/haoc/iee/iee-func.c index 0054c1b3d6169..8001ce51dd305 100644 --- a/arch/x86/kernel/haoc/iee/iee-func.c +++ b/arch/x86/kernel/haoc/iee/iee-func.c @@ -34,11 +34,12 @@ struct iee_free_slab_work { void iee_free_slab(struct kmem_cache *s, struct slab *slab, void (*do_free_slab)(struct work_struct *work)) { - if(haoc_enabled) - return; struct iee_free_slab_work *iee_free_slab_work = kmalloc(sizeof(struct iee_free_slab_work), GFP_ATOMIC); + if (!iee_free_slab_work) + return; + iee_free_slab_work->s = s; iee_free_slab_work->slab = slab; INIT_WORK(&iee_free_slab_work->work, do_free_slab); From a27f1e839578f44146672e1d1f87acf46a6d8a74 Mon Sep 17 00:00:00 2001 From: WangYuli Date: Tue, 11 Aug 2026 17:18:27 +0800 Subject: [PATCH 08/13] HAOC: IEE: Fix per-cpu IEE stack allocation _iee_stack_init() allocates two pages per CPU: the first one is stored in a local variable, set read-only and then leaked, while the per-cpu stack pointer is derived from the second one, which stays writable. A stack obviously must be writable, so the RO setting on the first page is both misplaced and the page itself is lost. Allocate a single page per CPU, use it as the stack, and drop the bogus set_memory_ro(). Fixes: a8edec335283 ("HAOC: Add support for x86 Isolated Execution Environment") Assisted-by: Kimi Code:K3 Signed-off-by: WangYuli --- arch/x86/kernel/haoc/iee/iee-init.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/arch/x86/kernel/haoc/iee/iee-init.c b/arch/x86/kernel/haoc/iee/iee-init.c index f567cbffdb5cb..4daf4b6f14068 100644 --- a/arch/x86/kernel/haoc/iee/iee-init.c +++ b/arch/x86/kernel/haoc/iee/iee-init.c @@ -141,16 +141,18 @@ static void __init _iee_stack_init(void) { int cpu; struct iee_stack *iee_stack; - void *stack_base; struct page *page; for_each_possible_cpu(cpu) { - stack_base = (void *)page_address(alloc_pages(GFP_KERNEL, IEE_STACK_ORDER)); iee_stack = per_cpu_ptr(&iee_stacks, cpu); page = alloc_pages(GFP_KERNEL, IEE_STACK_ORDER); - iee_stack->stack = (void *)page_address(page) + PAGE_SIZE * (1 << IEE_STACK_ORDER); - pr_info("IEE: cpu %d, iee_stack 0x%lx", cpu, (unsigned long)iee_stack->stack); - set_memory_ro((unsigned long)stack_base, (1 << IEE_STACK_ORDER)); + if (!page) + panic("IEE: failed to allocate iee stack for cpu %d\n", + cpu); + iee_stack->stack = (void *)page_address(page) + + PAGE_SIZE * (1 << IEE_STACK_ORDER); + pr_info("IEE: cpu %d, iee_stack 0x%lx", cpu, + (unsigned long)iee_stack->stack); } } From c1afd2eedac8dca43900bacafa85150cc10b59fd Mon Sep 17 00:00:00 2001 From: WangYuli Date: Tue, 11 Aug 2026 17:19:07 +0800 Subject: [PATCH 09/13] HAOC: CREDP: Clear abort token before dropping the cred reference abort_creds() drops the caller's reference with put_cred() before asking the IEE to clear token.new_cred. put_cred() may be the last reference and release the object, so iee_abort_creds() potentially runs after the cred has been freed. Clear the token first, while the object is still alive, mirroring how commit_creds() performs the IEE operation before dropping references. Fixes: a7369f49fb0a ("HAOC: CREDP: protect commit_creds() from ROP attack.") Assisted-by: Kimi Code:K3 Signed-off-by: WangYuli --- kernel/cred.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/cred.c b/kernel/cred.c index d7042a4633408..df02c0290c1d7 100644 --- a/kernel/cred.c +++ b/kernel/cred.c @@ -619,11 +619,11 @@ void abort_creds(struct cred *new) atomic_long_read(&new->usage)); BUG_ON(atomic_long_read(&new->usage) < 1); - put_cred(new); #ifdef CONFIG_CREDP if (haoc_enabled) iee_abort_creds(new); #endif + put_cred(new); } EXPORT_SYMBOL(abort_creds); From b41560b63ef6c912ebd38d7351cea9b343b47656 Mon Sep 17 00:00:00 2001 From: WangYuli Date: Wed, 12 Aug 2026 13:48:39 +0800 Subject: [PATCH 10/13] HAOC: IEE: Enforce CR4 pinning in the SI handler's CR4 write path When haoc_enabled is set, native_write_cr4() routes every CR4 write through iee_write_cr4() -> _iee_si_handler(IEE_WRITE_CR4) before its own pinning check runs. The handler's CR4 branch only stripped SMEP and wrote the value straight to hardware -- unlike the sibling IEE_WRITE_CR0 branch, which faithfully replicates the CR0.WP pinning enforcement. The gate exit path only restores SMEP/SMAP (via iee_cr4_set_mask), so a write that cleared UMIP or FSGSBASE, or set CET, took effect silently and permanently. This renders CR4 pinning -- x86's defense-in-depth against CR4 tampering after a control-flow hijack -- void on haoc kernels with zero warning: - clearing UMIP re-enables userspace SIDT/SGDT/SLDT/SMSW/STR and effectively defeats KASLR; - clearing FSGSBASE #UDs the kernel's own ALTERNATIVE-patched {RD,WR}GSBASE sites; - setting CET makes the gate's CR0.WP toggle #GP (SDM Vol. 3A section 2.5), wedging the IEE gate. The defect was left half-finished from day one: the introducing commit turned cr4_pinned_mask/cr4_pinned_bits/cr_pinning into globals and added extern declarations for them in iee-si.h, yet the handler only ever consumed cr_pinning (in the CR0 branch). Replicate the pinning enforcement in the handler, with two deviations forced by the execution environment: - SMEP is excluded from the check/force set: the handler executes from user-mapped .iee.si_text with SMEP already cleared by the gate, so writing CR4 with SMEP=1 would fault on the very next instruction fetch. The gate exit restores SMEP/SMAP per iee_cr4_set_mask and is deliberately left untouched -- forcing pinned bits in the gate would break setup_smep()/setup_smap(), which legitimately run through the gate before pinning starts. - The value is corrected *before* writing instead of the native write-check-rewrite sequence, so a malicious value never reaches hardware at all; WARN_ONCE() then fires as usual. The check is gated on the cr_pinning static key, exactly like the native path, so early-boot CR4 writes (PSE/PGE/FSGSBASE/SMEP/SMAP/ UMIP setup, all before setup_cr_pinning()) are unaffected. Note that under IEE, CET is pinned to 0 by design: setup_cet() refuses to enable it because the gate's WP toggling conflicts with CR4.CET=1 (commit a8edec335283b: "IEE depends on CR0.wp"). This change preserves that behavior and only restores enforcement for the remaining pinned bits. Verified by booting with haoc=1 under QEMU: IEE shadow mapping, .iee.text U-page remapping and si_test all succeed. Fixes: 7d1832b64006 ("Haoc: Add support for x86 Sensitive Instruction Protection(IEE_SIP)") Assisted-by: Kimi Code:K3 Signed-off-by: WangYuli --- arch/x86/kernel/haoc/iee/iee-si.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/arch/x86/kernel/haoc/iee/iee-si.c b/arch/x86/kernel/haoc/iee/iee-si.c index 2633ec9b8fe0b..1ed1cf772994b 100644 --- a/arch/x86/kernel/haoc/iee/iee-si.c +++ b/arch/x86/kernel/haoc/iee/iee-si.c @@ -46,9 +46,30 @@ unsigned long __iee_si_code notrace _iee_si_handler(int flag, ...) break; } case IEE_WRITE_CR4: { + unsigned long bits_changed = 0; + /* + * The handler runs from user-mapped .iee.si_text with SMEP + * already cleared by the gate: writing CR4 with SMEP=1 + * here would fault on the next instruction fetch. SMEP is + * restored by the gate exit per iee_cr4_set_mask, so it + * takes no part in the pinning enforcement below. + */ + const unsigned long check_mask = cr4_pinned_mask & ~X86_CR4_SMEP; + const unsigned long check_bits = cr4_pinned_bits & ~X86_CR4_SMEP; + val = va_arg(pArgs, u64); - val &= ~(X86_CR4_SMEP); + val &= ~X86_CR4_SMEP; + if (static_branch_likely(&cr_pinning)) { + if (unlikely((val & check_mask) != check_bits)) { + bits_changed = (val & check_mask) ^ check_bits; + val = (val & ~check_mask) | check_bits; + } + } asm volatile("mov %0,%%cr4" : "+r" (val) : : "memory"); + /* Warn after we've corrected the changed bits. */ + if (static_branch_likely(&cr_pinning)) + WARN_ONCE(bits_changed, "pinned CR4 bits changed: 0x%lx!?\n", + bits_changed); break; } case IEE_LOAD_IDT: { From 0bd115c717b5ac4f367a290c9561349504cd3f52 Mon Sep 17 00:00:00 2001 From: WangYuli Date: Wed, 12 Aug 2026 13:48:51 +0800 Subject: [PATCH 11/13] HAOC: IEE: Restore const on cr4_pinned_mask Upstream keeps cr4_pinned_mask "static const". The IEE_SIP commit dropped both qualifiers so the SI handler could reference it from another TU, and added a runtime reassignment in parse_haoc_enabled() -- which stores exactly the value the initializer already carries (the IEE pinned set is identical to the upstream one), leaving the mask in writable .data for no reason. An attacker with an arbitrary kernel write could zero it and defeat CR4 pinning even on the non-haoc path: a hardening regression against the upstream baseline, not merely a missing improvement. Drop the redundant early-param assignment along with the extern declaration it needed in iee-init.c. With no writer left anywhere, the mask can simply be const again: only the "static" stays dropped (the handler needs cross-TU access), and the extern declaration in iee-si.h gains the const qualifier to match. This is strictly stronger than __ro_after_init: the object lives in .rodata, i.e. read-only for the kernel's entire lifetime rather than only after mark_rodata_ro(), and any future write attempt fails at compile time. Fixes: 7d1832b64006 ("Haoc: Add support for x86 Sensitive Instruction Protection(IEE_SIP)") Signed-off-by: WangYuli --- arch/x86/include/asm/haoc/iee-si.h | 2 +- arch/x86/kernel/cpu/common.c | 2 +- arch/x86/kernel/haoc/iee/iee-init.c | 12 +----------- 3 files changed, 3 insertions(+), 13 deletions(-) diff --git a/arch/x86/include/asm/haoc/iee-si.h b/arch/x86/include/asm/haoc/iee-si.h index 3792532cc432b..07b5400039ff6 100644 --- a/arch/x86/include/asm/haoc/iee-si.h +++ b/arch/x86/include/asm/haoc/iee-si.h @@ -7,7 +7,7 @@ #define __iee_si_code __section(".iee.si_text") #define __iee_si_data __section(".iee.si_data") -extern unsigned long cr4_pinned_mask; +extern const unsigned long cr4_pinned_mask; extern struct static_key_false cr_pinning; extern unsigned long cr4_pinned_bits; extern unsigned long iee_cr4_set_mask; diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 01f41bdc42591..778fa180e0ef1 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -406,7 +406,7 @@ static __always_inline void setup_umip(struct cpuinfo_x86 *c) /* These bits should not change their value after CPU init is finished. */ #ifdef CONFIG_IEE_SIP -unsigned long cr4_pinned_mask = +const unsigned long cr4_pinned_mask = X86_CR4_SMEP | X86_CR4_SMAP | X86_CR4_UMIP | X86_CR4_FSGSBASE | X86_CR4_CET; DEFINE_STATIC_KEY_FALSE_RO(cr_pinning); diff --git a/arch/x86/kernel/haoc/iee/iee-init.c b/arch/x86/kernel/haoc/iee/iee-init.c index 4daf4b6f14068..9aff3d6b9083c 100644 --- a/arch/x86/kernel/haoc/iee/iee-init.c +++ b/arch/x86/kernel/haoc/iee/iee-init.c @@ -174,20 +174,10 @@ void __init iee_init(void) bool __ro_after_init haoc_enabled; EXPORT_SYMBOL(haoc_enabled); -#ifdef CONFIG_IEE_SIP -extern unsigned long cr4_pinned_mask; -#endif static int __init parse_haoc_enabled(char *str) { int ret = kstrtobool(str, &haoc_enabled); - #ifdef CONFIG_IEE_SIP - if(haoc_enabled) - { - cr4_pinned_mask = - X86_CR4_SMEP | X86_CR4_SMAP | X86_CR4_UMIP | - X86_CR4_FSGSBASE | X86_CR4_CET; - } - #endif + return ret; } early_param("haoc", parse_haoc_enabled); From 2e0676fdeab6b8e26932bf0885935851be499471 Mon Sep 17 00:00:00 2001 From: WangYuli Date: Wed, 12 Aug 2026 13:49:16 +0800 Subject: [PATCH 12/13] x86/boot/compressed: Drop IEE_SIP CR3-write hooks from the decompressor The decompressor is linked separately (eg. ld.lld -pie) from its own object list and never sees arch/x86/kernel/haoc/iee/ objects. The IEE_SIP commit nevertheless templated #ifdef CONFIG_IEE_SIP if (haoc_enabled) iee_write_cr3_early(...); else write_cr3(...); #endif onto three CR3 writes in ident_map_64.c and one in pgtable_64.c, apparently as part of mechanically covering every raw CR3 write in the tree. The references can never be satisfied in the decompressor: haoc_enabled is defined only in iee-init.c (main kernel) and is parsed from the "haoc=" early_param in setup_arch(), long after the decompressor has exited. No IEE state (shadow mappings, IEE stacks, SIP page setup) exists at decompression time at all, so the branch is dead by design -- the original commit even chose the plain "_early" CR3 write for the true arm, which is exactly what the code now does unconditionally. A regular -O2 build links today only by an optimizer accident: both arms lower to the identical "mov %0,%%cr3" asm (the nested iee_rwx_gate() call inside native_write_cr3() is dead under the outer else), so clang folds the branch and drops the dead load of haoc_enabled. Rebuilding the same translation units at -O0 exposes the latent breakage immediately: ld.lld: error: undefined hidden symbol: haoc_enabled >>> referenced by ident_map_64.c >>> ident_map_64.o:(initialize_identity_maps) >>> referenced by ident_map_64.c >>> ident_map_64.o:(set_clr_page_flags) >>> referenced by pgtable_64.c >>> pgtable_64.o:(configure_5level_paging) >>> referenced 3 more times ld.lld: error: undefined hidden symbol: iee_rwx_gate >>> referenced by pgtable_64.c >>> pgtable_64.o:(iee_write_cr3) >>> referenced by ident_map_64.c >>> ident_map_64.o:(iee_write_cr3) Any change that makes the two arms diverge, a different compiler, or a different optimization level re-exposes it. Remove the four decompressor sites, and fence the IEE hunks in the shared headers behind !__DISABLE_EXPORTS so the inline chain (write_cr3() -> native_write_cr3() -> iee_write_cr3()) can never reintroduce the references into freestanding translation units. __DISABLE_EXPORTS is already defined for all of boot/compressed (as well as real-mode and purgatory, both likewise IEE-free), the main kernel never defines it, and asm/ibt.h already uses exactly this idiom to keep kernel-runtime mechanisms out of freestanding builds. desc.h gets the same guard defensively; nothing in the decompressor includes it today. No functional change to the decompressor: with the branch folded away at -O2, the generated code is identical before and after. Verified with a full LLVM=1 build and a haoc=1 QEMU boot. Fixes: 7d1832b64006 ("Haoc: Add support for x86 Sensitive Instruction Protection(IEE_SIP)") Assisted-by: Kimi Code:K3 Signed-off-by: WangYuli --- arch/x86/boot/compressed/ident_map_64.c | 21 --------------------- arch/x86/boot/compressed/pgtable_64.c | 7 ------- arch/x86/include/asm/desc.h | 8 ++++---- arch/x86/include/asm/special_insns.h | 6 +++--- 4 files changed, 7 insertions(+), 35 deletions(-) diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c index 848a958c639f8..8ecb4d40e20dd 100644 --- a/arch/x86/boot/compressed/ident_map_64.c +++ b/arch/x86/boot/compressed/ident_map_64.c @@ -180,14 +180,7 @@ void initialize_identity_maps(void *rmode) sev_prep_identity_maps(top_level_pgt); /* Load the new page-table. */ - #ifdef CONFIG_IEE_SIP - if(haoc_enabled) - iee_write_cr3_early(top_level_pgt); - else - write_cr3(top_level_pgt); - #else write_cr3(top_level_pgt); - #endif /* * Now that the required page table mappings are established and a @@ -231,14 +224,7 @@ static pte_t *split_large_pmd(struct x86_mapping_info *info, pmd = __pmd((unsigned long)pte | info->kernpg_flag); set_pmd(pmdp, pmd); /* Flush TLB to establish the new PMD */ - #ifdef CONFIG_IEE_SIP - if(haoc_enabled) - iee_write_cr3_early(top_level_pgt); - else - write_cr3(top_level_pgt); - #else write_cr3(top_level_pgt); - #endif return pte + pte_index(__address); } @@ -339,14 +325,7 @@ static int set_clr_page_flags(struct x86_mapping_info *info, snp_set_page_private(__pa(address & PAGE_MASK)); /* Flush TLB after changing encryption attribute */ - #ifdef CONFIG_IEE_SIP - if(haoc_enabled) - iee_write_cr3_early(top_level_pgt); - else - write_cr3(top_level_pgt); - #else write_cr3(top_level_pgt); - #endif return 0; } diff --git a/arch/x86/boot/compressed/pgtable_64.c b/arch/x86/boot/compressed/pgtable_64.c index ab495a0b16f22..dc21b0d89d667 100644 --- a/arch/x86/boot/compressed/pgtable_64.c +++ b/arch/x86/boot/compressed/pgtable_64.c @@ -200,14 +200,7 @@ asmlinkage void configure_5level_paging(struct boot_params *bp, void *pgtable) * Move the top level page table out of trampoline memory. */ memcpy(pgtable, trampoline_32bit, PAGE_SIZE); - #ifdef CONFIG_IEE_SIP - if(haoc_enabled) - iee_write_cr3_early((unsigned long)pgtable); - else - native_write_cr3((unsigned long)pgtable); - #else native_write_cr3((unsigned long)pgtable); - #endif /* Restore trampoline memory */ memcpy(trampoline_32bit, trampoline_save, TRAMPOLINE_32BIT_SIZE); diff --git a/arch/x86/include/asm/desc.h b/arch/x86/include/asm/desc.h index 4a90cf3c78dc7..94a8674872fd8 100644 --- a/arch/x86/include/asm/desc.h +++ b/arch/x86/include/asm/desc.h @@ -13,7 +13,7 @@ #include #include -#ifdef CONFIG_IEE_SIP +#if defined(CONFIG_IEE_SIP) && !defined(__DISABLE_EXPORTS) #include extern bool haoc_enabled; #endif @@ -215,7 +215,7 @@ static inline void native_load_gdt(const struct desc_ptr *dtr) asm volatile("lgdt %0"::"m" (*dtr)); } -#ifdef CONFIG_IEE_SIP +#if defined(CONFIG_IEE_SIP) && !defined(__DISABLE_EXPORTS) static __always_inline void iee_load_idt_early(const struct desc_ptr *dtr) { asm volatile("lidt %0"::"m" (*dtr)); @@ -224,7 +224,7 @@ static __always_inline void iee_load_idt_early(const struct desc_ptr *dtr) static __always_inline void native_load_idt(const struct desc_ptr *dtr) { - #ifdef CONFIG_IEE_SIP + #if defined(CONFIG_IEE_SIP) && !defined(__DISABLE_EXPORTS) if(haoc_enabled) iee_load_idt((void *)dtr); else @@ -261,7 +261,7 @@ static inline void native_idt_invalidate(void) .size = 0 }; -#ifdef CONFIG_IEE_SIP +#if defined(CONFIG_IEE_SIP) && !defined(__DISABLE_EXPORTS) // The native_idt_invalidate() is only called by machine_kexec(). // In the kdump path, IEE_SIP should not be used, so we directly // call iee_load_idt_early(), which is the original version of diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/special_insns.h index 1ab08117dea8c..0a3737ba7a7c9 100644 --- a/arch/x86/include/asm/special_insns.h +++ b/arch/x86/include/asm/special_insns.h @@ -9,7 +9,7 @@ #include #include #include -#ifdef CONFIG_IEE_SIP +#if defined(CONFIG_IEE_SIP) && !defined(__DISABLE_EXPORTS) #include extern bool haoc_enabled; #endif @@ -53,7 +53,7 @@ static inline unsigned long __native_read_cr3(void) return val; } -#ifdef CONFIG_IEE_SIP +#if defined(CONFIG_IEE_SIP) && !defined(__DISABLE_EXPORTS) static inline void iee_write_cr3_early(unsigned long val) { asm volatile("mov %0,%%cr3" : : "r" (val) : "memory"); @@ -62,7 +62,7 @@ static inline void iee_write_cr3_early(unsigned long val) static inline void native_write_cr3(unsigned long val) { - #ifdef CONFIG_IEE_SIP + #if defined(CONFIG_IEE_SIP) && !defined(__DISABLE_EXPORTS) if(haoc_enabled) iee_write_cr3(val); else From ee491ed6e016f5ad3d85bd8ff609f6c60df6183c Mon Sep 17 00:00:00 2001 From: WangYuli Date: Wed, 12 Aug 2026 15:11:11 +0800 Subject: [PATCH 13/13] HAOC: arm64: Make CREDP select IEE_PTRP CREDP's credential operations locate the per-task task_token through the IEE alias of the task and use token->new_cred as the prepare_creds()/commit_creds() validation channel, but the machinery that gives each task a dedicated, protected token page (token page allocation and IEE shadow PTE remapping) is only built with IEE_PTRP. With CREDP=y and IEE_PTRP=n every token write goes through the 1:1 alias and lands on the task itself, corrupting task_struct usage/flags, and the validation value sits in unprotected memory, which makes the ROP check both harmful and useless. The same Kconfig hole was closed on x86 by "HAOC: IEE: Give init_task a dedicated token page on x86"; apply the same select to arm64. Fixes: a7369f49fb0a ("HAOC: CREDP: protect commit_creds() from ROP attack.") Signed-off-by: WangYuli --- arch/arm64/kernel/haoc/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/kernel/haoc/Kconfig b/arch/arm64/kernel/haoc/Kconfig index 7aa900608650f..b2fe32a2f0701 100644 --- a/arch/arm64/kernel/haoc/Kconfig +++ b/arch/arm64/kernel/haoc/Kconfig @@ -34,6 +34,7 @@ config IEE_SIP config CREDP bool "Struct cred protection(CREDP)" depends on IEE + select IEE_PTRP help Protects kernel struct cred. All modifications of cred must be made and verified by IEE APIs, and critical dereferences of cred would be monitored