[PW_SID:1162871] riscv: add SBI Supervisor Software Events support - #2642
Open
linux-riscv-bot wants to merge 9 commits into
Open
[PW_SID:1162871] riscv: add SBI Supervisor Software Events support#2642linux-riscv-bot wants to merge 9 commits into
linux-riscv-bot wants to merge 9 commits into
Conversation
Add definitions for the SBI Supervisor Software Events extension [1]. This extension enables the SBI to inject events into supervisor software much like ARM SDEI. [1] https://lists.riscv.org/g/tech-prs/message/515 Signed-off-by: Clément Léger <cleger@rivosinc.com> Co-developed-by: Himanshu Chauhan <himanshu.chauhan@oss.qualcomm.com> Signed-off-by: Himanshu Chauhan <himanshu.chauhan@oss.qualcomm.com> Co-developed-by: Zhanpeng Zhang <zhangzhanpeng.jasper@bytedance.com> Signed-off-by: Zhanpeng Zhang <zhangzhanpeng.jasper@bytedance.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
The SBI SSE extension allows firmware to notify supervisor software of events that must be delivered independently of normal S-mode interrupts. Firmware saves the minimal state required to enter the supervisor handler, and Linux builds the synthetic handler context around it. SSE can arrive while Linux is already in an exception entry path. At that point sscratch and tp may be in the middle of the normal trap-entry exchange, so they cannot always identify current. Store current in a per-CPU slot and use the hart ID passed by firmware to recover it. Give each event, including each CPU instance of a local event, a dedicated stack and shadow call stack. Synchronize vmapped stack ranges before unmasking events so that the handler cannot take a vmalloc fault while running in an NMI-like context. The handler is a synthetic supervisor episode, but completion must resume the context interrupted by the SSE. Preserve stvec and, when the hypervisor extension is present, hstatus across the handler. Read the interrupted a6 and a7 values from the SSE attributes, construct pt_regs for the interrupted context, and write back any changes made by the handler. Nested exceptions on the SSE event stack temporarily replace both TASK_TI_KERNEL_SP and TASK_TI_USER_SP. Preserve their original values across the handler and restore them before completing the event, so a nested exception cannot leave the interrupted task referring to the event stack. Keep an explicit EVENT_REGISTER not-supported result distinct from other firmware failures. This lets clients select another delivery mechanism only when firmware has positively rejected the requested event. Signed-off-by: Clément Léger <cleger@rivosinc.com> Co-developed-by: Himanshu Chauhan <himanshu.chauhan@oss.qualcomm.com> Signed-off-by: Himanshu Chauhan <himanshu.chauhan@oss.qualcomm.com> Co-developed-by: Zhanpeng Zhang <zhangzhanpeng.jasper@bytedance.com> Signed-off-by: Zhanpeng Zhang <zhangzhanpeng.jasper@bytedance.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
SSE delivery is independent of normal S-mode interrupts. Firmware may also retain an event registration until Linux explicitly unregisters it. A hart must therefore stop accepting SSE events before Linux stops servicing the registered handler. Mask SSE on the local hart before panic stop, CPU stop, restart, poweroff, and crash shutdown paths. This prevents firmware from entering Linux-owned handler state after the corresponding CPU or kernel context is no longer valid. A crash kernel cannot identify or take ownership of registrations inherited from the crashed kernel. Reject a later normal kexec while such SSE state may still exist, rather than transferring unknown firmware state to another kernel. Signed-off-by: Zhanpeng Zhang <zhangzhanpeng.jasper@bytedance.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Add a driver-level interface for RISC-V SSE. Linux clients can register handlers, select a target CPU for global events, and enable, disable or unregister events. The architecture entry wrapper completes an event after the registered handler returns. PMU and GHES drivers can use this interface. Represent global events with one firmware registration and local events with one registration per possible CPU. Keep registration and enable state stable across CPU hotplug, and validate firmware-provided hart IDs before converting them to Linux CPU IDs. Use phys_addr_t for attribute buffers to match the physical addresses passed to firmware. Require MMU support because the current event-stack implementation relies on vmapped memory and TLB synchronization. Serialize client list updates with the SSE mutex and the CPU read lock. Normal CPU hotplug callbacks provide the matching write-side exclusion, and CPUHP state removal holds the SSE mutex. These rules avoid holding an additional spinlock across firmware calls. Local event operations select a per-CPU registration, so require callers to remain on the current CPU and use lockdep assertions to verify that contract. Propagate firmware failures from register, disable and unregister operations. Update Linux state only after successful firmware operations, and release an event only after all registrations are gone. Preserve the difference between SBI_ERR_NOT_SUPPORTED and a generic SBI failure so clients only select a fallback after firmware explicitly rejects an SSE operation. If setup fails partway through a local event, roll back only the CPU instances changed by that invocation. Keep non-fallback errors across CPUs. This lets a client distinguish an event rejected as unsupported by every failing hart from an unknown firmware failure. A failed registration rollback can leave firmware state without a client handle. Retain these events on a driver-owned cleanup list with a no-op handler. This lets CPU hotplug and shutdown retry cleanup without relying on client callback lifetime. Mask SSE before CPU teardown. If a normal CPU-offline teardown fails after partially changing local events, reconstruct their requested registration and enable state. Unmask the hart, then return the original error to abort the offline operation. Shutdown and CPUHP state removal cannot fail, so record incomplete cleanup and refuse an unsafe normal kexec instead. Close enable admission before reboot or kexec teardown. Protect the shutdown check and firmware enable operation with RCU, then drain existing enable calls before CPUHP removes registrations. This prevents a client from re-enabling an event between the teardown disable and unregister operations. A crash kernel cannot identify registrations inherited from the crashed kernel. Leave SSE masked, report the retained firmware state separately from an unavailable extension, and do not let clients select a delivery path that may still be owned by the old kernel. Signed-off-by: Clément Léger <cleger@rivosinc.com> Co-developed-by: Himanshu Chauhan <himanshu.chauhan@oss.qualcomm.com> Signed-off-by: Himanshu Chauhan <himanshu.chauhan@oss.qualcomm.com> Co-developed-by: Zhanpeng Zhang <zhangzhanpeng.jasper@bytedance.com> Signed-off-by: Zhanpeng Zhang <zhangzhanpeng.jasper@bytedance.com> Acked-by: Conor Dooley <conor.dooley@microchip.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
RISC-V enables interrupts in handle_page_fault() before checking whether the fault occurred with fault handling disabled. A nofault access from an atomic context can therefore run tracepoints and open an interrupt window before reaching the exception-table fixup. Handle an exception-table entry before entering the generic fault path when fault handling is disabled. Also keep interrupts disabled until such a fault has been resolved. This makes RISC-V consistent with the expectation that an in-atomic nofault access does not enter the normal fault-handling path. It also removes one source of re-entry when perf sampling is delivered through an SBI Supervisor Software Event (SSE). Signed-off-by: Zhanpeng Zhang <zhangzhanpeng.jasper@bytedance.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
PMU overflow delivery through SSE enters Linux with a synthetic supervisor context on a dedicated event stack. The perf unwinder must use the context interrupted by the overflow rather than treating the SSE handler frame as the sampled frame. Use the interrupted pt_regs published by the RISC-V SSE entry path. Walk a kernel callchain only when the interrupted PC, SP, and frame pointer are consistent with the current task stack. For sensitive entry windows and IRQ stacks whose bounds cannot be proven, retain the interrupted PC without following an unsafe frame chain. User callchains continue through the existing nofault RISC-V user unwinder. If hstatus.SPV says the interrupted context was a guest, do not interpret the guest stack through the host address space. DWARF callchains additionally copy a raw user stack. The generic arch_perf_out_copy_user() implementation can take an exception-table handled fault when a source page is not resident. Repeated nested faults from the SSE handler can corrupt the interrupted kernel context under load. Provide an SSE-specific RISC-V copy path. Verify that the active page table belongs to current, use fast-only GUP to acquire each resident source page without falling back to a faulting slow path, and copy through the kernel mapping while holding the page reference. Stop at the first unavailable page and preserve perf's existing truncated-user-stack semantics. Keep the generic in-atomic user copy unchanged outside an SSE handler. Susheng Yang reported this failure with perf callchain workloads. Reported-by: Susheng Yang <yang.susheng@zte.com.cn> Signed-off-by: Zhanpeng Zhang <zhangzhanpeng.jasper@bytedance.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Register a handler for the local PMU overflow SSE event so that RISC-V perf can receive overflows even when normal S-mode interrupts are masked. Reuse the existing overflow handler and pass it the interrupted pt_regs rebuilt by the architecture SSE entry path. Select the delivery mechanism once during PMU probe. Prefer SSE when its event can be registered and enabled. If the extension or PMU event is explicitly unsupported, use the ordinary PMU interrupt. Do not enable the interrupt after any other SSE setup failure or when a crash kernel may have inherited firmware state. Install the PMU enable and disable callbacks only after SSE delivery becomes active. Keep the local PMU SSE event disabled across CPU power management. On entry, the generic SSE notifier masks the hart before the lower-priority PMU notifier disables the event and stops the counters. On exit, the SSE notifier first unmasks the hart while the event remains disabled. The PMU notifier then restores counters and event userpage state before enabling the event. An unmask failure stops the notifier chain and leaves the counters stopped. Ordinary PMU interrupts retain their existing notifier ordering. An SSE overflow can arrive as soon as the event is enabled during probe. Publish the counter mask before SSE setup, so an early handler can stop the counter source even before perf starts admitting normal samples. After a real overflow, restart only counters whose perf state is still running. Honor a non-zero return from perf_event_overflow() and leave throttled events stopped. Guest attribution is not part of this version. Detect an interrupted guest from hstatus.SPV and skip its sample while still updating the period and counter state, rather than exposing guest state as a host sample. The perf PMU callbacks cannot return errors. If an SSE transition or interrupted-context read fails, latch the failure per CPU and stop its mapped events through the normal perf state transitions. Do not reset the firmware counter mapping behind perf, restart a failed event, or attempt a runtime switch to IRQ delivery. During cleanup, close callback admission and synchronously drain each CPU before disabling and unregistering the SSE event. If firmware refuses the cleanup, stop the counter source and transfer the event to the SSE core so later CPU hotplug or shutdown processing can retry without using freed PMU callback state. Signed-off-by: Clément Léger <cleger@rivosinc.com> Co-developed-by: Himanshu Chauhan <himanshu.chauhan@oss.qualcomm.com> Signed-off-by: Himanshu Chauhan <himanshu.chauhan@oss.qualcomm.com> Co-developed-by: Zhanpeng Zhang <zhangzhanpeng.jasper@bytedance.com> Signed-off-by: Zhanpeng Zhang <zhangzhanpeng.jasper@bytedance.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Add an SSE selftest module and runner. Loading the module executes
smoke tests for the SSE framework, and the runner reports any error
emitted by the module.
Add stress={0,1,2} modes to exercise repeated handler entry and
completion, single and multiple read-only SBI calls from a handler, and
self re-injection. Check the SBI return values so handler execution alone
cannot produce a false pass, and avoid touching an unreserved PMU counter
owned by another user.
Harden the test against false failures and leaks by using TEST_PROGS for
the runner, using phys_addr_t for SBI attribute buffers, adding timeouts
to busy waits, pinning each priority chain with migrate_disable(), and
holding the CPU read lock while a fast-test target is selected, injected,
and completed. Unregister all registered events on error, check teardown
failures, and let kthread_stop() drive monitor-thread exit.
Track handler progress across fast, priority, and stress paths. If
firmware reports injectable events but the test cannot acquire or handle
any of them, report SKIP instead of silently passing a capability-only
run.
Signed-off-by: Clément Léger <cleger@rivosinc.com>
Co-developed-by: Himanshu Chauhan <himanshu.chauhan@oss.qualcomm.com>
Signed-off-by: Himanshu Chauhan <himanshu.chauhan@oss.qualcomm.com>
Co-developed-by: Zhanpeng Zhang <zhangzhanpeng.jasper@bytedance.com>
Signed-off-by: Zhanpeng Zhang <zhangzhanpeng.jasper@bytedance.com>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
On RISC-V, PMU overflow interrupts can be delivered through the SBI
Supervisor Software Events (SSE) mechanism. A perf event that samples
the raw user stack (PERF_SAMPLE_STACK_USER, as perf record
--call-graph dwarf does) then copies a chunk of the interrupted task's
user stack from an NMI-like context. If that copy is allowed to take a
nested page fault it can corrupt the interrupted task's kernel state and
hang or crash the machine under load; this is what
riscv_perf_out_copy_user() and the nofault page-fault change fix.
The existing SSE selftest module exercises the framework (register,
enable, inject, complete, priorities, stress) but never drives the perf
user-stack copy that motivated the no-fault path. Add a userspace test
that closes that gap:
- Open a sampling hardware PMU event with PERF_SAMPLE_STACK_USER over a
deep, partially non-resident user stack, drain the ring buffer, and
verify every SAMPLE record is well formed and never reports more
dumped bytes than were requested. This checks that a non-resident
page truncates the dump cleanly instead of faulting or overrunning.
- Drive a multi-CPU unix-socket + deep-recursion workload under
high-frequency DWARF sampling; the pass criterion is simply that the
machine survives, since the original bug took it down.
The test reports SKIP when hardware PMU sampling is unavailable or
perf_event_paranoid forbids it, so it is safe to run unprivileged or in
constrained environments. It is placed under the RISC-V SSE selftests
because SSE delivery is the RISC-V-specific condition it protects, and
is wired into the sse subtarget Makefile alongside the module runner.
Signed-off-by: Zhanpeng Zhang <zhangzhanpeng.jasper@bytedance.com>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR for series 1162871 applied to workflow__riscv__for-next
Name: riscv: add SBI Supervisor Software Events support
URL: https://patchwork.kernel.org/series/1162871/
Version: 10