[PW_SID:1162506] [v6] mm: retry page faults once under the per-VMA lock - #2639
Open
linux-riscv-bot wants to merge 1 commit into
Open
[PW_SID:1162506] [v6] mm: retry page faults once under the per-VMA lock#2639linux-riscv-bot wants to merge 1 commit into
linux-riscv-bot wants to merge 1 commit into
Conversation
The per-VMA lock fault path releases the per-VMA lock when the folio cannot be locked. It then waits for the folio to become lockable without holding the VMA lock and retries the page fault. However, the retry always takes the mmap_lock instead of the per-VMA lock. This can cause serious lock contention when a writer is holding the mmap_lock at the same time. Add a single retry under the per-VMA lock in the architecture fault handler. This does not touch any page fault code in mm. The retry is very likely to succeed because the first page fault has already waited for the folio to become lockable. For example, this usually means that any required I/O has completed. This allows faults that can make progress on an immediate retry to stay on the per-VMA lock path, avoiding waits on the mmap_lock when it is write-contended. This reduces page-fault latency and mmap_lock contention. Some faults may retry unnecessarily, for example, those in __vmf_anon_prepare() or device-private fault handling, which require the mmap_lock. However, these cases are expected to be infrequent and only add one cheap per-VMA lock attempt. If the second attempt still returns VM_FAULT_RETRY, the fault continues through the existing mmap_lock path. Based on the stress model from Kunwu Chan and Wang Lian in RFC v2, we adapted a benchmark [1] to a 20-core Intel i7-12700 desktop by reducing the thread count and adjusting the memcg limits. The benchmark uses concurrent page faults under memcg pressure with parallel munmap to amplify mmap_lock read-write contention. Filemap throughput (higher is better) +---------+------------+---------------------+ | Threads | Vanilla | Patched | +---------+------------+---------------------+ | 40 | 1069.34 /s | 1400.13 /s (+30.9%) | +---------+------------+---------------------+ | 60 | 1038.12 /s | 1683.37 /s (+62.2%) | +---------+------------+---------------------+ | 80 | 1042.62 /s | 1767.83 /s (+69.6%) | +---------+------------+---------------------+ mmap_lock contention count (lower is better) +---------+-----------+---------+-----------+ | Threads | Vanilla | Patched | Reduction | +---------+-----------+---------+-----------+ | 40 | 3,187,336 | 52,086 | -98.4% | +---------+-----------+---------+-----------+ | 60 | 4,385,154 | 65,079 | -98.5% | +---------+-----------+---------+-----------+ | 80 | 5,337,890 | 69,708 | -98.7% | +---------+-----------+---------+-----------+ These results show that retrying once under the per-VMA lock keeps more file-backed faults on the fast path, improving throughput and reducing mmap_lock contention. Using benchmark [2], we tested this on a 20-core Intel i7-12700 desktop with a 2GB swapfile. The benchmark uses one pressure thread under memcg limits to keep a 128MB non-zero anonymous mapping under swap pressure, 12 reader threads to fault it back in, and optional mmap writer threads to amplify mmap_lock read-write contention. Each test ran for 60 seconds and reported completed reader rounds per second under swap pressure. Swap throughput (higher is better) +--------------+-------------+---------------------------+ | mmap writers | Vanilla | Patched | +--------------+-------------+---------------------------+ | 0 | 17303.09 /s | 17899.48 /s (+3.4%) | +--------------+-------------+---------------------------+ | 4 | 12596.23 /s | 16095.20 /s (+27.8%) | +--------------+-------------+---------------------------+ | 8 | 0.58 /s | 15420.57 /s (+2658619.0%) | +--------------+-------------+---------------------------+ With increasing mmap_lock write pressure, Vanilla degrades sharply and drops to near zero at eight writers. Patched kernel holds up much better. Performance was evaluated on a Pixel 6 running Android 17, using Baidu Tieba (com.baidu.tieba) and Tencent Video (com.tencent.qqlive), both very popular Android apps, as the workloads. For each workload, we performed 100 cold app launches with each kernel variant (vanilla and patched). Each run recorded cold app launch time, measured as the time to first frame, and the main thread's mmap_lock wait events. Shorter launch times indicate better performance. Baidu Tieba cold app launch time +-----------+----------+----------+--------+ | Statistic | Vanilla | Patched | Change | +-----------+----------+----------+--------+ | Mean | 3,672 ms | 3,580 ms | -2.5% | +-----------+----------+----------+--------+ | Maximum | 4,469 ms | 4,156 ms | -7.0% | +-----------+----------+----------+--------+ Baidu Tieba cold app launch time distribution +-------------+------------+------------+ | Time (ms) | Vanilla | Patched | +-------------+------------+------------+ | 2,750-2,999 | 0 (0.0%) | 1 (1.0%) | +-------------+------------+------------+ | 3,000-3,249 | 8 (8.0%) | 14 (14.0%) | +-------------+------------+------------+ | 3,250-3,499 | 17 (17.0%) | 30 (30.0%) | +-------------+------------+------------+ | 3,500-3,749 | 38 (38.0%) | 27 (27.0%) | +-------------+------------+------------+ | 3,750-3,999 | 26 (26.0%) | 18 (18.0%) | +-------------+------------+------------+ | 4,000-4,249 | 8 (8.0%) | 10 (10.0%) | +-------------+------------+------------+ | 4,250-4,499 | 3 (3.0%) | 0 (0.0%) | +-------------+------------+------------+ Baidu Tieba main-thread mmap_lock wait statistics +----------------------+----------------+----------------+--------+ | Metric | Vanilla | Patched | Change | +----------------------+----------------+----------------+--------+ | Total Wait Time | 569.9 ms/run | 469.9 ms/run | -17.6% | +----------------------+----------------+----------------+--------+ | Read-Lock Wait Count | 32.4 waits/run | 10.9 waits/run | -66.2% | +----------------------+----------------+----------------+--------+ Tencent Video cold app launch time +-----------+----------+----------+--------+ | Statistic | Vanilla | Patched | Change | +-----------+----------+----------+--------+ | Mean | 1,907 ms | 1,840 ms | -3.5% | +-----------+----------+----------+--------+ | Maximum | 3,023 ms | 2,851 ms | -5.7% | +-----------+----------+----------+--------+ Tencent Video cold app launch time distribution +-------------+------------+------------+ | Time (ms) | Vanilla | Patched | +-------------+------------+------------+ | 1,250-1,499 | 3 (3.0%) | 7 (7.0%) | +-------------+------------+------------+ | 1,500-1,749 | 32 (32.0%) | 33 (33.0%) | +-------------+------------+------------+ | 1,750-1,999 | 39 (39.0%) | 36 (36.0%) | +-------------+------------+------------+ | 2,000-2,249 | 12 (12.0%) | 15 (15.0%) | +-------------+------------+------------+ | 2,250-2,499 | 7 (7.0%) | 4 (4.0%) | +-------------+------------+------------+ | 2,500-2,749 | 5 (5.0%) | 3 (3.0%) | +-------------+------------+------------+ | 2,750-2,999 | 1 (1.0%) | 2 (2.0%) | +-------------+------------+------------+ | 3,000-3,249 | 1 (1.0%) | 0 (0.0%) | +-------------+------------+------------+ Tencent Video main-thread mmap_lock wait statistics +----------------------+----------------+---------------+--------+ | Metric | Vanilla | Patched | Change | +----------------------+----------------+---------------+--------+ | Total Wait Time | 139.6 ms/run | 66.4 ms/run | -52.4% | +----------------------+----------------+---------------+--------+ | Read-Lock Wait Count | 28.4 waits/run | 4.7 waits/run | -83.5% | +----------------------+----------------+---------------+--------+ Across both workloads, the single retry under the per-VMA lock substantially reduced mmap_lock read-side contention, leading to lower app startup times at both the mean and the tail. [1] https://gist.github.com/zhr250/c36c2c54d9351df37e12fd072d4926ef [2] https://gist.github.com/zhr250/218ffe693f842346b56434483127422c Signed-off-by: Hongru Zhang <zhanghongru@xiaomi.com> Suggested-by: Barry Song <baohua@kernel.org> Suggested-by: Suren Baghdasaryan <surenb@google.com> Suggested-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Tested-by: Nanzhe Zhao <zhaonanzhe@xiaomi.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR for series 1162506 applied to workflow__riscv__fixes
Name: [v6] mm: retry page faults once under the per-VMA lock
URL: https://patchwork.kernel.org/series/1162506/
Version: 6