[PW_SID:1160850] kbuild: significantly speed up kernel builds - #2624
Open
linux-riscv-bot wants to merge 23 commits into
Open
[PW_SID:1160850] kbuild: significantly speed up kernel builds#2624linux-riscv-bot wants to merge 23 commits into
linux-riscv-bot wants to merge 23 commits into
Conversation
added 23 commits
September 8, 2026 23:05
Commit 3e86e4d ("kbuild: keep .modinfo section in vmlinux.unstripped") keeps .modinfo symbols out of System.map and kallsyms, which assumes unique IDs have a format like '__UNIQUE_ID_modinfo123'. However, commit afb026b ("compiler: Tweak __UNIQUE_ID() naming"), sent in the same cycle, changes this to '__UNIQUE_ID_modinfo_123'. As a result this regexp has never matched and every kernel since v6.18 has carried one kallsyms entries for every MODULE_INFO() declaration in the kernel whether the modules are compiled or not. That's 5,810 entries for an x86 defconfig build and 15,200 for arm64. On x86 defconfig that is 113 KiB of kallsyms tables and 32 KiB of bzImage, and every lookup walks past them. Fix the pattern. Fixes: 3e86e4d ("kbuild: keep .modinfo section in vmlinux.unstripped") No measurable change in build time, the smaller tables are not on any path the build waits for. Assisted-by: LLM Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Commit b18b047 ("kbuild: change scripts/mksysmap into sed script") converted scripts/mksysmap from a shell script to a sed script. However an error was made - escaping of '$' required \\ escaping in shell but only \ in a sed script. This was mostly corrected in commit 7a6c355 ("scripts/mksysmap: Fix escape chars '$'"), but this fix missed arm64 PIE namespace local symbols like __pi_$x and __pi_$d which appear in System.mapand /proc/kallsyms: $ grep __pi_\\$ /proc/kallsyms | sort -u 0000000000000000 d __pi_$d 0000000000000000 t __pi_$x Fix the escaping properly. Fixes: b18b047 ("kbuild: change scripts/mksysmap into sed script") No measurable change in build time. Assisted-by: LLM Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
The kallsyms program compresses symbols by figuring out the most commonly
used substrings in all of the input symbols then uses special character
codes to represent them.
For instance, 0xf7 might end up representing "write_", then every single
symbol that contains "write_" can use 0xf7 as a shorthand and save 5 bytes
each time.
'Special' character codes are any byte value that is not used in any
symbol, either due to being an invalid character, or not being present in
any symbol (e.g. if no symbol contains 'z', then 'z' can be used as special
character).
It does this by first figuring out which special characters are available
in insert_real_symbols_in_table(), then iterating through every available
special character, counting how many times each pair of adjacent characters
appear in symbols in build_initial_token_table().
These adjacent pairs are known as 'tokens'.
Token counts are initially obtained by build_initial_token_table(), then
optimize_result() calls find_best_token() to determine the token that
appeared the most number of times and assigns it the next special
character.
Finally, optimize_result() calls compress_symbols() to replace every token
in every symbol with its special character, which updates token_profit[] as
it does so.
This process is repeated for each remaining available special character,
with tokens now perhaps containing previously assigned special
characters (e.g. if 'wr' was assigned 0x80, then the token representing
'wri' would be '\x80i').
This compresses that token by 50% in each symbol it appears in (two bytes
are now represented by one) and thus by repeatedly doing this kallsyms
obtains good symbol compression.
However, compress_symbols() is seriously inefficient - it iterates through
EVERY symbol for EVERY special character assignment, i.e. ~256 *
nr_symbols.
Modern x86-64 kernels, for instance, have ~158,000 symbols, so millions of
iterations are performed, most of which are entirely unnecessary (tokens
don't appear in most symbols).
In practice kallsyms spends half its runtime doing this, two or three times
per vmlinux link step.
Fix this by tracking which symbols each token appears in token_syms[], and
only compress symbols which actually need to be updated.
Each time a token is compressed that token can no longer appear in any
symbol, so that token_syms[] entry can be freed.
However new token_syms[] entries must be created for each new token
containing the assigned special character, but this is bounded by the
number of replacements in the symbol which is very small.
In testing on an x86-64 platform using clang, each kallsyms invocation
dropped from 0.59s to 0.33s with CONFIG_KALLSYMS_ALL set and from 0.38s to
0.22s without it set.
The data was carefully checked and verified to be byte-for-byte identical
for six symbol sets (two vmlinux passes, vmlinux.o, three userspace
binaries) with all option combinations.
As part of this change, additionally refactor the code to be a little
easier to follow.
kallsyms runs two or three times on the serial tail of every build that
links vmlinux, no-op builds do not link and are unchanged.
Whole build, 128-thread Threadripper 9980X, best of N runs:
before after delta
-------------------------------
x86 defconfig, touch mm/vma.c, gcc 11.4s 10.8s -0.55s (-5%)
x86 defconfig, touch mm/vma.c, clang 11.4s 10.7s -0.66s (-6%)
x86 defconfig, clean, gcc 30.3s 29.5s -0.80s (-3%)
x86 defconfig, clean, clang 30.3s 29.7s -0.62s (-2%)
x86 allmodconfig, touch mm/vma.c, gcc 46.2s 45.3s -0.91s (-2%)
x86 allmodconfig, touch mm/vma.c, clang 44.2s 42.9s -1.3s (-3%)
Assisted-by: LLM
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
kallsyms generates an assembly file that consists mostly of .byte entries
containing compressed names, token strings and name-sorted sequence
numbers.
For an x86-64 build with 158k symbols that is a 37 MiB .S file which takes
0.57s to assemble each of the two to three times it is built over a kernel
build.
Each time it is generated it also takes kallsyms a similar amount of time
to output it.
Avoid this overhead by instead outputting this data as binary and importing
it into the assembly using the .incbin directive.
Tables that are wider than a byte remain part of the assembly to ensure
endianness and relative relocations are performed correctly.
With this change, the output assembly file shrinks from 37 MiB to 9.8 MiB,
with a 2.6 MiB binary data file alongside it, and the object remains
identical.
The generated binary file is deleted correctly on build clean along with
all other ephemeral data.
On an x86-64 system with CONFIG_KALLSYMS_ALL set:
before after delta
scripts/kallsyms 0.24s 0.18s 0.06s
assemble 0.57s 0.16s 0.41s
Per kallsyms invocation/assembly, for a total of 0.47s time saving upon
invocation.
An incremental build on the same system was reduced from 11.15s to 9.65s,
indicating a total of 1.5 seconds saved over the build.
The kallsyms runs and their assembly are on the serial tail of every build
that links vmlinux, no-op builds are unchanged.
Whole build, 128-thread Threadripper 9980X, best of N runs:
before after delta
-------------------------------
x86 defconfig, touch mm/vma.c, gcc 10.8s 9.9s -0.92s (-8%)
x86 defconfig, touch mm/vma.c, clang 10.7s 9.5s -1.2s (-11%)
x86 defconfig, clean, gcc 29.5s 28.7s -0.81s (-3%)
x86 defconfig, clean, clang 29.7s 28.6s -1.1s (-4%)
x86 allmodconfig, touch mm/vma.c, gcc 45.3s 44.0s -1.3s (-3%)
x86 allmodconfig, touch mm/vma.c, clang 42.9s 40.2s -2.7s (-6%)
Assisted-by: LLM
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Unless instructed otherwise, nm sorts by name.
There are places where this is unnecessary - an invocation from
scripts/sorttable every vmlinux link, scripts/check-function-names.sh run
after every vmlinux.o link, and the x86 VOFFSET and ZOFFSET listings
between vmlinux and bzImage.
Both GNU nm and llvm-nm accept the same '-p' parameter to disable sorting
in these instances, so use that to prevent this unnecessary work.
Each nm run on its own, x86-64, median of 5:
GNU nm 2.47 llvm-nm 22
before after delta before after delta
defconfig
sorttable, nm -S vmlinux 0.080s 0.039s -0.041s 0.265s 0.113s -0.152s
check-function-names.sh 0.072s 0.033s -0.039s 0.244s 0.102s -0.142s
VOFFSET 0.077s 0.034s -0.043s 0.261s 0.113s -0.148s
ZOFFSET 0.007s 0.005s -0.002s 0.005s 0.005s 0.000s
TOTAL -0.125s -0.442s
allmodconfig
sorttable, nm -S vmlinux 0.156s 0.063s -0.093s 0.539s 0.223s -0.316s
check-function-names.sh 0.149s 0.054s -0.095s 0.537s 0.217s -0.320s
VOFFSET 0.146s 0.052s -0.094s 0.524s 0.221s -0.303s
ZOFFSET 0.007s 0.005s -0.002s 0.006s 0.004s -0.002s
TOTAL -0.284s -0.941s
llvm-nm appears to be a lot slower than GNU nm, so these builds naturally
improve the most.
The four run one after another in the serial tail of every build that links
vmlinux so impact kernel builds directly.
In an allmodconfig the decompressor is built in parallel while the modules,
so only the two before the vmlinux link contribute to build time.
The build outputs remain unchanged.
Whole build, 128-thread Threadripper 9980X, median of 3:
before after delta
----------------------------------
x86 defconfig, touch mm/vma.c, gcc 10.02s 9.83s -0.19s (-2%)
x86 defconfig, touch mm/vma.c, clang 9.52s 9.24s -0.28s (-3%)
x86 defconfig, clean, gcc 28.71s 28.58s -0.13s (-0.5%)
x86 defconfig, clean, clang 28.69s 28.31s -0.38s (-1%)
x86 allmodconfig, touch mm/vma.c, cl 40.52s 39.83s -0.69s (-2%)
Assisted-by: LLM
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
A kernel build consists of more than one linking pass on vmlinux.o and
vmlinux, at minimum two, and with CONFIG_KALLSYMS and BTF enabled on x86-64
for example there are 5 such stages.
For architectures that build their own relocation tables (x86, riscv, mips,
s390), vmlinux is linked with the --emit-relocs parameter specified.
However, this is only required on the final vmlinux link.
Symbol tables of trial links preceding it don't need it because they
already check that System.map matches kallsyms symbols on each build.
GNU ld is slow at emitting relocation tables, so this results in a
reduction in build time.
Whole build, 128-thread Threadripper 9980X, best of N runs:
before after delta
-------------------------------
x86 defconfig, touch mm/vma.c, gcc 9.8s 9.4s -0.43s (-4%)
x86 defconfig, clean, gcc 28.6s 28.1s -0.47s (-2%)
x86 allmodconfig, touch mm/vma.c, gcc 44.0s 42.7s -1.3s (-3%)
Note that this has little impact on LLVM ld which performs this operation
more efficiently.
Assisted-by: LLM
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Extend elf-parse to be able to read the symbol table of vmlinux in kallyms. This requires the ability to open ELF files in read-only mode, so provide elf_map_ro() to do so. It also requires accessors for section flags and symbol bindings, so provide these via shdr_flags() and sym_bind(). Also, check for the file being an ELF file first in elf_parse(). This is the logical thing to check for first, but additionally prevents kallsyms from having to check this it self. Assisted-by: LLM Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
mksysmap is a sed script consisting of 30 patterns which link-vmlinux.sh
uses to generate *.syms files, and which kallsyms is then called against to
generate *.kallsyms files, with the final vmlinux build ultimately
generating System.map.
For an x86-64 allmodconfig build, this involves three nm runs over a 250
MiB file and half a million lines written and read each time - 0.5s per
pass for llvm-nm, and 0.2s for GNU nm, with parsing on top of that.
This is unnecessary, instead have kallsyms simply read the ELF file
directly making use of the existing elf-parse library in scripts/.
This changes kallsyms such that its input is no longer the output from nm,
but rather an ELF file.
However, if the input file is empty, it outputs an empty table, which
retains the same behaviour on first pass that the build system expects.
System.map is byte-identical to nm | mksysmap for GNU nm and llvm-nm on
two x86 configurations each, and for llvm-nm on arm64, arm, s390 and
loongarch defconfigs, so are the kallsyms tables of every pass.
Relinking vmlinux, link steps included:
before after
allmodconfig clang 9.1s 7.9s
allmodconfig gcc 7.8s 7.6s
defconfig clang 3.7s 3.0s
defconfig gcc 3.5s 3.4s
Whole build, 128-thread Threadripper 9980X, best of N runs:
before after delta
-------------------------------
x86 defconfig, touch mm/vma.c, gcc 9.4s 8.9s -0.45s (-5%)
x86 defconfig, touch mm/vma.c, clang 9.2s 8.1s -1.1s (-12%)
x86 defconfig, clean, gcc 28.1s 27.6s -0.51s (-2%)
x86 defconfig, clean, clang 28.2s 27.2s -1.0s (-4%)
x86 allmodconfig, touch mm/vma.c, gcc 42.7s 42.0s -0.70s (-2%)
x86 allmodconfig, touch mm/vma.c, clang 40.2s 38.1s -2.1s (-5%)
Assisted-by: LLM
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Commit 3e86e4d ("kbuild: keep .modinfo section in vmlinux.unstripped") placed MODULE_INFO() strings in the .modinfo section in vmlinux.unstripped, however it left the section allocatable, so it gets assigned an address and on arm64, arm and riscv, the section is tagged as PT_LOAD. This is useless as the data is ultimately stripped anyway. Doing this results in a lot of unnecessary work - each pass copies the whole file, 450 MIB with relocations for an x86 allmodconfig build and 250 MiB for an arm64 allmodconfig build. This adds ~0.7s on the serial tail of every build for arm64 and ~0.2s for x86 (the tail is single-threaded work done after parallel work has finished). Nothing requires .modinfo to exist at an address, so fix this by using --dump-section which prevents the allocation. Image, bzImage, System.map and modules.builtin.modinfo are unchanged and the stripped vmlinux differs only in its program headers. The objcopy passes are on the serial tail of every build that links vmlinux, no-op builds are unchanged. Whole build, 128-thread Threadripper 9980X, best of N runs: before after delta ------------------------------- x86 defconfig, touch mm/vma.c, gcc 8.9s 8.8s -0.16s (-2%) x86 defconfig, touch mm/vma.c, clang 8.1s 7.9s -0.20s (-2%) x86 defconfig, clean, gcc 27.6s 27.5s -0.13s (0%) x86 defconfig, clean, clang 27.2s 27.0s -0.18s (-1%) x86 allmodconfig, touch mm/vma.c, gcc 42.0s 41.6s -0.40s (-1%) Assisted-by: LLM Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
When each object's command line is expanded, kbuild has to figure out on
multiple occasions whether it's built-in or part of a module and which
composite object it belongs to.
This causes the time spent on each directory in the kernel tree to grow
O(n^2) with its object count, which is especially problematic for instance
in drivers/gpu/drm/amd/amdgpu with 310 objects.
No-op builds (i.e. make -j $(nproc) when nothing has changed) are
particularly impacted by this.
Fix the issue by caching this data and looking it up instead of getting it
over and over again.
This has a particularly large impact on allmodconfig builds.
Whole build, 128-thread Threadripper 9980X, best of N runs:
before after delta
-------------------------------
x86 defconfig, no-op make, gcc 0.94s 0.92s -0.02s (-2%)
x86 defconfig, no-op make, clang 1.1s 1.1s -0.01s (-1%)
x86 allmodconfig, no-op make, gcc 13.0s 12.3s -0.64s (-5%)
x86 allmodconfig, no-op make, clang 13.8s 13.2s -0.64s (-5%)
x86 allmodconfig, touch mm/vma.c, gcc 41.6s 40.9s -0.70s (-2%)
x86 allmodconfig, touch mm/vma.c, clang 38.1s 37.6s -0.52s (-1%)
Assisted-by: LLM
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Each object's .cmd file lists its header dependencies, which often
consists of over a thousand dependencies.
When little has changed in the tree, this is what make spends most of its
time doing, spending over a second, single-threaded when parsing larger
directory trees.
It performs considerably more work that is actually necessary to get the
job done.
This can be done faster in C, so implement scripts/basic/depcheck to do so.
It does as little work as possible, reading the .cmd files from a
directory's targets and running stat on each dependency only a single time.
It generates a fragment holding only the saved command line for a target
whose dependencies all exist and are older than it, and the entire .cmd
file for any other.
That way, Makefile.build simply includes the fragment and the amount of
work make has to do is significantly reduced when there is not much work to
do.
If the operation fails, kbuild falls back to using .cmd files.
The result is exactly the same as before.
For drivers/gpu/drm/amd/amdgpu in an allmodconfig tree the fragment is a
fifth of the size of the .cmd files, make reads it in 10ms instead of
380ms, and the instance goes from 2.2s to 0.4s.
Whole build, 128-thread Threadripper 9980X, best of N runs:
before after delta
-------------------------------
x86 defconfig, no-op make, gcc 0.92s 0.62s -0.29s (-32%)
x86 defconfig, no-op make, clang 1.1s 0.78s -0.31s (-29%)
x86 allmodconfig, no-op make, gcc 12.3s 11.2s -1.2s (-10%)
x86 allmodconfig, no-op make, clang 13.2s 11.9s -1.3s (-10%)
x86 allmodconfig, touch mm/vma.c, gcc 40.9s 40.6s -0.3s (-1%)
x86 allmodconfig, touch mm/vma.c, clang 37.6s 36.5s -1.0s (-3%)
Assisted-by: LLM
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Each kernel make invocation begins with ~30 compiler and linker runs each
of which performs duplicate probe for a number of compiler and linker
options.
This is useless work - the compiler and its version is known, so use these
to determine which options are available, once.
A convention already exists for this - CC_HAS_xxx, LD_HAS_xxx in Kconfig
files (for example, CC_HAS_COUNTED_BY), so convert these probes to Kconfig
options where appropriate.
With gcc and clang, defconfig and allmodconfig, the recorded command
lines are unchanged, a build with nothing to do rebuilds nothing and W=1
continues to function correctly.
Doing this improves all builds, but has a particularly positive impact on
no-op builds (builds where nothing has changed).
Whole build, 128-thread Threadripper 9980X, best of N runs:
before after delta
-------------------------------
x86 defconfig, no-op make, gcc 0.62s 0.46s -0.17s (-27%)
x86 defconfig, no-op make, clang 0.78s 0.53s -0.25s (-32%)
x86 defconfig, touch mm/vma.c, gcc 8.7s 8.5s -0.17s (-2%)
x86 defconfig, touch mm/vma.c, clang 7.9s 7.6s -0.25s (-3%)
x86 defconfig, clean, clang 27.0s 26.8s -0.26s (-1%)
x86 allmodconfig, no-op make, gcc 11.2s 11.0s -0.18s (-2%)
x86 allmodconfig, no-op make, clang 11.9s 11.6s -0.28s (-2%)
x86 allmodconfig, touch mm/vma.c, clang 36.5s 36.2s -0.31s (-1%)
Assisted-by: LLM
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
modpost spends a long time md4 hashing module source at a per-byte
granularity.
Fix this by doing this hashing per-file instead by accumulating a per-file
buffer in parse_file().
All 11,189 .mod.c files and Module.symvers were confirmed to be identical
with this change applied.
This is especially impactful for allmodconfig incremental builds (where
CONFIG_MODULE_SRCVERSION_ALL is set).
Whole build, 128-thread Threadripper 9980X, best of N runs:
before after delta
-------------------------------
x86 allmodconfig, touch mm/vma.c, gcc 40.6s 38.2s -2.4s (-6%)
x86 allmodconfig, touch mm/vma.c, clang 36.2s 35.0s -1.2s (-3%)
Assisted-by: LLM
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
For every relocation modpost invokes check_section_mismatch() to determine
whether there is any kind of mismatch between the source and destination,
and if so which classification applies.
Each time it does this it invokes section_mismatch() which iterates
through the sectioncheck[] array every time it's called.
When walked a relocation section the source is fixed and there aren't many
targets, so the same names are looked up over and over again.
Therefore cache not only mismatch categorisation but also whether a
mismatch even exists for a given section and look up the sections in the
cache.
Special indices (undefined, absolute, common) take the uncached path as
before.
This results in very significant speed ups for allmodconfig incremental
builds.
modpost is on the serial tail of every build that links vmlinux.
Whole build, 128-thread Threadripper 9980X, best of N runs:
before after delta
-------------------------------
x86 defconfig, touch mm/vma.c, gcc 8.6s 8.4s -0.16s (-2%)
x86 defconfig, touch mm/vma.c, clang 7.7s 7.5s -0.15s (-2%)
x86 allmodconfig, touch mm/vma.c, gcc 38.2s 33.5s -4.6s (-12%)
x86 allmodconfig, touch mm/vma.c, clang 35.0s 31.1s -4.0s (-11%)
Assisted-by: LLM
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
modpost generates a descriptor for every module in the form of a
<module>.mod.c file with .modinfo strings, the __this_module descriptor,
exported symbol tables and (with CONFIG_MODVERSIONS set), the CRC of
imported symbols.
These files are compiled like any other kernel C file with all of the
-include preamble, as well as including linux/module.h, header dependencies
generated by fixdep of a few hundred headers, an objtool run and if LTO is
being performed, a link is performed to generate native code.
On an x86-64 allmodconfig build 11,189 *.mod.c files are built, each
taking ~0.24s of CPU time to compile, and module finalisation as a whole
6,300 CPU seconds, or 64 seconds of wall time when run over 128 threads.
It also generates ~1.3 GiB of *.mod.o.cmd files that every subsequent build
has to read back.
Avoid all this by emitting the descriptors as assembly instead.
The layout required (size and alignment of struct module, struct
modversion_info, the module's name offsets, init, and exit fields and
whether the architecture uses PREL32 ksymtab references) can all be derived
from scripts/mod/module-offsets.h.
An assembly file avoids all of the issues previously mentioned so this
conversion results in a very significant performance win on kernel build.
As a consequence of this change, since module-offsets.c includes
linux/module.h, scripts/mod is now built after the generated headers in
prepare0, rather than before.
Also update .gitignore and make clean to handle .mod.S files, but keep
.mod.c files there to ensure that users do not end up with untracked
changes/dirty trees after the change takes effect.
The sections were confirmed to be byte-for-byte identical to the C version
produced - each of .modinfo, .gnu.linkonce.this_module, __ksymtab*,
__ksymtab_strings, __kcrctab*, __kflagstab*, __versions,
__version_ext_crcs, __version_ext_names and their relocations - for all
8,135 modules of a clang allmodconfig build with CONFIG_COMPILE_TEST off
and CONFIG_MODVERSIONS, CONFIG_EXTENDED_MODVERSIONS and
CONFIG_MODULE_SRCVERSION_ALL on, and for a sample built with gcc.
What differs is what the compiler added around them: the __UNIQUE_ID_*
locals, the KASAN constructor for the .mod.c globals, and on x86 a
.note.gnu.property that the linker already drops from any module containing
an assembly file and the loader never reads.
None of these have any impact on the build, however.
x86_64 kernels built with gcc and clang, with CONFIG_MODVERSIONS,
CONFIG_EXTENDED_MODVERSIONS and CONFIG_MODULE_SRCVERSION_ALL, were booted,
every module and an external one loaded and unloaded, and the srcversions
checked against modinfo, i386 and arm64 defconfigs (gcc and clang) and a
ThinLTO build were also built as part of testing.
On the x86_64 allmodconfig with clang 22, "make modules" with every
*.mod.o and *.ko deleted goes from 64.5s (6,306 CPU-s) to 28.5s (518
CPU-s).
A consequence of this change is that the make jobs are now so small that
make cannot dispatch them quick enough, however the next commit in the
series addresses this issue.
This impacts clean and no-op builds with a large number of modules most
noticeably.
Whole build, 128-thread Threadripper 9980X, best of N runs:
before after delta
-------------------------------
x86 allmodconfig, no-op make, gcc 11.0s 1.9s -9.1s (-82%)
x86 allmodconfig, no-op make, clang 11.6s 2.4s -9.2s (-79%)
x86 allmodconfig, clean, gcc 342.1s 304.5s -37.6s (-11%)
x86 allmodconfig, clean, clang 338.5s 301.7s -36.8s (-11%)
Assisted-by: LLM
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Module finalisation on allmodconfig builds consists of a large number of
very short-lived jobs, and the make job dispatcher cannot possibly dispatch
jobs fast enough.
For allmodconfig x86-64 this can be on the order of ~22,000 jobs of a few
milliseconds in duration each.
However per-job cost grows with the variables the instance holds, here the
savedcmd_* of every .mod.o and .ko read back from the .cmd files, since it
walks them all to build each child's environment.
This makes module finalisation very inefficient when large numbers of
modules are being built.
Fix this by splitting modules.order into chunks of 128 at a time, run in
parallel.
Each instance holds only its own modules' variables and the top-level one
reads no per-module .cmd files at all, the same rules serve both levels,
and an instance is told its chunk with modfinal-first=<index>.
"make modules" with every *.mod.o and *.ko deleted goes from 28.9s to 15.9s
with clang 22. No-op "make modules" goes from 5.6s to 4.8s, as checking the
22,000 targets is spread over the chunks too.
Whole build, 128-thread Threadripper 9980X, best of N runs:
before after delta
-------------------------------
x86 allmodconfig, no-op make, gcc 1.9s 1.2s -0.74s (-39%)
x86 allmodconfig, no-op make, clang 2.4s 1.6s -0.77s (-32%)
x86 allmodconfig, clean, gcc 304.5s 291.4s -13.1s (-4%)
x86 allmodconfig, clean, clang 301.7s 297.0s -4.7s (-2%)
Assisted-by: LLM
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
modpost does a lot of single-threaded work hashing files from each object's
.cmd file.
This makes the build slower than it needs to be, so do this work in
parallel.
This is egregious for allmodconfig builds - for instance x86-64 can end up
opening 200,000 files individually and hashing them all serially.
Parallelise this operation by maintaining a thread pool for the hashing
work.
Combined with the per-file hashing commit this cuts modpost's run time
nearly in half for an allmodconfig build.
Module.symvers and every *.mod.S are byte for byte the same.
modpost is on the serial tail of every allmodconfig build, however
defconfig does not set CONFIG_MODULE_SRCVERSION_ALL and is unchanged.
Whole build, 128-thread Threadripper 9980X, best of N runs:
before after delta
-------------------------------
x86 allmodconfig, touch mm/vma.c, gcc 33.4s 30.2s -3.2s (-10%)
x86 allmodconfig, touch mm/vma.c, clang 31.1s 28.1s -3.0s (-10%)
Assisted-by: LLM
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Instruction relocations are looked up by destination in objtool via a hash
which is keyed on a 16-byte (OFFSET_STRIDE) window within the section being
walked.
It iterates through each 16-byte window, looking up relocations over
several passes, before moving on to the next 16-byte window, caching only
when a relocation is not found saving further lookups in this case.
Improve upon this by introducing a per-section relocation cache storing the
first relocation at or after each 64-byte window of data (an empirically
determined index range), indexed by chunk.
The lookup is implemented an array lookup and touches no shared state, so
can be used from multiple threads.
This relies upon the entries within a section being sorted, which is the
case for all sections supplied to objtool by the link step during the
kernel build.
For cases where sections are supplied out of order or grown one relocation
at a time (e.g. livepatch), fall back to using the existing hash mechanism.
DWARF sections are a special case - their relocations are never looked up
by destination at all and only need to be on their symbol's list for
elf_update_sym_relocs().
So do not index or hash DWARF relocations at all - however add a mechanism
such that if one were ever looked up, a linear scan will be used.
This is meaningful in practice as on an x86-64 kernel build with
CONFIG_DEBUG_INFO set objtool processing of vmlinux.o is dominated by DWARF
section processing.
For an allmodconfig build ~9 million relocations were hashed, and ~8.2
million of those were DWARF sections, which added overhead on cache miss
and pollution of the hash table. This is now eliminated.
The hash, when used, is read-mostly (every jump, call and memory operand)
and several relocations share a key due to the 16-byte stride, so keep the
hash sparse by scaling the hash by the number of objects to be hashed.
Another hotspot in objtool processing is determining 'dead end' functions,
i.e. functions which never return.
Implement a simple boolean per-function cache for this to avoid determining
this more than once per function.
The output of objtool before and after this change was confirmed to be
byte-for-byte identical both for x86_64 defconfig and allmodconfig with
gcc and clang.
objtool on vmlinux.o is on the serial tail of every build that links
vmlinux, no-op builds are unchanged.
Whole build, 128-thread Threadripper 9980X, best of N runs:
before after delta
-------------------------------
x86 defconfig, touch mm/vma.c, gcc 8.4s 8.1s -0.28s (-3%)
x86 defconfig, touch mm/vma.c, clang 7.5s 7.1s -0.41s (-5%)
x86 defconfig, clean, gcc 27.1s 26.8s -0.34s (-1%)
x86 defconfig, clean, clang 26.6s 26.2s -0.40s (-1%)
x86 allmodconfig, touch mm/vma.c, gcc 30.2s 28.4s -1.8s (-6%)
x86 allmodconfig, touch mm/vma.c, clang 28.1s 26.0s -2.1s (-7%)
Assisted-by: LLM
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
During a kernel build objtool is used to decode vmlinux.o's instructions
and resolve every jump and call destination.
This forms a large part of the work objtool does during the build process,
and it is all done in serial.
Decode these in parallel at a function granularity to speed things up.
Only the instruction hash is shared between the threads and nothing is ever
removed from it, so an insertion is a compare-and-swap on the bucket head.
Also update the instruction hash to be more efficient - it was previously
hardcoded to a size of 2^20 buckets.
This is insufficient for an x86-64 allmodconfig kernel where an x86-64
build decodes ~16 million instructions and in practice find_insn() was
observed working chains of four or more entries on lookup.
Instead size it based on the amount of text to be decoded at roughly one
bucket per instruction, identical to the relocation hash (set to
OFFSET_STRIDE).
This results in 2^20 buckets (~8 MiB memory usage) for a defconfig
vmlinux.o and 2^22 (~32 MiB memory usage) for an allmodconfig kernel, so it
is not an egregious use of memory.
Threads are only created for objects with 8 MiB or more of text, meaning
that runs involving smaller objects remain unaffected.
Only decoding and the branch passes are threaded, so the gain flattens out
at 16 threads and more only add overhead - on the clang allmodconfig
vmlinux.o objtool takes 5.58s on 1 thread, 3.96s on 8, 3.91s on 16 and
4.03s on 128. Cap the thread count at 16, or the number of CPUs if fewer.
The passes which resolve jump and call destinations and annotate call
sites all run over the same regions.
They only write to lists in struct objtool_file, so have each range work
with their own copy of this data structure, which are then joined, in range
order afterwards in a map-reduce fashion.
The output of objtool before and after this change was confirmed to be
byte-for-byte identical for x86_64 defconfig and allmodconfig with gcc and
clang, and for a loongarch defconfig, where objtool runs on every object.
On a 128-thread machine, objtool on the clang allmodconfig vmlinux.o goes
from 5.9s to 3.7s (6.5s to 3.7s together with the previous patch), and on
defconfig from 1.87s to 1.26s.
objtool on vmlinux.o is on the serial tail of every build that links
vmlinux, no-op builds are unchanged.
Whole build, 128-thread Threadripper 9980X, best of N runs:
before after delta
-------------------------------
x86 defconfig, touch mm/vma.c, gcc 8.1s 7.5s -0.65s (-8%)
x86 defconfig, touch mm/vma.c, clang 7.1s 6.6s -0.51s (-7%)
x86 defconfig, clean, gcc 26.8s 26.3s -0.46s (-2%)
x86 defconfig, clean, clang 26.2s 25.8s -0.45s (-2%)
x86 allmodconfig, touch mm/vma.c, gcc 28.4s 23.7s -4.7s (-17%)
x86 allmodconfig, touch mm/vma.c, clang 26.0s 22.1s -3.9s (-15%)
Assisted-by: LLM
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Rust crates are compiled in a serial chain on the critical path - core, bindings, kernel crates and then the drivers - each a rustc invocation. With CONFIG_RUST set and every rust driver enabled, the entire chain takes around 25 seconds on a 128-thread machine, with the associated C code taking 24 seconds. Each time a change is made to any of the bindings, it triggers a rebuild. Use the -Zthreads option to have rustc perform builds in parallel. This has been available since rust 1.84 (rust-lang/rust#132282), and the kernel requires rust 1.85 or above. A future -j/--jobs option is planned for rustc:(rust-lang/compiler-team#1005), so check to see if this available and if so use it. If the user's rustc supports neither, then it falls back gracefully and neither are used. The threads are taken from make's jobserver, so a parallel build is not oversubscribed. It was found that benefits level off at 8 threads (16 was found to be around the same, and 32 slower). Generated output was confirmed byte-for-byte identical. Observed build time changes (using rustc 1.98, clang): before after core.o 7.7s 4.9s bindings.o 4.3s 3.1s kernel.o 2.0s 1.4s clean build 39.3s 34.6s touch rust/kernel/lib.rs 12.9s 12.1s touch rust/bindings/bindings_helper.h 19.5s 17.5s Whole build, 128-thread Threadripper 9980X, best of N runs: before after delta ------------------------------- x86 defconfig+RUST, clean 41.0s 36.4s -4.6s (-11%) x86 defconfig+RUST, touch lib.rs 8.3s 7.7s -0.56s (-7%) Assisted-by: LLM Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
rust/exports.c includes the exports_*_generated.h listing the symbols of the core, bindings, kernel and helpers objects, but this is not expressed in its Makefile. This worked previously because the headers are always-y targets, and make prepare built all of the rust crates first, which is an implicit dependency. Fix this as it is a dependency of the subsequent commit. Assisted-by: LLM Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
When CONFIG_RUST is specified make prepare builds all of the rust
components before descending into the tree.
This means nothing can be done until all of these are compiled, resulting
in a stall at the start of every clean CONFIG_RUST build.
This also implicitly slows down every rust-capable LLVM allmodconfig build
as this enables the CONFIG_RUST option.
Fix this by allowing rust crates to be built alongside the C code.
Remove the rust build from the make prepare step, then establish a
dependency between rust code located elsewhere in the tree upon rust
components contained in rust/.
Do this by establishing a top-level list of directories containing rust
code, KBUILD_RUST_DIRS, upon which the dependency is expressed.
Finally, ensure that no rust code is located elsewhere and fail the build
if any is found to ensure that nothing is missed in future.
Care is taken to ensure the rust/ dependency is also established for any
module being built to account for external rust modules which rely upon it.
Every Rust and C object was confirmed to be byte-for-byte identical after
this change.
Whole build, 128-thread Threadripper 9980X, best of N runs:
before after delta
-------------------------------
x86 defconfig+RUST, clean 36.5s 32.6s -3.9s (-11%)
x86 allmodconfig, clean 297.0s 278.3s -18.7s (-6%)
Assisted-by: LLM
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
The gzip step of a kernel build is very lengthily, especially for larger
builds such as allmodconfig.
gzip itself cannot be run in parallel, however an alternative tool exists
that can, providing the same feature set as gzip itself - pigz - which
works as a drop-in replacement.
On a 128-core Threadripper, gzip -9 of a 36 MiB x86-64 vmlinux.bin takes
1.6s, and with pigz it takes 0.09s, so the performance increase is
significant.
It is already possible to specify the KGZIP environmental variable to make
use of pigz, however it seems sensible to make use of pigz if it is
available.
Therefore default to using pigz if it is available on the system upon which
the kernel is being built, otherwise fall back to gzip.
The output is byte-for-byte identical between pigz/gzip invocations, but
gzip and pigz do not produce the same stream as one another.
Therefore, for reproducible builds, the same set of tools should be used.
This seems to already be an implicit requirement in any case, but update
the reproducible build documentation to make this clear.
Also update the kbuild documentation to reflect the change.
Every x86 build ends with the compression of vmlinux.bin, 36MB for
defconfig and over 200MB for allmodconfig, no-op builds are unchanged.
Whole build, 128-thread Threadripper 9980X, best of N runs:
before after delta
-------------------------------
x86 defconfig, touch mm/vma.c, gcc 7.4s 5.4s -2.0s (-27%)
x86 defconfig, touch mm/vma.c, clang 6.6s 4.9s -1.7s (-26%)
x86 defconfig, clean, gcc 26.2s 24.4s -1.8s (-7%)
x86 defconfig, clean, clang 25.8s 24.3s -1.5s (-6%)
x86 allmodconfig, touch mm/vma.c, gcc 23.7s 15.3s -8.4s (-35%)
x86 allmodconfig, touch mm/vma.c, clang 22.1s 15.2s -6.9s (-31%)
x86 allmodconfig, clean, gcc 291.4s 275.2s -16.2s (-6%)
x86 allmodconfig, clean, clang 278.3s 265.7s -12.6s (-5%)
Link: https://zlib.net/pigz/
Assisted-by: LLM
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Author
|
Patch 1: "[01/23] scripts/mksysmap: drop the MODULE_INFO() symbols from kallsyms" |
Author
|
Patch 1: "[01/23] scripts/mksysmap: drop the MODULE_INFO() symbols from kallsyms" |
Author
|
Patch 1: "[01/23] scripts/mksysmap: drop the MODULE_INFO() symbols from kallsyms" |
Author
|
Patch 1: "[01/23] scripts/mksysmap: drop the MODULE_INFO() symbols from kallsyms" |
Author
|
Patch 1: "[01/23] scripts/mksysmap: drop the MODULE_INFO() symbols from kallsyms" |
Author
|
Patch 1: "[01/23] scripts/mksysmap: drop the MODULE_INFO() symbols from kallsyms" |
Author
|
Patch 1: "[01/23] scripts/mksysmap: drop the MODULE_INFO() symbols from kallsyms" |
Author
|
Patch 21: "[21/23] rust: make exports.o depend on the headers generated for it" |
Author
|
Patch 21: "[21/23] rust: make exports.o depend on the headers generated for it" |
Author
|
Patch 21: "[21/23] rust: make exports.o depend on the headers generated for it" |
Author
|
Patch 21: "[21/23] rust: make exports.o depend on the headers generated for it" |
Author
|
Patch 21: "[21/23] rust: make exports.o depend on the headers generated for it" |
Author
|
Patch 21: "[21/23] rust: make exports.o depend on the headers generated for it" |
Author
|
Patch 22: "[22/23] kbuild: build rust crates in parallel with the rest of the build" |
Author
|
Patch 22: "[22/23] kbuild: build rust crates in parallel with the rest of the build" |
Author
|
Patch 22: "[22/23] kbuild: build rust crates in parallel with the rest of the build" |
Author
|
Patch 22: "[22/23] kbuild: build rust crates in parallel with the rest of the build" |
Author
|
Patch 22: "[22/23] kbuild: build rust crates in parallel with the rest of the build" |
Author
|
Patch 22: "[22/23] kbuild: build rust crates in parallel with the rest of the build" |
Author
|
Patch 22: "[22/23] kbuild: build rust crates in parallel with the rest of the build" |
Author
|
Patch 22: "[22/23] kbuild: build rust crates in parallel with the rest of the build" |
Author
|
Patch 22: "[22/23] kbuild: build rust crates in parallel with the rest of the build" |
Author
|
Patch 22: "[22/23] kbuild: build rust crates in parallel with the rest of the build" |
Author
|
Patch 22: "[22/23] kbuild: build rust crates in parallel with the rest of the build" |
Author
|
Patch 22: "[22/23] kbuild: build rust crates in parallel with the rest of the build" |
Author
|
Patch 23: "[23/23] kbuild: use pigz for gzip compression if available" |
Author
|
Patch 23: "[23/23] kbuild: use pigz for gzip compression if available" |
Author
|
Patch 23: "[23/23] kbuild: use pigz for gzip compression if available" |
Author
|
Patch 23: "[23/23] kbuild: use pigz for gzip compression if available" |
Author
|
Patch 23: "[23/23] kbuild: use pigz for gzip compression if available" |
Author
|
Patch 23: "[23/23] kbuild: use pigz for gzip compression if available" |
Author
|
Patch 23: "[23/23] kbuild: use pigz for gzip compression if available" |
Author
|
Patch 23: "[23/23] kbuild: use pigz for gzip compression if available" |
Author
|
Patch 23: "[23/23] kbuild: use pigz for gzip compression if available" |
Author
|
Patch 23: "[23/23] kbuild: use pigz for gzip compression if available" |
Author
|
Patch 23: "[23/23] kbuild: use pigz for gzip compression if available" |
Author
|
Patch 23: "[23/23] kbuild: use pigz for gzip compression if available" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR for series 1160850 applied to workflow__riscv__fixes
Name: kbuild: significantly speed up kernel builds
URL: https://patchwork.kernel.org/series/1160850/
Version: 1