trans/codegen_c: Honor "fastcall"/"stdcall"/"sysv64"/"win64" ABI on GCC - #433
Open
rautyrauty wants to merge 1 commit into
Open
rautyrauty wants to merge 1 commit into
rautyrauty wants to merge 1 commit into
Conversation
This is necessary because mrustc previously ignored every non-default calling convention on GCC: emit_type_fn and emit_function_header only handled MSVC's "system" -> __stdcall. For any other ABI string the C output came out as plain cdecl on x86 (and the default SysV on x86_64), silently disagreeing with the declared Rust ABI. On i586 this breaks `psm` (used by rustc via `stacker` for deep typecheck recursion). Its `extern_item!` macro picks the ABI per architecture, and under `target_arch = "x86"` that is `extern "fastcall"`, matching the hand-written assembly stubs in `vendor/psm/src/arch/x86.s` which read arguments from %ecx and %edx. With mrustc's cdecl emission those registers held whatever happened to be there at the call site, and `rust_psm_on_stack` then jumped through a garbage callback pointer onto an unrelated mmap page - producing the SIGSEGV during the i586 stage-2 libcore compile, with PC at a no-symbol address inside a high-mmap region and corrupt return frames in the backtrace. The other architectures did not notice because the ABI the same macro picks for them is already what mrustc emits: plain C on aarch64, "sysv64" on x86_64 (the default C ABI on Linux) and "aapcs" on arm (likewise the default). "aapcs" is therefore left unhandled here rather than mapped to an attribute. Link: https://gcc.gnu.org/onlinedocs/gcc/x86-Function-Attributes.html
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is necessary because mrustc previously ignored every non-default calling convention on GCC: emit_type_fn and emit_function_header only handled MSVC's "system" -> __stdcall. For any other ABI string the C output came out as plain cdecl on x86 (and the default SysV on x86_64), silently disagreeing with the declared Rust ABI.
On i586 this breaks
psm(used by rustc viastackerfor deep typecheck recursion). Itsextern_item!macro picks the ABI per architecture, and undertarget_arch = "x86"that isextern "fastcall", matching the hand-written assembly stubs invendor/psm/src/arch/x86.swhich read arguments from %ecx and %edx. With mrustc's cdecl emission those registers held whatever happened to be there at the call site, andrust_psm_on_stackthen jumped through a garbage callback pointer onto an unrelated mmap page - producing the SIGSEGV during the i586 stage-2 libcore compile, with PC at a no-symbol address inside a high-mmap region and corrupt return frames in the backtrace.The other architectures did not notice because the ABI the same macro picks for them is already what mrustc emits: plain C on aarch64, "sysv64" on x86_64 (the default C ABI on Linux) and "aapcs" on arm (likewise the default). "aapcs" is therefore left unhandled here rather than mapped to an attribute.
Link: https://gcc.gnu.org/onlinedocs/gcc/x86-Function-Attributes.html