AddressSanitizer AOT GPU tests failing on PVC with new offload model

[sarnex]

Describe the bug

https://github.com/intel/llvm/actions/runs/36034404391/job/107791962707?pr=23256

PVC new offload model only

    ********************
    FAIL: SYCL :: AddressSanitizer/aot/gpu.cpp (2517 of 2591)
    ******************** TEST 'SYCL :: AddressSanitizer/aot/gpu.cpp' FAILED ********************
    Exit Code: 1
    
    Command Output (stdout):
    --
    # RUN: at line 4
    /__w/llvm/llvm/toolchain/bin/dpclang++ -Werror  --offload-new-driver  -fsycl -fsycl-targets=spir64  /__w/llvm/llvm/llvm/sycl/test-e2e/AddressSanitizer/aot/gpu.cpp -Xarch_device -fsanitize=address -fsycl-targets=spir64_gen -Xsycl-target-backend=spir64_gen '-device pvc,dg2' -O0 -g /__w/llvm/llvm/llvm/sycl/test-e2e/AddressSanitizer/aot/Inputs/host-usm-oob.cpp -o /__w/llvm/llvm/build-e2e/AddressSanitizer/aot/Output/gpu.cpp.tmp.out
    # executed command: /__w/llvm/llvm/toolchain/bin/dpclang++ -Werror --offload-new-driver -fsycl -fsycl-targets=spir64 /__w/llvm/llvm/llvm/sycl/test-e2e/AddressSanitizer/aot/gpu.cpp -Xarch_device -fsanitize=address -fsycl-targets=spir64_gen -Xsycl-target-backend=spir64_gen '-device pvc,dg2' -O0 -g /__w/llvm/llvm/llvm/sycl/test-e2e/AddressSanitizer/aot/Inputs/host-usm-oob.cpp -o /__w/llvm/llvm/build-e2e/AddressSanitizer/aot/Output/gpu.cpp.tmp.out
    # .---command stderr------------
    # | clang-linker-wrapper: /__w/llvm/llvm/src/llvm/lib/Support/Path.cpp:875: void llvm::sys::fs::createUniquePath(const Twine &, SmallVectorImpl<char> &, bool): Assertion `llvm::is_contained(ModelStorage, '%') && "createUniquePath: Model must contain at least one '%'"' failed.
    # | PLEASE submit a bug report to https://github.com/intel/llvm/issues and include the crash backtrace and instructions to reproduce the bug.
    # | Stack dump:
    # | 0.	Program arguments: /__w/llvm/llvm/toolchain/bin/clang-linker-wrapper --device-compiler=spir64-unknown-unknown=-g --device-compiler=spir64-unknown-unknown=-ftarget-register-alloc-mode=pvc:-ze-intel-enable-auto-large-GRF-mode --device-compiler=spir64_gen-unknown-unknown=-options "--device-compiler=spir64_gen-unknown-unknown=-g -cl-opt-disable" --host-triple=x86_64-unknown-linux-gnu -sycl-device-library-location=/__w/llvm/llvm/toolchain/bin/../lib/dpcpp-7/sycl --sycl-post-link-options=-O2 --sycl-post-link-options=-device-globals --llvm-spirv-options=-spirv-max-version=1.5 --device-compiler=sycl:spir64_gen-unknown-unknown=-device --device-compiler=sycl:spir64_gen-unknown-unknown=pvc,dg2 --linker-path=/usr/bin/ld -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o /__w/llvm/llvm/build-e2e/AddressSanitizer/aot/Output/gpu.cpp.tmp.out /lib/x86_64-linux-gnu/crt1.o /lib/x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/14/crtbegin.o -L/__w/llvm/llvm/toolchain/lib/clang/24/lib/x86_64-unknown-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/14 -L/usr/lib/gcc/x86_64-linux-gnu/14/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib64 -L/__w/llvm/llvm/toolchain/bin/../lib -L/lib -L/usr/lib -L/runtimes/oneapi-tbb/env/../lib/intel64/gcc4.8 -L/usr/local/cuda/lib64/stubs /tmp/lit-tmp-4a8gc6ho/gpu-336aaf.o /tmp/lit-tmp-4a8gc6ho/host-usm-oob-05a369.o /__w/llvm/llvm/toolchain/bin/../lib/libsycl.so /__w/llvm/llvm/toolchain/bin/../lib/libsycl-devicelib-host.a -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-linux-gnu/14/crtend.o /lib/x86_64-linux-gnu/crtn.o
    # | Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
    # | 0  clang-linker-wrapper 0x00000000016716cb llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 43
    # | 1  clang-linker-wrapper 0x000000000166e5a1 llvm::sys::RunSignalHandlers() + 369
    # | 2  clang-linker-wrapper 0x00000000016724eb
    # | 3  libc.so.6            0x0000740120f83330
    # | 4  libc.so.6            0x0000740120fdcc0c pthread_kill + 284
    # | 5  libc.so.6            0x0000740120f8327e gsignal + 30
    # | 6  libc.so.6            0x0000740120f668ff abort + 223
    # | 7  libc.so.6            0x0000740120f6681b
    # | 8  libc.so.6            0x0000740120f79517
    # | 9  clang-linker-wrapper 0x00000000016643d1
    # | 10 clang-linker-wrapper 0x00000000016645ca
    # | 11 clang-linker-wrapper 0x0000000001664aef
    # | 12 clang-linker-wrapper 0x0000000001664bf1 llvm::sys::fs::createTemporaryFile(llvm::Twine const&, llvm::StringRef, llvm::SmallVectorImpl<char>&, llvm::sys::fs::OpenFlags) + 33
    # | 13 clang-linker-wrapper 0x000000000082d3c8
    # | 14 clang-linker-wrapper 0x000000000083938f
    # | 15 clang-linker-wrapper 0x0000000000837752
    # | 16 clang-linker-wrapper 0x000000000081faf9
    # | 17 clang-linker-wrapper 0x000000000081893a main + 4218
    # | 18 libc.so.6            0x0000740120f681ca
    # | 19 libc.so.6            0x0000740120f6828b __libc_start_main + 139
    # | 20 clang-linker-wrapper 0x0000000000817185 _start + 37
    # | Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
    # | 0  clang-linker-wrapper 0x00000000016716cb llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 43
    # | 1  clang-linker-wrapper 0x000000000166e5d4 llvm::sys::RunSignalHandlers() + 420
    # | 2  clang-linker-wrapper 0x00000000016724eb
    # | 3  libc.so.6            0x0000740120f83330
    # | 4  libc.so.6            0x0000740120fdcc0c pthread_kill + 284
    # | 5  libc.so.6            0x0000740120f8327e gsignal + 30
    # | 6  libc.so.6            0x0000740120f668ff abort + 223
    # | 7  libc.so.6            0x0000740120f6681b
    # | 8  libc.so.6            0x0000740120f79517
    # | 9  clang-linker-wrapper 0x00000000016643d1
    # | 10 clang-linker-wrapper 0x00000000016645ca
    # | 11 clang-linker-wrapper 0x0000000001664aef
    # | 12 clang-linker-wrapper 0x0000000001664bf1 llvm::sys::fs::createTemporaryFile(llvm::Twine const&, llvm::StringRef, llvm::SmallVectorImpl<char>&, llvm::sys::fs::OpenFlags) + 33
    # | 13 clang-linker-wrapper 0x000000000082d3c8
    # | 14 clang-linker-wrapper 0x000000000083938f
    # | 15 clang-linker-wrapper 0x0000000000837752
    # | 16 clang-linker-wrapper 0x000000000081faf9
    # | 17 clang-linker-wrapper 0x000000000081893a main + 4218
    # | 18 libc.so.6            0x0000740120f681ca
    # | 19 libc.so.6            0x0000740120f6828b __libc_start_main + 139
    # | 20 clang-linker-wrapper 0x0000000000817185 _start + 37
    # | clang++: error: unable to execute command: Aborted (core dumped)
    # | clang++: error: linker command failed with exit code -2 (use -v to see invocation)
    # | dpclang++: failed to execute '/__w/llvm/llvm/toolchain/bin/clang++': Aborted (core dumped)
    # `-----------------------------
    # error: command failed with exit status: 1
    
    --
    
    ********************
    FAIL: SYCL :: AddressSanitizer/common/kernel-filter-aot-gpu.cpp (2519 of 2591)
    ******************** TEST 'SYCL :: AddressSanitizer/common/kernel-filter-aot-gpu.cpp' FAILED ********************
    Exit Code: 1
    
    Command Output (stdout):
    --
    # RUN: at line 4
    /__w/llvm/llvm/toolchain/bin/dpclang++ -Werror  --offload-new-driver  -fsycl -fsycl-targets=spir64  /__w/llvm/llvm/llvm/sycl/test-e2e/AddressSanitizer/common/kernel-filter-aot-gpu.cpp -Xarch_device -fsanitize=address -fsycl-targets=spir64_gen -Xsycl-target-backend=spir64_gen '-device pvc,dg2' /__w/llvm/llvm/llvm/sycl/test-e2e/AddressSanitizer/common/kernel-filter.cpp -g -O2 -fsanitize-ignorelist=/__w/llvm/llvm/llvm/sycl/test-e2e/AddressSanitizer/common/ignorelist.txt -o /__w/llvm/llvm/build-e2e/AddressSanitizer/common/Output/kernel-filter-aot-gpu.cpp.tmp1
    # executed command: /__w/llvm/llvm/toolchain/bin/dpclang++ -Werror --offload-new-driver -fsycl -fsycl-targets=spir64 /__w/llvm/llvm/llvm/sycl/test-e2e/AddressSanitizer/common/kernel-filter-aot-gpu.cpp -Xarch_device -fsanitize=address -fsycl-targets=spir64_gen -Xsycl-target-backend=spir64_gen '-device pvc,dg2' /__w/llvm/llvm/llvm/sycl/test-e2e/AddressSanitizer/common/kernel-filter.cpp -g -O2 -fsanitize-ignorelist=/__w/llvm/llvm/llvm/sycl/test-e2e/AddressSanitizer/common/ignorelist.txt -o /__w/llvm/llvm/build-e2e/AddressSanitizer/common/Output/kernel-filter-aot-gpu.cpp.tmp1
    # .---command stderr------------
    # | clang-linker-wrapper: /__w/llvm/llvm/src/llvm/lib/Support/Path.cpp:875: void llvm::sys::fs::createUniquePath(const Twine &, SmallVectorImpl<char> &, bool): Assertion `llvm::is_contained(ModelStorage, '%') && "createUniquePath: Model must contain at least one '%'"' failed.
    # | PLEASE submit a bug report to https://github.com/intel/llvm/issues and include the crash backtrace and instructions to reproduce the bug.
    # | Stack dump:
    # | 0.	Program arguments: /__w/llvm/llvm/toolchain/bin/clang-linker-wrapper --device-compiler=spir64-unknown-unknown=-g --device-compiler=spir64-unknown-unknown=-ftarget-register-alloc-mode=pvc:-ze-intel-enable-auto-large-GRF-mode --device-compiler=spir64_gen-unknown-unknown=-options --device-compiler=spir64_gen-unknown-unknown=-g --host-triple=x86_64-unknown-linux-gnu -sycl-device-library-location=/__w/llvm/llvm/toolchain/bin/../lib/dpcpp-7/sycl --sycl-post-link-options=-O2 --sycl-post-link-options=-device-globals --llvm-spirv-options=-spirv-max-version=1.5 --device-compiler=sycl:spir64_gen-unknown-unknown=-device --device-compiler=sycl:spir64_gen-unknown-unknown=pvc,dg2 --linker-path=/usr/bin/ld -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o /__w/llvm/llvm/build-e2e/AddressSanitizer/common/Output/kernel-filter-aot-gpu.cpp.tmp1 /lib/x86_64-linux-gnu/crt1.o /lib/x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/14/crtbegin.o -L/__w/llvm/llvm/toolchain/lib/clang/24/lib/x86_64-unknown-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/14 -L/usr/lib/gcc/x86_64-linux-gnu/14/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib64 -L/__w/llvm/llvm/toolchain/bin/../lib -L/lib -L/usr/lib -L/runtimes/oneapi-tbb/env/../lib/intel64/gcc4.8 -L/usr/local/cuda/lib64/stubs /tmp/lit-tmp-4a8gc6ho/kernel-filter-aot-gpu-a77306.o /tmp/lit-tmp-4a8gc6ho/kernel-filter-3fb838.o /__w/llvm/llvm/toolchain/bin/../lib/libsycl.so /__w/llvm/llvm/toolchain/bin/../lib/libsycl-devicelib-host.a -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-linux-gnu/14/crtend.o /lib/x86_64-linux-gnu/crtn.o
    # | Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
    # | 0  clang-linker-wrapper 0x00000000016716cb llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 43
    # | 1  clang-linker-wrapper 0x000000000166e5a1 llvm::sys::RunSignalHandlers() + 369
    # | 2  clang-linker-wrapper 0x00000000016724eb
    # | 3  libc.so.6            0x00007f6f788b4330
    # | 4  libc.so.6            0x00007f6f7890dc0c pthread_kill + 284
    # | 5  libc.so.6            0x00007f6f788b427e gsignal + 30
    # | 6  libc.so.6            0x00007f6f788978ff abort + 223
    # | 7  libc.so.6            0x00007f6f7889781b
    # | 8  libc.so.6            0x00007f6f788aa517
    # | 9  clang-linker-wrapper 0x00000000016643d1
    # | 10 clang-linker-wrapper 0x00000000016645ca
    # | 11 clang-linker-wrapper 0x0000000001664aef
    # | 12 clang-linker-wrapper 0x0000000001664bf1 llvm::sys::fs::createTemporaryFile(llvm::Twine const&, llvm::StringRef, llvm::SmallVectorImpl<char>&, llvm::sys::fs::OpenFlags) + 33
    # | 13 clang-linker-wrapper 0x000000000082d3c8
    # | 14 clang-linker-wrapper 0x000000000083938f
    # | 15 clang-linker-wrapper 0x0000000000837752
    # | 16 clang-linker-wrapper 0x000000000081faf9
    # | 17 clang-linker-wrapper 0x000000000081893a main + 4218
    # | 18 libc.so.6            0x00007f6f788991ca
    # | 19 libc.so.6            0x00007f6f7889928b __libc_start_main + 139
    # | 20 clang-linker-wrapper 0x0000000000817185 _start + 37
    # | Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
    # | 0  clang-linker-wrapper 0x00000000016716cb llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 43
    # | 1  clang-linker-wrapper 0x000000000166e5d4 llvm::sys::RunSignalHandlers() + 420
    # | 2  clang-linker-wrapper 0x00000000016724eb
    # | 3  libc.so.6            0x00007f6f788b4330
    # | 4  libc.so.6            0x00007f6f7890dc0c pthread_kill + 284
    # | 5  libc.so.6            0x00007f6f788b427e gsignal + 30
    # | 6  libc.so.6            0x00007f6f788978ff abort + 223
    # | 7  libc.so.6            0x00007f6f7889781b
    # | 8  libc.so.6            0x00007f6f788aa517
    # | 9  clang-linker-wrapper 0x00000000016643d1
    # | 10 clang-linker-wrapper 0x00000000016645ca
    # | 11 clang-linker-wrapper 0x0000000001664aef
    # | 12 clang-linker-wrapper 0x0000000001664bf1 llvm::sys::fs::createTemporaryFile(llvm::Twine const&, llvm::StringRef, llvm::SmallVectorImpl<char>&, llvm::sys::fs::OpenFlags) + 33
    # | 13 clang-linker-wrapper 0x000000000082d3c8
    # | 14 clang-linker-wrapper 0x000000000083938f
    # | 15 clang-linker-wrapper 0x0000000000837752
    # | 16 clang-linker-wrapper 0x000000000081faf9
    # | 17 clang-linker-wrapper 0x000000000081893a main + 4218
    # | 18 libc.so.6            0x00007f6f788991ca
    # | 19 libc.so.6            0x00007f6f7889928b __libc_start_main + 139
    # | 20 clang-linker-wrapper 0x0000000000817185 _start + 37
    # | clang++: error: unable to execute command: Aborted (core dumped)
    # | clang++: error: linker command failed with exit code -2 (use -v to see invocation)
    # | dpclang++: failed to execute '/__w/llvm/llvm/toolchain/bin/clang++': Aborted (core dumped)
    # `-----------------------------
    # error: command failed with exit status: 1

To reproduce

1. Include a code snippet that is as short as possible
2. Specify the command which should be used to compile the program
3. Specify the command which should be used to launch the program
4. Indicate what is wrong and what was expected

Environment

No response

Additional context

No response

[bader]

This looks like a regression. @sarnex, can we find the commit that introduced this?

[sarnex]

i can do it early next week, it's EOD here and i'm OOO tomorrow, someone can do it sooner if they have time

[sarnex]

ok im back, i will work on it today

[sarnex]

first guilty commit is 0ebdb74 but it's not a real regression, that commit changed the compile flags from

/llvm/build/bin/clang++ -Werror  --offload-new-driver  -fsycl -fsycl-targets=spir64  /llvm/sycl/test-e2e/AddressSanitizer/aot/gpu.cpp -Xarch_device -fsanitize=address   -O0 -g /llvm/sycl/test-e2e/AddressSanitizer/aot/Inputs/host-usm-oob.cpp -o /llvm/build/tools/sycl/test-e2e/AddressSanitizer/aot/Output/gpu.cpp.tmp.out

to

/llvm/build/bin/clang++ -Werror --offload-new-driver -fsycl -fsycl-targets=spir64 /llvm/sycl/test-e2e/AddressSanitizer/aot/gpu.cpp -Xarch_device -fsanitize=address -fsycl-targets=spir64_gen -Xsycl-target-backend=spir64_gen '-device pvc,dg2' -O0 -g /llvm/sycl/test-e2e/AddressSanitizer/aot/Inputs/host-usm-oob.cpp -o /llvm/build/tools/sycl/test-e2e/AddressSanitizer/aot/Output/gpu.cpp.tmp.out

previously it wasn't even doing AOT. The change just exposed this existing issue. I will add this tracker to our new offload meeting tracker, someone needs to investigate.