Place obj file in seperate dir

[comicfans]

without --objdir-as-tempdir, nvcc will put temp cpp file under shared /tmp directory, while multi compile running in parallel, they create non-deterministic file. using

llvm-nm -a cuda_library.a |grep tmp

it might shows
tmpxft_00000002_00000000-6_kernel.cudafe1.cpp or
tmpxft_00000002_00000001-6_kernel.cudafe1.cpp

apply --objdir-as-tempdir arguments fix this by place stable file under object directory , it explicitly declared same file compiled with different options shouldn't be in same directory. rules_cuda now placed object file and pic object file under same directory, which might break --objdir-as-tempdir.

this PR address this by place output under different category folder.

Note: test is written by codex, revert place-file-under-category commit confirms the test failed as expected.

[cloudhan]

See #372 (comment)

IIRC, objtemp is not mandatory to achieve reproducible build. The key is the random seed thing.

[comicfans]

See #372 (comment)

IIRC, objtemp is not mandatory to achieve reproducible build. The key is the random seed thing.

Thanks for the info. seems nvcc_fixed_random_seed_feature only works with cuda > 12.9 ? I'm working with 12.6 so not sure how this affect build behavior. according to https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/#frandom-seed-frandom-seed it affect symbol/variable name generation. and --objdump affect temp file path embedded in achive file so both should be required. for my 12.6 cuda environment, set --objtemp and -Xcompiler=-fdebug-compilation-dir=. flags gives same hash. but this might not as comprehensive as #372

[cloudhan]

prior to 12.9, there is no way to stably make the build reproducible, IIUC. Because nvcc almost always encode full path info in the final artifact, and objtemp itself is not stable...

[comicfans]

prior to 12.9, there is no way to stably make the build reproducible, IIUC. Because nvcc almost always encode full path info in the final artifact, and objtemp itself is not stable...

I did a quick check by running strings ( grep '/' and cpp) on the generated library.a and looking for embedded absolute paths. before adding --objtemp and -Xcompiler=-fdebug-compilation-dir=. my cuda library.a contains following strings:

/               0           0     0     0       538       `
//                                              20        `
fused_kernel.pic.o/
/0              0           0     0     644     63824     `
clang version 20.1.0 (https://github.com/llvm/llvm-project 24a30daaa559829ad079f2ff7f73eb4e18095f88)
/tmp/tmpxft_00000002_00000000-6_fused_kernel.cudafe1.cpp
/home/xinyuwang/.cache/bazel/_bazel_xinyuwang/06ac47915257f6c9e982f482518f8fc1/sandbox/linux-sandbox/696/execroot/_main
clang version 20.1.0 (https://github.com/llvm/llvm-project 24a30daaa559829ad079f2ff7f73eb4e18095f88)
/home/xinyuwang/.cache/bazel/_bazel_xinyuwang/06ac47915257f6c9e982f482518f8fc1/sandbox/linux-sandbox/696/execroot/_main
proj/dir/csrc
external/rules_cuda++toolchain+cuda_cudart_v12.6.77/cudart/include
/tmp
external/rules_cuda++toolchain+cuda_nvcc_v12.6.85/nvcc/include/crt
/tmp/tmpxft_00000002_00000000-6_fused_kernel.cudafe1.cpp
tmpxft_00000002_00000000-6_fused_kernel.cudafe1.cpp

After adding those options, the corresponding paths became relative:

/               0           0     0     0       538       `
//                                              20        `
fused_kernel.pic.o/
/0              0           0     0     644     63808     `
clang version 20.1.0 (https://github.com/llvm/llvm-project 24a30daaa559829ad079f2ff7f73eb4e18095f88)
bazel-out/k8-fastbuild/bin/proj/dir/_objs/fused_kernel/pic_object_file/fused_kernel.pic.o.cudafe1.cpp
clang version 20.1.0 (https://github.com/llvm/llvm-project 24a30daaa559829ad079f2ff7f73eb4e18095f88)
proj/dir/csrc
external/rules_cuda++toolchain+cuda_cudart_v12.6.77/cudart/include
bazel-out/k8-fastbuild/bin/proj/dir/_objs/fused_kernel/pic_object_file
external/rules_cuda++toolchain+cuda_nvcc_v12.6.85/nvcc/include/crt
bazel-out/k8-fastbuild/bin/proj/dir/_objs/fused_kernel/pic_object_file/fused_kernel.pic.o.cudafe1.cpp
fused_kernel.pic.o.cudafe1.cpp

IIUC, the bazel-out/k8-fastbuild/bin prefix should remain stable as long as the same configuration is used, so these paths should be treated as deterministic, or did I miss something ?

When you say that objtemp itself is not stable prior to CUDA 12.9, do you mean that using -objtemp may cause nvcc to crash or produce incorrect output, or that it can still generate nondeterministic artifacts under some conditions?

[cloudhan]

it can still generate nondeterministic artifacts.