CI: dist-s390x-linux build went from 40min. to 160min with new LLVM pass manager

[hkratz]

CI time quadrupled when the new pass manager was enabled:

• Last CI before the new pass manager was enabled: https://github.com/rust-lang-ci/rust/runs/3707892000?check_suite_focus=true
• CI with the new pass manager was enabled: https://github.com/rust-lang-ci/rust/runs/3708445190?check_suite_focus=true

Right now that does not really matter much because Github's Apple CI runners use very old hardware and take about the same time, but once that is fixed this will block a faster CI. No idea if that platform has any significant userbase.

Maybe disable the new PM for that platform or report it upstream?

[workingjubilee]

This regression absolutely needs to be reported upstream, as LLVM dev talk has mentioned completely removing the old pass manager within 1~2 versions.

[hkratz]

Most time is apparently being spent compiling rustc_ast_lowering:

Building stage1 compiler artifacts (x86_64-unknown-linux-gnu -> s390x-unknown-linux-gnu)
...
[RUSTC-TIMING] rustc_ast_lowering test:false 6556.620

https://github.com/rust-lang-ci/rust/runs/3708445190?check_suite_focus=true#step:26:12286

This crate usually takes less than 30 seconds to build.

[hkratz]

The build timeout in #88379 (comment) might be caused by this as well.

[estebank]

Most time is apparently being spent compiling rustc_ast_lowering

That is very surprising and am wondering what we could be doing for that not to be O(n).

[Mark-Simulacrum]

cc @rust-lang/wg-llvm

I am a little tempted to suggest a patch that passes -Zno-new-pass-manager or w/e on s390x, but that seems pretty unfortunate as well, given the relatively quick deprecation timeline.

[camelid]

Is there anything I can do to unblock #88379? Or will it require the patch that @Mark-Simulacrum is suggesting?

[camelid]

Just checking, has the regression been reported upstream yet? I didn't see a report when I searched the LLVM Bugzilla. (I don't have an LLVM Bugzilla account, so I can't report it.)

[cuviper]

I'm going to try to find the culprit, but in the meantime to unblock folks that hack could be an extra target condition in should_use_new_llvm_pass_manager. Note that the host arch is apparently not the issue, since these are all running on x86_64, cross-compiling to various targets.

[cuviper]

I have the dist-s390x-linux docker build running locally, with LLVM debuginfo, and perf top shows a couple hotspots.

First it was sitting here at about 31% of the samples:

llvm::Instruction* const* llvm::SmallVectorTemplateCommon<llvm::Instruction*, void>::reserveForParamAndGetAddressImpl<llvm::SmallVectorTemplateBase<llvm::Instruction*, true> >(llvm::SmallVectorTemplateBase<llvm::Instruction*, true>*, llvm::Instruction* const&, unsigned long) at /checkout/src/llvm-project/llvm/include/llvm/ADT/SmallVector.h:217
 (inlined by) llvm::SmallVectorTemplateBase<llvm::Instruction*, true>::reserveForParamAndGetAddress(llvm::Instruction*&, unsigned long) at /checkout/src/llvm-project/llvm/include/llvm/ADT/SmallVector.h:522
 (inlined by) llvm::SmallVectorTemplateBase<llvm::Instruction*, true>::push_back(llvm::Instruction*) at /checkout/src/llvm-project/llvm/include/llvm/ADT/SmallVector.h:547
 (inlined by) PushDefUseChildren at /checkout/src/llvm-project/llvm/lib/Analysis/ScalarEvolution.cpp:4382
 (inlined by) llvm::ScalarEvolution::forgetLoop(llvm::Loop const*) at /checkout/src/llvm-project/llvm/lib/Analysis/ScalarEvolution.cpp:7487

Then it moved away from that to a new hotspot at 28%:

llvm::ilist_detail::node_options<llvm::Instruction, false, false, void>::const_pointer llvm::ilist_detail::NodeAccess::getValuePtr<llvm::ilist_detail::node_options<llvm::Instruction, false, false, void> >(llvm::ilist_node_impl<llvm::ilist_detail::node_options<llvm::Instruction, false, false, void> > const*) at /checkout/src/llvm-project/llvm/include/llvm/ADT/ilist_node.h:184
 (inlined by) llvm::ilist_detail::SpecificNodeAccess<llvm::ilist_detail::node_options<llvm::Instruction, false, false, void> >::getValuePtr(llvm::ilist_node_impl<llvm::ilist_detail::node_options<llvm::Instruction, false, false, void> > const*) at /checkout/src/llvm-project/llvm/include/llvm/ADT/ilist_node.h:229
 (inlined by) llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::Instruction, false, false, void>, true, true>::operator*() const at /checkout/src/llvm-project/llvm/include/llvm/ADT/ilist_iterator.h:139
 (inlined by) llvm::simple_ilist<llvm::Instruction>::back() const at /checkout/src/llvm-project/llvm/include/llvm/ADT/simple_ilist.h:141
 (inlined by) llvm::BasicBlock::getTerminator() const at /checkout/src/llvm-project/llvm/lib/IR/BasicBlock.cpp:151

Next it moved to this at 49%:

llvm::MachineBasicBlock** std::__copy_move<false, false, std::random_access_iterator_tag>::__copy_m<std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, llvm::MachineBasicBlock**>(std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, llvm::MachineBasicBlock**) at /usr/include/c++/5/bits/stl_algobase.h:340
 (inlined by) llvm::MachineBasicBlock** std::__copy_move_a<false, std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, llvm::MachineBasicBlock**>(std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, llvm::MachineBasicBlock**) at /usr/include/c++/5/bits/stl_algobase.h:402
 (inlined by) llvm::MachineBasicBlock** std::__copy_move_a2<false, std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, llvm::MachineBasicBlock**>(std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, llvm::MachineBasicBlock**) at /usr/include/c++/5/bits/stl_algobase.h:440
 (inlined by) llvm::MachineBasicBlock** std::copy<std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, llvm::MachineBasicBlock**>(std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, llvm::MachineBasicBlock**) at /usr/include/c++/5/bits/stl_algobase.h:472
 (inlined by) llvm::MachineBasicBlock** std::__uninitialized_copy<true>::__uninit_copy<std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, llvm::MachineBasicBlock**>(std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, llvm::MachineBasicBlock**) at /usr/include/c++/5/bits/stl_uninitialized.h:93
 (inlined by) llvm::MachineBasicBlock** std::uninitialized_copy<std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, llvm::MachineBasicBlock**>(std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, llvm::MachineBasicBlock**) at /usr/include/c++/5/bits/stl_uninitialized.h:126
 (inlined by) void llvm::SmallVectorTemplateBase<llvm::MachineBasicBlock*, true>::uninitialized_copy<std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, llvm::MachineBasicBlock**>(std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, llvm::MachineBasicBlock**) at /checkout/src/llvm-project/llvm/include/llvm/ADT/SmallVector.h:490
 (inlined by) void llvm::SmallVectorImpl<llvm::MachineBasicBlock*>::append<std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, void>(std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >) at /checkout/src/llvm-project/llvm/include/llvm/ADT/SmallVector.h:652
 (inlined by) llvm::MachineBasicBlock** llvm::SmallVectorImpl<llvm::MachineBasicBlock*>::insert<std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, void>(llvm::MachineBasicBlock**, std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >, std::reverse_iterator<__gnu_cxx::__normal_iterator<llvm::MachineBasicBlock**, std::vector<llvm::MachineBasicBlock*, std::allocator<llvm::MachineBasicBlock*> > > >) at /checkout/src/llvm-project/llvm/include/llvm/ADT/SmallVector.h:851
 (inlined by) llvm::LiveVariables::MarkVirtRegAliveInBlock(llvm::LiveVariables::VarInfo&, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::SmallVectorImpl<llvm::MachineBasicBlock*>&) at /checkout/src/llvm-project/llvm/lib/CodeGen/LiveVariables.cpp:112

It's still going, but I'm not sure this is helpful enough to keep watching this way...

[cuviper]

See also #89524, which is not s390x specific.