Enable Cargo's new build-dir layout by ranger-ross · Pull Request #155439 · rust-lang/rust

ranger-ross · 2026-04-17T13:50:00Z

This PR enables the new Cargo build-dir layout in boostrap builds with -Zbuild-dir-new-layout.

See: #t-infra/bootstrap > Has anyone tested `./x` with the new build-dir layout?

weihanglo · 2026-04-17T13:59:14Z

    // got a list of prefix/extensions and we basically just need to find the
-    // most recent file in the `deps` folder corresponding to each one.
-    let contents = target_deps_dir
+    // most recent file in the `build` folder corresponding to each one.


Not yet read the code by myself. Just wonder why bootstrap are doing this? And if bootstrap needs to do it, it should be a feature in Cargo, at least a (perma-)unstable feature.

View changes since the review

See the commit block around line 2374. Basically we need the non-uplifted name that contains the unique hash. The artifact notifications provide the uplifted name where possible.

ranger-ross · 2026-04-17T15:28:43Z

+    //
+    // Cargo's build folder is structured as `build/<pkg>/<hash>/out/<artifacts>` so
+    // we need to traverse multiple directory layers to get to actual files.
+    let read_dir = |path: &Path| path.read_dir().ok().into_iter().flatten().filter_map(Result::ok);


checks failed because if there are any lingering files/dirs in <build-dir>/<target>/build that are not using the new layout it panics.

Update the original code to not panic if we call .read_dir() on a non-directory. Lets see if CI is happy with this

View changes since the review

ranger-ross · 2026-04-19T12:01:59Z

Looks like the miri job timed out.
Out of curiosity, how stable is this job?

(Sorry, haven't touch the rl/r repo much so I wondering if the failure is related to the new layout or if its normal for spurious failures)

mati865 · 2026-04-19T12:48:31Z

x86_64 Linux jobs rarely hang, I've retried it to be sure.

ranger-ross · 2026-04-19T15:34:48Z

Okay it's hanged on both runs so it's probably the changes.

Is there a way for me to run this check on my machine? I was trying to parse through the GitHub actions to see what command is being run but I was struggling to find it.

weihanglo · 2026-04-19T15:46:14Z

@ranger-ross see https://rustc-dev-guide.rust-lang.org/tests/docker.html#testing-with-docker

cargo run --manifest-path src/ci/citool/Cargo.toml run-local x86_64-gnu-miri

This is the easiest way to run it locally.

ranger-ross · 2026-04-20T14:09:41Z

Okay, so I did some testing and locally it runs fine. (takes around ~20 mins on my machine)

After looking a bit closer at the job logs, it actually never got stuck.
It just take WAY longer than normal....
According to the github stats x86_64-gnu-miri normally takes ~58 mins to run, but we took over 6 hrs and hit the timeout.

Looking at the other jobs, most of them did not have a large perf regression (except x86_64-gnu-tools which was also pretty bad)

That is concerning... 😅

rustbot · 2026-04-27T15:15:47Z

Some changes occurred in compiler/rustc_codegen_cranelift

cc @bjorn3

The Miri subtree was changed

cc @rust-lang/miri

Some changes occurred in src/tools/rustfmt

cc @rust-lang/rustfmt

Some changes occurred in src/tools/compiletest

cc @jieyouxu

ranger-ross · 2026-04-27T15:27:49Z

I realized the issue is not with the speed of compilation, but rather that for some reason in CI the compiler (and many other units) are being re-built multiple times which slows everything down and times out.

I haven't figured out why yet. While testing the CI on my fork, I've noticed its inconsistent and sometimes everything only builds once. I'm still investigating why this could be happening. rust-lang/cargo#16345 comes to mind as something that might change this behavior, but I am still unsure whats happening.

I went ahead and pushed some new changes with more fixes for tests and cranelift.

I'll continue investigating this week, but any rl/r experts see any obvious issues with my changes lmk
@rustbot author

ytmimi · 2026-04-27T21:40:03Z

-    // Chop off `deps`.
+    // Chop off `out`.
+    me.pop();
+    // Chop off `<hash>`.
+    me.pop();
+    // Chop off `<pkgname>`.
+    me.pop();
+    // Chop off `build`.


Is this going to break things in rustfmt's CI when we do a subtree push?

View changes since the review

ahh yeah it would probably break, maybe better for me to remove that from this PR and raise a separate PR in the rustfmt repo?

I'm still trying to understand what changes we need in this PR verse what changes can be split out into smaller more targeted changes

In that case, I'd prefer we open up a PR to add the new cargo build dir layout support to rustfmt directly and verify that things are working before making any changes to the rustfmt subtree here in r-l/rust

fair enough. I tested on my fork and I think rust-lang/rust CI fails with out rustfmt changes.

I went ahead and opened rust-lang/rustfmt#6879

@ranger-ross If you're able to duplicate the rustfmt code changes you made in rust-lang/rustfmt#6879 here that would be great.

I added those commits to this PR. I also noticed that RUSTFLAGS: -D warning was missing from the github actions workflows in rust-lang/rust so I went ahead and added that while I was at it. I guess it doesn't matter for this repo but oh well.

We've recently made some changes to those CI files in rustfmt so it's possible that the RUSTFLAGS: -D warning changes just haven't been synced up yet, but I don't think there's any issue in explicitly including that here.

rust-bors · 2026-06-13T14:56:38Z

💔 Test for 8e04c17 failed: CI. Failed job:

try - x86_64-msvc-ext1 (web logs, enhanced plaintext logs)

rustbot · 2026-06-15T13:11:16Z

Some changes occurred in src/tools/cargo

cc @ehuss

ranger-ross · 2026-06-15T13:12:27Z

Added some extra logging to Cargo to get an idea of what is happening.

@bors try jobs=x86_64-msvc-ext1

Enable Cargo's new build-dir layout try-job: x86_64-msvc-ext1

rust-bors · 2026-06-15T15:29:49Z

💔 Test for 385b483 failed: CI. Failed job:

try - x86_64-msvc-ext1 (web logs, enhanced plaintext logs)

ranger-ross · 2026-06-15T17:37:43Z

@bors try jobs=x86_64-msvc-ext1

rust-bors · 2026-06-15T19:26:00Z

💔 Test for 9f3aa86 failed: CI. Failed job:

try - x86_64-msvc-ext1 (web logs, enhanced plaintext logs)

This also syncs `RUSTFLAGS: -D warnings` which is added in rust-lang/rustfmt

This allows compiletest to support the new Cargo `build-dir` layout which passes more `-L` flags as the `deps` dir has been split per build unit. This can be an issue on Windows as the max command size is fairly small.

This reduces the size of PATH on windows during macro expansion to avoid hitting windows limits. With the new Cargo build-dir layout this becomes more important as Cargo now passes more `-L` args which end up bloating PATH.

rustbot · 2026-06-19T14:09:22Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

ranger-ross · 2026-06-19T14:10:34Z

@bors try jobs=x86_64-msvc-ext1

rust-bors · 2026-06-19T16:39:47Z

☀️ Try build successful (CI)
Build commit: efd5bf1 (efd5bf1b4495b0894eace2381078876673f8b432)
Base parent: 0c1748c (0c1748ce7c56091b659757e488dbaf782b814137)

ranger-ross · 2026-06-19T16:44:42Z

@bors try jobs=x86_64-msvc-1

rust-bors · 2026-06-19T19:09:52Z

💔 Test for e07d099 failed: CI. Failed job:

try - x86_64-msvc-1 (web logs, enhanced plaintext logs)

rust-log-analyzer · 2026-06-19T19:09:59Z

The job x86_64-msvc-1 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

[RUSTC-TIMING] similar test:false 3.919
[RUSTC-TIMING] tidy test:true 8.844
[RUSTC-TIMING] tidy test:false 18.166
    Finished `release` profile [optimized] target(s) in 1m 37s
     Running unittests src\lib.rs (build\x86_64-pc-windows-msvc\bootstrap-tools\x86_64-pc-windows-msvc\release\build\tidy/3846a9382910394a\out\tidy-3846a9382910394a.exe)

running 34 tests
test alphabetical::tests::bless_multiline ... ok
test alphabetical::tests::bless_funny_numbers ... ok
test alphabetical::tests::bless_smoke ... ok
---
[TIMING:end] test::RustdocJSStd { build_compiler: Compiler { stage: 2, host: x86_64-pc-windows-msvc, forced_compiler: false }, target: x86_64-pc-windows-msvc } -- 60.330
[TIMING:start] test::RustdocJSNotStd { target: x86_64-pc-windows-msvc, compiler: Compiler { stage: 2, host: x86_64-pc-windows-msvc, forced_compiler: false } }
[TIMING:start] test::Compiletest { test_compiler: Compiler { stage: 2, host: x86_64-pc-windows-msvc, forced_compiler: false }, target: x86_64-pc-windows-msvc, mode: rustdoc-js, suite: "rustdoc-js", path: "tests/rustdoc-js", compare_mode: None }
##[group]Testing stage2 with compiletest suite=rustdoc-js mode=rustdoc-js (x86_64-pc-windows-msvc)
FATAL: failed to run "D:\\a\\rust\\rust\\build\\x86_64-pc-windows-msvc\\stage2\\bin\\rustc.exe" "-Cdebuginfo=0" "--print=all-target-specs-json" "-Zunstable-options"
--- stdout

--- stderr


thread 'main' (1108) panicked at src\tools\compiletest\src\common.rs:1207:9:
fatal error
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Bootstrap failed while executing `test --stage 2 --skip=compiler --skip=src`
Build completed unsuccessfully in 2:16:41
make: *** [Makefile:115: ci-msvc-py] Error 1
  local time: Fri Jun 19 19:09:38 CUT 2026
  network time: Fri, 19 Jun 2026 19:09:38 GMT
##[error]Process completed with exit code 2.
##[group]Run echo "disk usage:"
echo "disk usage:"

rustbot assigned bjorn3 Apr 17, 2026

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Apr 17, 2026

ranger-ross mentioned this pull request Apr 17, 2026

Stabilize build-dir layout v2 rust-lang/cargo#16807

Open

3 tasks

bjorn3 reviewed Apr 17, 2026

View reviewed changes

Comment thread src/bootstrap/src/core/build_steps/compile.rs Outdated

weihanglo reviewed Apr 17, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

ranger-ross force-pushed the cargo-new-build-dir-layout branch from bf07fb8 to eae2cbf Compare April 17, 2026 15:27

ranger-ross commented Apr 17, 2026

View reviewed changes

This comment was marked as outdated.

Sign in to view

ranger-ross force-pushed the cargo-new-build-dir-layout branch from eae2cbf to ad47c85 Compare April 27, 2026 15:15

rustbot added A-compiletest Area: The compiletest test runner A-testsuite Area: The testsuite used to check the correctness of rustc T-rustfmt Relevant to the rustfmt team, which will review and decide on the PR/issue. labels Apr 27, 2026

This comment has been minimized.

Sign in to view

ranger-ross changed the title ~~Enable Cargo's new build-dir layout in boostrap~~ Enable Cargo's new build-dir layout Apr 27, 2026

RalfJung reviewed Apr 27, 2026

View reviewed changes

Comment thread src/tools/miri/test-cargo-miri/run-test.py Outdated

RalfJung reviewed Apr 27, 2026

View reviewed changes

Comment thread src/tools/compiletest/src/runtest/run_make.rs Outdated

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 27, 2026

bjorn3 reviewed Apr 27, 2026

View reviewed changes

Comment thread compiler/rustc_codegen_cranelift/build_system/build_sysroot.rs

ytmimi reviewed Apr 27, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Jun 15, 2026

Auto merge of #155439 - ranger-ross:cargo-new-build-dir-layout, r=<try>

385b483

Enable Cargo's new build-dir layout try-job: x86_64-msvc-ext1

This comment has been minimized.

Sign in to view

ranger-ross mentioned this pull request Jun 16, 2026

Improved the test error messages when 'rustc -V' fails rust-lang/cargo#17108

Merged

This comment has been minimized.

Sign in to view

ranger-ross added 7 commits June 19, 2026 22:58

chore: Enable Cargo's new build-dir layout in boostrap

210119b

test: Enable Cargo's new layout in tests

676a208

test: Add support for cargo's build-dir v2 layout (rustfmt)

ca74f29

chore: Move CI to use Cargo's v2 layout (rustfmt)

6df137e

This also syncs `RUSTFLAGS: -D warnings` which is added in rust-lang/rustfmt

fix: Use arg files in bootstrap rustc and rustdoc

ef16686

fix: Use arg files in compiletest

872eb11

This allows compiletest to support the new Cargo `build-dir` layout which passes more `-L` flags as the `deps` dir has been split per build unit. This can be an issue on Windows as the max command size is fairly small.

This comment has been minimized.

Sign in to view

Uh oh!

Conversation

ranger-ross commented Apr 17, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

weihanglo Apr 17, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bjorn3 Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

ranger-ross Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ranger-ross commented Apr 19, 2026

Uh oh!

mati865 commented Apr 19, 2026

Uh oh!

ranger-ross commented Apr 19, 2026

Uh oh!

weihanglo commented Apr 19, 2026

Uh oh!

ranger-ross commented Apr 20, 2026

Uh oh!

This comment was marked as outdated.

rustbot commented Apr 27, 2026

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

ranger-ross commented Apr 27, 2026

Uh oh!

Uh oh!

ytmimi Apr 27, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ranger-ross Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

ytmimi Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

ranger-ross Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

ytmimi Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ranger-ross Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

ytmimi Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

rust-bors Bot commented Jun 13, 2026

Uh oh!

This comment has been minimized.

rustbot commented Jun 15, 2026

Uh oh!

This comment has been minimized.

ranger-ross commented Jun 15, 2026

Uh oh!

This comment has been minimized.

rust-bors Bot commented Jun 15, 2026

Uh oh!

This comment has been minimized.

ranger-ross commented Jun 15, 2026

Uh oh!

This comment has been minimized.

rust-bors Bot commented Jun 15, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

ranger-ross commented Apr 17, 2026 •

edited by rustbot

Loading

weihanglo Apr 17, 2026 •

edited by rustbot

Loading

ranger-ross Apr 17, 2026 •

edited

Loading

ytmimi Apr 27, 2026 •

edited by rustbot

Loading

ytmimi Apr 29, 2026 •

edited

Loading