transmutability: Stall new solver when inference variables remain

[lapla-cogito]

fixes #141400, fixes #148809

The new trait solver's consider_builtin_transmute_candidate() was passing types with inference variables directly to rustc_transmute(), which attempts layout computation and ICEs.

The old solver already guards against this in assemble_candidates_for_transmutability() by setting candidates.ambiguous = true. Add the equivalent guard in the new solver by returning forced_ambiguity when the predicate contains non-region inference variables.

[rustbot]

r? @lcnr

rustbot has assigned @lcnr.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[lcnr]

It seems wrong to return an error here. It should not be impossible to transmute. We should instead stall somehow here. There should probably just be an Answer::Maybe or sth to deal with cases where there are still inference variable to get the trait solver to try again later

[rust-bors]

☔ The latest upstream changes (presumably #151003) made this pull request unmergeable. Please resolve the merge conflicts.

[reddevilmidzy]

@lapla-cogito any updates on this? thanks.

[lapla-cogito]

I will update this patch in a few days.

[rustbot]

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

[lapla-cogito]

There should probably just be an Answer::Maybe or sth to deal with cases where there are still inference variable to get the trait solver to try again later

Since adding Answer::Maybe would ultimately result in the trait solver converting it to Certainty::AMBIGUOUS, I determined that this would only increase indirect references. That's why I implemented the changes as they are now (another reason being that the old solver already handles this conversion in a similar way, see the updated PR description).

@rustbot ready

[lcnr]

can you move this check into ecx.is_transmutable? That should be able to just return Certainty::AMBIGUOUS which is then properly handled already

we want to handle structurally_normalize_const to return something ambiguous or to take something that references an infer var and the returned assume then does not as the infer var wasn't used.

[lapla-cogito]

Thanks, done.

[reddevilmidzy]

issue resolved in #154991

[lcnr]

View changes since the review

also need to check assume 🤔

[lapla-cogito]

Sorry for the delay, I've been thinking about this a bit, and now that 154991 has been merged, I'm not sure why we still need to check assume here. IIUC, assume can only become Unevaluated when GCE is enabled, but currently GCE and the next solver cannot be used together. Given that, I'm not sure whether there is any actual issue that could arise from not checking assume here.

Of course, if this is intended to account for some future change (for example, if GCE and the next solver were to become usable together although my understanding is that GCE is being superseded by things like GCA, so I'm not sure how realistic that scenario is), then I can see the rationale. Otherwise, it seems to me that simply adding a test case would be sufficient.

[Enselic]

issue resolved in #154991

Triage: Do you mean that this PR is not needed any longer?

[reddevilmidzy]

issue resolved in #154991

Triage: Do you mean that this PR is not needed any longer?

I think we just need to add the test

[lcnr]

@rustbot author

#149228 (comment) is also still relevant

[rustbot]

Reminder, once the PR becomes ready for a review, use @rustbot ready.