repr(ordered_fields)

[RustyYato]

View all comments

Add repr(ordered_fields) and provide a migration path to switch users from repr(C) to repr(ordered_fields), then change the meaning of repr(C) in the next edition.

This RFC is meant to be an MVP, and any extensions (for example, adding more reprs) are not in scope. This is done to make it as easy as possible to accept this RFC and make progress on the issue of repr(C) serving two opposing roles.

Rendered

To avoid endless bikeshedding, I'll make a poll if this RFC is accepted with all the potential names for the new repr. If you have a new name, I'll add it to the list of names in the unresolved questions section, and will include it in the poll.

[clarfonthey]

Not to add too many extra colours to the list, but repr(consistent) feels like a good name for this, since the purpose is to provide a consistent layout that does not depend on generics, compiler version, or target. The important thing is just that it's consistent, not that it matches what C does.

(Note: those three things should cover every case I've seen that uses repr(C) that should use repr(ordered_fields), but please feel free to correct me if I missed anything.)

Whereas repr(C) is explicitly, match what C does.

Also, while it may be more technical than most users need to understand, it would be helpful if the RFC reiterated the current issues with repr(C) that we want to fix, and potential future differences between repr(C) and repr(ordered_fields) that could pop up. I've read some of them but am not 100% sure of the details, and it would be nice to keep as part of the RFC.

[Lokathor]

Just as a small point of style the Guide Level Explanation is usually "what would be written in the rust tutorial book", and the Reference Level Explanation is "what could be written into the Rust Reference". This isn't a strict requirement, but personally I'd like to see the Reference Level part written out. Using the present tense, as if the RFC was accepted and implemented.

[programmerjake]

Suggested change

	`repr(ordered_fields)` is a new representation that can be applied to `struct`, `enum`, and `union` to give them a consistent, cross-platform, and predictable in memory layout.
	`repr(ordered_fields)` is a new representation that can be applied to `struct`, `enum`, and `union` to give them a consistent, cross-platform, and predictable in-memory layout.

"cross-platform" -- the layout will differ when there are different layouts for struct members' types, in particular primitive types can have different alignments which changes the amount of padding.

e.g., #[repr(ordered_fields)] struct S(u8, f64); doesn't have the same layout on x86_64 and i686

[RustyYato]

Good point, this will need to be documented as a hazard in the ordered_fields docs. However, the repr itself will be cross-platform. For example, #[repr(ordered_fields)] struct Cross([u8; 3], SomeEnum); will be truly cross-platform (given that SomeEnum is!).

[moonheart08]

Not to add too many extra colours to the list, but repr(consistent) feels like a good name for this, since the purpose is to provide a consistent layout that does not depend on generics, compiler version, or target. The important thing is just that it's consistent, not that it matches what C does.

(Note: those three things should cover every case I've seen that uses repr(C) that should use repr(ordered_fields), but please feel free to correct me if I missed anything.)

Whereas repr(C) is explicitly, match what C does.

Also, while it may be more technical than most users need to understand, it would be helpful if the RFC reiterated the current issues with repr(C) that we want to fix, and potential future differences between repr(C) and repr(ordered_fields) that could pop up. I've read some of them but am not 100% sure of the details, and it would be nice to keep as part of the RFC.

Just voicing support for repr(consistent) as naming.
Aside from the above, it more clearly hones in on the primary promises of the RFC, which is not just ordering but also exact type representation for things like enums. Field ordering is not the only thing it promises.

[joshtriplett]

Nominating this so that we can do a preliminary vibe-check on it in a lang triage meeting.

[scottmcm]

Big fan of doing this split, especially for structs. (It's less obvious what choices to make for other things, IMHO, but at least for structs this is something I've wanted for ages, so that for example Layout::extend can talk about it instead of C.)

Pondering the bikeshed: declaration_order or something could also be used to directly say what you're getting.

(This could be contrasted with other potential reprs that I wouldn't expect this RFC to add, but could consider as future work, like a deterministic_by_size_and_alignment where some restricted set of optimizations are allowed but you can be sure that usize and NonNull<String> can be mixed between different types while still getting the "same" field offsets, for example.)

[RustyYato]

I think this is also useful for unions, so we don't need to rely on repr(C) to ensure that all fields of a union are at offset 0.

This could be contrasted with other potential reprs that I wouldn't expect this RFC to add...

This also works as an argument against names like repr(consistent), since there are multiple consistent and useful repr, making it not descriptive enough.

[scottmcm]

I do think that declaration_order or ordered_fields is a bit weird on a union, because of course they're not really in any "order".

It makes me ponder whether we should just have repr(offset_zero) for unions to be explicit about it, or something.

(Which makes me think of other things like addressing rust-lang/unsafe-code-guidelines#494 by having a different constructs for "bag of maybeuninit stuff that overlap" vs "distinct options with an active-variant rule for enum-but-without-stored-discriminant". But those are definitely not this RFC.)

[RustyYato]

I don't mind spelling it as repr(offset_zero) for unions if that helps get this RFC accepted 😄. However, I have a sneaking suspicion that this isn't the contentious part of this RFC.
I know the name isn't optimal (intentionally). This can be hashed out after the RFC is accepted (or even give a different name for all of struct, union, and enum).
The most important bit for me is just that we do the split (for all of struct, union, and enum, to be consistent).

[RustyYato]

I've updated how enums's tags are specified, now they just defer to whatever repr(C)'s tag type is. This is done to reduce the friction of switching from repr(C) to repr(ordered_fields). To ensure that all uses of repr(ordered_fields) can be cross-platform, I've adding a lint to ensure that the user also adds an explicit repr for repr(uN)/repr(iN).

[Mark-Simulacrum]

I think this is too noisy. Most code out there using repr(C) is probably fine - IIUC, if you're not targeting Windows or AIX, maybe definitely fine? - and having a bunch of allow(...) across a bunch of projects seems unfortunate.

Maybe we can either (a) only enable the lint for migration, i.e., the next edition's cargo fix would add allows for you or (b) we find some new name... C2 for the existing repr(C) usage to avoid allows. But (b) also seems too noisy to me.

[clarfonthey]

Maybe it could just be an optional edition compatibility lint, so if someone enables e.g. rust_20xx_compatibility it shows up but otherwise not.

[RustyYato]

(a) only enable the lint for migration

That was the intention, hence the name edition_2024_repr_c. I'll make this more clear, that this is intended to be a migration lint.

Rustfix would update to #[repr(ordered_fields)] to preserve the current behavior. For the FFI crates, #![allow(edition_2024_repr_c)] at the top of lib.rs would suffice. If you have a mix of FFI and non-FFI uses of repr(C), then you'll have to do the work to figure out which is which, no matter what option is chosen to update repr(C) - even adding repr(C2), since then the FFI use case would need to update all their reprs to repr(C2).

Overall, I think this scheme only significantly burdens those who have a mix of FFI and non-FFI uses of repr(C). But they were going to be burdened no matter what option was chosen.

[RustyYato]

Is the new wording/lints too noisy still?

[joshtriplett]

I think the new wording is still too noisy. We shouldn't assume that most people using repr(C) are using it for ordering rather than FFI.

[RustyYato]

That wasn't my intention, but I don't see another way to do all of the following in the next edition:

• make repr(C) mean - same layout/ABI as what the standard C compiler does
• make repr(ordered_fields) - the same algorithm that's listed for repr(C) in the Rust reference
• ensure that everyone who upgrades to the next edition gets the layout they need (as long as they read the warnings and follow the given advice)
• make it as painless as possible for people who don't mix FFI and stable ordering cases (which I suspect is the vast majority of people). In other words, each crate currently uses repr(C) either exclusively for FFI or exclusively for some stable layout.
• for people who do mix FFI and stable ordering cases in one crate, at least the warning should give them all the places they need to double-check, and they can silence the warning on a case-by-case basis.

I'm open to suggestions on how to handle the diagnostics. Within these constraints, I think my solution is the only real option we have. If there are some objections to these constraints, I would like to hear those too, maybe I missed the mark with these constraints, and missed a potential solution because of it.

[RustyYato]

@DemiMarie said this in a comment:

Would it be possible to rename repr(C) to something else? That way repr(C) becomes a compile-time error rather than silently changing behavior.

I laid out my design axioms above. I believe that making a new repr and deprecating repr(C) is too costly. Especially when the most prevalent use-case of repr(C) is for FFI. A use-case which would benefit from bug fixes that could happen after this RFC.

[DemiMarie]

What is the plan for when is depending on the exact details of C struct layout? Should one add explicit compile-time assertions in this case?

[RustyYato]

I'm not sure what you are asking. If you are asking, when is it ok to depend on the layout being compatible with the target's C compiler, then that is when you are using repr(C#editionNext) in any edition, or using repr(C) in future editions.

[DemiMarie]

I'm not sure what you are asking. If you are asking, when is it ok to depend on the layout being compatible with the target's C compiler, then that is when you are using repr(C#editionNext) in any edition, or using repr(C) in future editions.

There are cases where one needs to not only be compatible with the target C compiler, but also make assumptions about what that layout is. For instance, that is needed when a protocol is defined by a C struct. A target with a different C struct layout could not be supported.

In this case, could one use offset_of and size_of to check that the layout is what one expects?

[RalfJung]

If you want to make assumptions about the C layout you'll have to talk to the vendor responsible for that. I don't know what kind of an answer you are expecting here, Rust obviously can't make any promises.

Obviously you can use all the usual Rust ways of inspecting the layout, just like for repr(Rust).

[briansmith]

The RFC should be clarified to indicate how each kind of zero-sized type within a repr(C#editionNext) type are handled, especially non-array types. This clarification is particularly important since one of the main motivations for this proposed change is changing the handling of zero-sized types.

A zero-sized repr(Rust) type must still remain zero-sized in a repr(C#editionNext) type to support important patterns. This particular example that would be particularly good to include, since this is used in real-world code and the zero-sized type is needed to deal with the lifetime 'a:

#[repr(C#editionNext)]
struct Slice<'a, T> {
    p: *const T,
    len: usize,
    _marker: PhantomData<&'a ()> // zero-sized `repr(Rust)` type
}

An #[repr(C#editionNext) struct that doesn't contain any non-ZST fields isn't necessarily a ZST, so this isn't equivalent to the above:

// Whether this is a ZST depends on the target, right?
// (The C compiler may pad a struct that only contains
// ZSTs so that it has at least size 1, as there are no ZSTs
// in ISO C.)
#[repr(C#editionNext)]
struct PhantomCNext<T>(PhantomData<T>);

#[repr(C#editionNext)]
struct ProbablyNotWhatYouWantedSlice<'a, T> {
    p: *const T,
    len: usize,
    _marker: PhantomCNext <&'a ()> // not necessarily zero-sized.
}

Here is another thing I have seen in real-world code. This does what one expects with C#editionCurr but NOT C#editionNext.

#[repr(C#editionCurr)]
struct AtLeastNCurr<T, const N: usize> {
    // `hd` is guaranteed to have size `size_of::<T>() * N`.
    hd: [T; N], 
    tl: [T],
}

#[repr(C#editionNext)]
struct AtLeastNNextBroken<T, const N: usize> {
    // `hd` is guaranteed to have size `size_of::<T>() * N` ***EXCEPT*** when `N` is zero.
    hd: [T; N], 
    tl: [T],
}

[RustyYato]

Those are good points. I'm not sure we should allow unsized types in repr(C#editionNext). I know flexible length arrays exists, but they are not the same as a trailing [T] field.

A zero-sized repr(Rust) type must still remain zero-sized in a repr(C#editionNext) type to support important patterns.

Yes, this is a good point. I'll make a note that 1-ZST fields are ignored for the purpose of layout in repr(C#editionNext). For overaligned ZSTs, I'm not sure how to specify them.

// hd is guaranteed to have size size_of::<T>() * N EXCEPT when N is zero.

Yes, this is surprising, but unavoidable.

[briansmith]

A zero-sized repr(Rust) type must still remain zero-sized in a repr(C#editionNext) type to support important patterns.

Yes, this is a good point. I'll make a note that 1-ZST fields are ignored for the purpose of layout in repr(C#editionNext). For overaligned ZSTs, I'm not sure how to specify them.

I am not sure that "1-ZST" is the right term to use. Is [0; u8] a 1-ZST? I think for every kind of ZST, we have to enumerate what effect, if any, repr(C#editionNext) has on it. Maybe arrays are the only exception? IDK. Presumably there could be more in the future.

[Jules-Bertholet]

With rust-lang/rust#157973, you could say "fields whose types have trivial ABI".

[RustyYato]

The reason I'm hesitant is I don't want to overpromise anything for repr(C#editionNext). It's fine if it's a bit underspecified for now.

Is [0; u8] a 1-ZST?

I think you meant [u8; 0], but yes, it is a 1-ZST.

I am not sure that "1-ZST" is the right term to use.

It is the easiest case, since they never have an effect on layout (in all versions of Rust up to now). Since they are zero sized and have the minimal alignment, it is always safe to ignore them for the purpose of layout.

A more aligned ZST is complicated, since they likely should have an effect on the offsets of later fields, but since C doesn't have ZSTs, we can't rely on it for a specification. But banning [T; 0] where T is has an alignment > 1 from repr(C#editionNext) is probably going to break some code (esp, since it's a common pattern to get the right alignment for a generic type).

I'm really not sure how to handle them in non-trivial cases.

#[repr(C#editionNext)]
struct Foo {
    a: u8,
    b: [u64; 0],
    c: u8,
}

What offset should c be at, how about b? I would expect both to be at 8, but that's not what would happen if we simply ignored all ZSTs for the purpose of layout.

With rust-lang/rust#157973, you could say "fields whose types have trivial ABI".

That would probably be better.

[Jules-Bertholet]

Note that a zero-length array field is the current recommendation for representing C flexible array members in Rust. https://rust-lang.github.io/rust-bindgen/using-fam.html

[briansmith]

Just to clarify: How zero-length array fields affect layout in #[repr(C)] is up to the ABI spec (C compiler, etc.). They shouldn't be banned (unless the ABI bans them) and they shouldn't be ignored (unless the ABI spec says to).

But, most (all?) other 1-ZST types should be ignored. But, again #[repr(C)] struct MaybeNotZeroSized(PhantomData<()>) shouldn't be considered a ZST because the ABI would likely require the structure to have size at least 1 since there are no ZSTs in C.

[RustyYato]

Note that a zero-length array field is the current recommendation for representing C flexible array members in Rust.

There could be a carve out for that use-case, given the recommendation. But I'm still wary of adding more guarantees in general.

Just to clarify: How zero-length array fields affect layout in #[repr(C)] is up to the ABI spec (C compiler, etc.). They shouldn't be banned (unless the ABI bans them) and they shouldn't be ignored (unless the ABI spec says to).

Yes, I never proposed banning them outright, esp. since they are used so heavily on the Rust side of FFI (like in flexible array members and PhantomData as you have shown).

And as the RFC specifies, repr(C#editionNext) will follow exactly what the target C compiler does.

But, again #[repr(C)] struct MaybeNotZeroSized(PhantomData<()>) shouldn't be considered a ZST because the ABI would likely require the structure to have size at least 1 since there are no ZSTs in C.

Yes, after adding that 1-ZSTs (or types have trivial ABI) are ignored, then this just follows from the RFC.

[briansmith]

Consider:

#[repr(transparent)]
struct ZeroBytesTransparent([u8; 0]);

#[repr(C#editionNext)]
struct Foo {
    a: u8,
    b: ZeroBytesTransparent,
    c: u8,
}

What is the layout of the above, when the C ABI gives zero-length arrays non-zero length? That is, do the field layout rules consider #[repr(transparent)] or not?

Consider:

struct ZeroBytesRust([u8; 0]);

#[repr(C#editionNext)]
struct Foo {
    a: u8,
    b: ZeroBytesRust,
    c: u8,
}

I argue here that field b should not affect the layout because ZeroBytesRust is not #[repr(C)] or #[repr(transparent)].

[Jules-Bertholet]

rust-lang/rust#157973 accounts for all of that.

[briansmith]

Note that a zero-length array field is the current recommendation for representing C flexible array members in Rust. https://rust-lang.github.io/rust-bindgen/using-fam.html

I was expecting from the RFC that in the new #[repr(C#editionNext)], field: [T; 0] would map to whatever the C compiler would do for T field[0] (e.g. foo: [u8; 0], -> uint8_t foo[0]; and field: [T] would map to whatever the C compiler would do for T field[];, so that [T] would be the way to indicate a flexible array member if the C ABI supports them at all, and [T; 0] would represent a zero-length array if the C ABI supports them at all, and potentially either or both could be rejected if the ABI doesn't support them.

[Jules-Bertholet]

• field: [T] results in a fat pointer, which C doesn't have.
• Except for a Clang bug, T field[0]; and T field[]; are equivalent on every C compiler I know of that supports both. This results from the historical fact of T field[0]; being a common extension that C99 decided to standardize with T field[]; syntax.

[briansmith]

• field: [T] results in a fat pointer, which C doesn't have.

No. See https://rust-lang.github.io/rust-bindgen/using-fam.html#using-dynamically-sized-types, for example.

• Except for a Clang bug, T field[0]; and T field[]; are equivalent on every C compiler I know of that supports both. This results from the historical fact of T field[0]; being a common extension that C99 decided to standardize with T field[]; syntax.

How they are handled in Rust would be determined by the ABI / C compiler, right? The issue isn't with C compilers that treat them the same, but where C compilers might not treat them the same. For example, a C compiler could flat-out reject uint8_t field[0]; since it isn't valid ISO C.

[briansmith]

It seems like the intended process for this RFC is to have the RFC make the minimal changes that will allow #[repr(ordered_fields)] to be added, while everything about #[repr(C#editionNext)] would be specified externally. But, as comments showed, there are still some universal things that need to be specified for #[repr(C#editionNext)]:

• Is a #[repr(C#editionCurr)] field allowed in a #[repr(C#editionNext)] type?
• Now to handle non-array#[repr(Rust)] ZST fields in a #[repr(C#editionNext)] type?
• What is the syntactic mapping between Rust and C for flexible array members?

Is it desired to resolve these issues in this RFC, or is there another place (where) to have that discussion?

View changes since the review

[Jules-Bertholet]

• field: [T] results in a fat pointer, which C doesn't have.

No. See https://rust-lang.github.io/rust-bindgen/using-fam.html#using-dynamically-sized-types, for example.

That page doesn't contradict anything I've written?

The issue isn't with C compilers that treat them the same, but where C compilers might not treat them the same. For example, a C compiler could flat-out reject uint8_t field[0]; since it isn't valid ISO C.

If the C compiler simply rejects one of the two, we can just match the one it doesn't reject. There's only an problem if it accepts both but with different layout/ABI. And in that case, it's almost certainly just a bug in the C compiler, like the Clang issue I linked where it doesn't match GCC. Yes, a hypothetical C compiler could theoretically choose to deliberately distinguish them, but there's a lot of cursed stuff that a sufficiently adversarial C compiler could do (e.g. make int and long the same size but different alignments).

[RalfJung]

and field: [T] would map to whatever the C compiler would do for T field[];

I can't make sense of this -- one cannot even pass a [T] by value. [T] is a type with a dynamically known length thanks to the slice metadata. T field[] has entirely unknown length. Those types are so fundamentally different IMO it makes no sense to compare them (and the docs you link to are more confusing than helpful IMO).