Repository navigation
Struct promoted to auto-layout if it contains a non-blittable field type + explicit layout field type #44579
Description
Description
We have discovered an unexpected behavior of the runtime regarding layout of a struct in memory with .NET CLR.
In the following example, the struct BoolAndExplicitStruct layout will be promoted to an auto-layout because of the presence of the field public bool Bool and public ExplicitLayoutStruct ExplicitLayout together:
[StructLayout(LayoutKind.Explicit)]
struct ExplicitLayoutStruct {
[FieldOffset(0)] public byte B00;
}
struct RegularStruct
{
public int Field;
}
struct BoolAndExplicitStruct
{
// bool AND a field with an explicit layout struct
// change the layout of this struct to Auto instead of sequential
public bool Bool;
public ExplicitLayoutStruct ExplicitLayout;
public int Int32;
public RegularStruct RegularStruct;
public long Int64;
}
// Field Int64: starts at offset 0
// Field Int32: starts at offset 8
// Field Bool: starts at offset 12
// Field ExplicitLayout: starts at offset 16
// Field RegularStruct: starts at offset 24This is causing lots of trouble on the Burst compiler on our side, because as we are using these structs in native code, we expect the same sequential+explicit layout than .NET, but we were not expecting an implicit auto-layout promotion in this case (We haven't implemented today the auto-layout because we fear its implementation dependent behavior)
It seems to be related around this function CheckIfDisqualifiedFromManagedSequential or the caller of this function, but haven't identified exactly in the code when this flip occurs.
Do you know the reason of this auto-layout promotion? Could we revert that behavior? (We will be happy to make a PR)
Configuration
Happens on all OS, all CPU and all .NET version (including .NET framework) but not on Mono (because Mono doesn't have auto-layout promotion afaik)
Regression?
Nope.
Activity
Dotnet-GitSync-Bot commentedon Nov 12, 2020
I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.
MichalStrehovsky commentedon Nov 12, 2020
I'll just paste the relevant part of the doc for StructLayoutAttribute like I did in discord: "For non-blittable types, it controls the layout when the class or structure is marshaled to unmanaged code, but does not control the layout in managed memory."
The native layout of the struct is as one would expect based on the fact that C# structs are implicitly marked Sequential:
Marshal.OffsetOf<BoolAndExplicitStruct>("Bool") == 0Sequential layout specification doesn't control the managed view because
BoolAndExplicitStructis not blittable (there's the bool field) - this behavior falls out from the doc.There was a similar conversation about StructLayout specification not applying to the managed view e.g. here: #1770 (comment)
xoofx commentedon Nov 12, 2020
That's a bit incorrect. The presence of a bool doesn't change the layout. You can check by yourself. Here, that's the presence of a non-blittable type AND an explicit struct layout field. That's why it is so unexpected.
For example the following
BoolAndExplicitStructstruct is perfectly sequential:but if you uncomment
public ExplicitLayoutStruct ExplicitLayoutyou will get an auto-layout.I was not aware of any auto-layout promotion except when you have a managed object inside one of the struct. But this auto-layout seems completely arbitrary and, well, quite wrong to me.
xoofx commentedon Nov 12, 2020
Until now, I was always expecting that auto-layout promotion would only happen in the presence of some managed objects in any embedded structs. But this auto-layout promotion in the presence of only unmanaged type is breaking these expectations quite hard, it makes basically native struct sharing barely impossible without using explicit layout everywhere (and explicit layout is awful to work with when you start to have a variable pointer type 32/64 bits).
MichalStrehovsky commentedon Nov 12, 2020
The presence of the bool makes the struct non-blittable - at that point the runtime is free not to respect the StructLayoutAttribute. It will often end up sequential in the managed view too, but details of when that happens are undocumented. I tried to look up the reasoning behind why the runtime ends up laying it out differently in this case, but that change was made before 2010 and is more difficult to dig out.
xoofx commentedon Nov 12, 2020
Indeed, and I would love to understand the reason for this specific case, and if we can revert that behavior for unmanaged types (blittable or not)
Ultrahead commentedon Nov 12, 2020
I always wondered why it was decided to map a boolean against an int in memory instead of a byte?
jkoritzinsky commentedon Nov 12, 2020
Once the struct is considered non-blittable, the runtime checks if the struct is disqualified from being managed sequential. Since one of the fields is not managed sequential (it's explicit), the type is disqualified from having a sequential managed layout.
xoofx commentedon Nov 12, 2020
Why having an explicit layout of a field is disqualifying the struct for being sequential? Why promoting to auto-layout? Also, if you remove the bool but keep the explicit layout field in my example above, it won't promote the struct to auto-layout. So what you describe is not exactly what is happening.
In managed, a bool is mapped to a byte. The problem mentioned above doesn't change this layout for the bool type (it's still 1 byte). The problem is implicit promotion to auto-layout of the struct, but an auto-layout won't change the size of the blittable types, unlike P/Invoke interop that will involve marshalling behaviors.
Why in the context of interop marshalling a bool is translated to int? I believe it goes back to the Windows API having declared the C++
BOOLto actually map to aDWORDso anint32, so .NET selected this behavior back in the days to match the Windows layout for BOOL.jkoritzinsky commentedon Nov 12, 2020
If you remove the bool field then the structure is blittable, so layout is preserved. Explicit only disqualifies the type if the type is non-blittable, as I said.
The reasoning behind this design is from long before my time and from before the source code was imported into the CoreCLR repo, so I'm not sure what the original reasoning was.
35 remaining items
xoofx commentedon Nov 13, 2020
If we can do that without requiring any user code changes, that would be fantastic.
But I agree, if it is breaking any actual code relying on this undocumented layout then we will have to find another way to support that (e.g maybe a
[MarshalAs(U1)]on the bool to tell the layout that it should not promote to auto in that case because we want the same layout for native than managed, even though that would still be super painful for our users)Ultrahead commentedon Nov 13, 2020
@xoofx Do many of those structs you mentioned contain char types?
zacknewman commentedon Aug 28, 2021
The documentation also states, "The following complex types are also blittable types:
• One-dimensional arrays of blittable primitive types, such as an array of integers. However, a type that contains a variable array of blittable types is not itself blittable.
• Formatted value types that contain only blittable types (and classes if they are marshaled as formatted types). For more information about formatted value types, see Default marshaling for value types." (emphasis added)
This means that
decimalis also blittable since it is has aLayoutKind.Sequentiallayout and its 3 fields,_flags,_hi32, and_lo64are all blittable. However if you replace theboolfield inBoolAndExplicitStructwith adecimal, the layout is still not sequential. Clearly something is wrong with at least the documentation.tannergooding commentedon Aug 30, 2021
@zacknewman, the types in the "core library" (
System.Private.Corelib,mscorlib, etc) are often special and cannot necessarily be understood by simply looking at the C# source code.decimalis meant to map to the Win32DECIMALtype (which is what the type is based on, rather than something like IEEE 754 decimal types which are xplat and standardized) and that has special alignment requirements and a different layout in native: https://github.com/dotnet/runtime/blob/main/src/coreclr/vm/fieldmarshaler.cpp#L133-L137Likewise:
Vector<T>is explicitly blocked from interop because it is variable sized.Vector64/128/256<T>are currently blocked today because they explictly map to the__m64,__m128, and__m256ABI types and there is a known bug on Windows x64 for returned values that would break interopSystem.Runtime.InteropServices.CLong/CULongare not user-defined structs for interop, but rather are "primitive integral types" mapping to the C/C++longandunsigned longkeywords, respectivelyzacknewman commentedon Aug 31, 2021
@tannergooding, thank you for that info. I figured it had something to do with
decimalbeing special. I find it unfortunate that while the documentation expressly mentions that "primitives" likeSystem.BoolandSystem.Charare not blittable, it fails to mentionSystem.Decimaltoo. This led me to being overly cautious when constructing my own "primitives" (e.g.,U128) where I was worried relying on aLayoutKind.Sequentiallayout would not actually enforce its layout in managed memory (since it didn't fordecimal) thus motivating me to use the attribute[StructLayout(LayoutKind.Explicit, CharSet = CharSet.Unicode, Pack = 8, Size = 16)]which in turn now causesLayoutKind.Sequentialissues when used alongside non-blittable types just likeExplicitLayoutStructdoes in this example.I know the documentation I cited is not a substitute for a formal specification, so perhaps I expect too much from it. On a related note, do you know if that documentation is applicable for "constructed" types too? Or is there an unwritten assumption that it only applies to "plain" types? For example if I have the type constructor:
is the memory layout in managed memory for
Maybe<uint>1 byte containing_tagfollowed by 3 bytes of padding followed by 4 bytes containing_someor is the runtime allowed to store it differently since the documentation doesn't apply to "constructed" types? I've experimented with several type constructors with variable number of type parameters and several constructed types from said constructors, and the memory layout was always what I expected/hoped it to be; but this falls embarrassingly short of a mathematical proof.tannergooding commentedon Aug 31, 2021
This can likely be updated to include the fact. It may also be possible to force
decimalto be blittable by manually forcing the alignment to be8in the VM, as we do with a few other types likeVector64/128/256, but I'd defer to @jkotas and @jkoritzinsky on if they thinks that's a good idea.AFAIR, generic types should have the same guarantees provided
TSomeis blittable (and should be blocked from P/Invoke otherwise). I believe I added all the relevant tests and confirmed this behavior in the VM when I added the support back in #103zacknewman commentedon Aug 31, 2021
@tannergooding, you're awesome. Thank you.
jkotas commentedon Aug 31, 2021
The current decimal implementation has
longfield that ensures the sufficient alignment. We used to have a quirk to force decimal alignment in the VM. The quirk was deleted in #38603 .decimal is non-blittable for backwards compatibility only. I am not sure whether it is worth fixing that. It is just going to break random old libraries like what happens every time we touch anything in build-in interop.