Repository navigation
Failing tests on .NET7 #2117
Description
Activity
Do we have the output images? Are they badly off? If so that would likely indicate a JIT error which we can report upstream.
Do we have the output images? Are they badly off? If so that would likely indicate a JIT error which we can report upstream.
Visually, I cant tell the difference, but the report says 30% of the pixels are different.
Expected:
Actual:
I could not pinpoint whats going wrong, since it only happens on Release mode.
To report an issue upstream, we need a smaller reproduction.The difference is that the RGB components in the actual output for transparent areas are
255,255,255vs0,0,0
That it fails for skew and not rotate is confusing as the code differs only on the transform (though that might make it possible to identify a cause).
I cannot replicate this locally. 😔
I had a look at the latest preview release notes and it would take a smarter person than I to narrow down a potential cause without replication.
I can replicate it locally, but it seems not to happen always (but very often).
Intermittent! Whoa that’s weird!
To add another weird observation to the list: When executing the test via visual studio they always pass. Only executing the tests via command line fails occasionally.
Reacted by James Jackson-SouthI think it's got something to do with the way we unpremultiply
Vector4instances inNumericswhere theWvalue is0.We actually get 1 of two possible values from both the scalar and simd versions.
If the input
XYZvalues are0then you get<NaN, NaN, NaN, 0>
If the values are greater than0then you get<∞, ∞, ∞, 0>Now what I think is happening is that when we are converting the values to bytes on some chipsets
NaNis somehow being converted into255. What the exact cause (and why it doesn't seem to affect other tests) I do not know yet.Maybe handling of NaN/Infinite has changed in .Net7.0? It's still weird that this does not always happens.
edit: the only thing I could find NaN related changed in .Net7.0 seems to be the
Equalmethod: https://docs.microsoft.com/en-us/dotnet/core/compatibility/core-libraries/7.0/equals-nanI cant see, how this could affect the Skew Processor, though.
We need to add a bunch of diagnostic information to the output when something fails. I'm going to write some code in our tests to do this.
Reacted by Brian Popow@brianpopow I managed to capture the environmental values for the Windows failure.
OS=Windows 10.0.20348 Intel Xeon Platinum 8272CL CPU 2.60GHz, 1 CPU, 2 logical and 2 physical cores .NET SDK=7.0.100-preview.6.22352.1 [Host] : .NET 7.0.0 (7.0.22.32404), X64 RyuJITReacted by Brian PopowYep. Definitely different.
What I don't understand is why
Bgra32only? I couldn't find anything in our pipeline that would make conversion fromVector4different. We just shuffle the values first.11 remaining items
Ok, so I could reliably repro on RC1, but I cannot repro on RC2 (nightly build
7.0.100-rc.2.22464.26).There were ~161 commits between these two: dotnet/runtime@release/7.0-rc1...release/7.0-rc2
Of those commits, the most likely to impact this would have been:
- [release/7.0] Fix use of uninitialized memory for Vector3 constants dotnet/runtime#74880 - Fix use of uninitialized memory for Vector3 constants
- [release/7.0] Ensure that the SSE fallback for Vector3.Dot masks off the unused element of op1 and op2 dotnet/runtime#74980 - Ensure that the SSE fallback for Vector3.Dot masks off the unused element of op1 and op2
Both of these should only have impacted
Vector3and only CG2/R2R (Crossgen or Ready To Run) scenarios. I'd expect most of the hardware being run against was SSE4.1 or later, but its possible that there is some specific edge case or dependence on some BCL method (which would be cg2/r2r) that was causing this instead.It would be great if someone else could validate that it is fixed as well, and if so I can dig a little bit deeper to finalize the root cause.
-- Noting that the latest .NET 7 RC2 nightly build may also require the 6.0.10 SDK which I'm not sure where to get atm. I ended up changing the
TargetFrameworksunderSIXLABORS_TESTING_PREVIEWto only benet7.0to work around this.Hmmm, maybe I spoke too soon.
I'm still seeing the following in a clean build, but it won't repro after having been hit once:
[xUnit.net 00:00:05.20] Skew_IsNotBoundToSinglePixelType<Bgra32>(provider: TestPattern100x50[Bgra32], x: 20, y: 10) [FAIL] [xUnit.net 00:00:05.20] Skew_IsNotBoundToSinglePixelType<Bgra32>(provider: TestPattern100x50[Bgra32], x: -20, y: -10) [FAIL] Failed Skew_IsNotBoundToSinglePixelType<Bgra32>(provider: TestPattern100x50[Bgra32], x: 20, y: 10) [22 ms] Error Message: SixLabors.ImageSharp.Tests.TestUtilities.ImageComparison.ImageDifferenceIsOverThresholdException : Image difference is over threshold! Test Environment OS : Windows Test Environment is CI : False Test Environment is .NET Core : True Test Environment is Mono : False Report ImageFrame {i}: Total difference: 29.9761% [δ(65535,65535,65535,0) @ (5,0)]; [δ(65535,65535,65535,0) @ (6,0)]; [δ(65535,65535,65535,0) @ (7,0)]; [δ(65535,65535,65535,0) @ (8,0)]; [δ(65535,65535,65535,0) @ (9,0)]... Stack Trace: at SixLabors.ImageSharp.Tests.TestUtilities.ImageComparison.ImageComparerExtensions.VerifySimilarity[TPixelA,TPixelB](ImageComparer comparer, Image`1 expected, Image`1 actual) in D:\Users\tagoo\source\repos\ImageSharp\tests\ImageSharp.Tests\TestUtilities\ImageComparison\ImageComparer.cs:line 91 at SixLabors.ImageSharp.Tests.TestImageExtensions.CompareToReferenceOutput[TPixel](Image`1 image, ImageComparer comparer, ITestImageProvider provider, Object testOutputDetails, String extension, Boolean grayscale, Boolean appendPixelTypeToFileName, Boolean appendSourceFileOrDescription, IImageDecoder decoder) in D:\Users\tagoo\source\repos\ImageSharp\tests\ImageSharp.Tests\TestUtilities\TestImageExtensions.cs:line 227 at SixLabors.ImageSharp.Tests.TestUtils.RunValidatingProcessorTest[TPixel](TestImageProvider`1 provider, Action`1 process, Object testOutputDetails, ImageComparer comparer, Boolean appendPixelTypeToFileName, Boolean appendSourceFileOrDescription) in D:\Users\tagoo\source\repos\ImageSharp\tests\ImageSharp.Tests\TestUtilities\TestUtils.cs:line 239 at SixLabors.ImageSharp.Tests.Processing.Processors.Transforms.SkewTests.Skew_IsNotBoundToSinglePixelType[TPixel](TestImageProvider`1 provider, Single x, Single y) in D:\Users\tagoo\source\repos\ImageSharp\tests\ImageSharp.Tests\Processing\Processors\Transforms\SkewTests.cs:line 49 at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor) at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) Failed Skew_IsNotBoundToSinglePixelType<Bgra32>(provider: TestPattern100x50[Bgra32], x: -20, y: -10) [5 ms] Error Message: SixLabors.ImageSharp.Tests.TestUtilities.ImageComparison.ImageDifferenceIsOverThresholdException : Image difference is over threshold! Test Environment OS : Windows Test Environment is CI : False Test Environment is .NET Core : True Test Environment is Mono : False Report ImageFrame {i}: Total difference: 29.9761% [δ(65535,65535,65535,0) @ (0,0)]; [δ(65535,65535,65535,0) @ (1,0)]; [δ(65535,65535,65535,0) @ (2,0)]; [δ(65535,65535,65535,0) @ (3,0)]; [δ(65535,65535,65535,0) @ (4,0)]... Stack Trace: at SixLabors.ImageSharp.Tests.TestUtilities.ImageComparison.ImageComparerExtensions.VerifySimilarity[TPixelA,TPixelB](ImageComparer comparer, Image`1 expected, Image`1 actual) in D:\Users\tagoo\source\repos\ImageSharp\tests\ImageSharp.Tests\TestUtilities\ImageComparison\ImageComparer.cs:line 91 at SixLabors.ImageSharp.Tests.TestImageExtensions.CompareToReferenceOutput[TPixel](Image`1 image, ImageComparer comparer, ITestImageProvider provider, Object testOutputDetails, String extension, Boolean grayscale, Boolean appendPixelTypeToFileName, Boolean appendSourceFileOrDescription, IImageDecoder decoder) in D:\Users\tagoo\source\repos\ImageSharp\tests\ImageSharp.Tests\TestUtilities\TestImageExtensions.cs:line 227 at SixLabors.ImageSharp.Tests.TestUtils.RunValidatingProcessorTest[TPixel](TestImageProvider`1 provider, Action`1 process, Object testOutputDetails, ImageComparer comparer, Boolean appendPixelTypeToFileName, Boolean appendSourceFileOrDescription) in D:\Users\tagoo\source\repos\ImageSharp\tests\ImageSharp.Tests\TestUtilities\TestUtils.cs:line 239 at SixLabors.ImageSharp.Tests.Processing.Processors.Transforms.SkewTests.Skew_IsNotBoundToSinglePixelType[TPixel](TestImageProvider`1 provider, Single x, Single y) in D:\Users\tagoo\source\repos\ImageSharp\tests\ImageSharp.Tests\Processing\Processors\Transforms\SkewTests.cs:line 49 at InvokeStub_SkewTests.Skew_IsNotBoundToSinglePixelType(Object, Object, IntPtr*) at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)I've root caused the bug. There looks to be a bug with
maxpsandminps.This is the disassembly for
Bgra32.FromVector4in .NET 6:push rdi push rsi sub rsp,38h vzeroupper mov rdi,rcx mov rsi,rdx vmovupd xmm0,xmmword ptr [rsi] vmovupd xmmword ptr [rsp+28h],xmm0 mov rcx,7FF8BAA93EA8h mov edx,156h call CORINFO_HELP_GETSHARED_NONGCSTATIC_BASE (07FF919E3B470h) mov rax,1A9DCFC5410h mov rax,qword ptr [rax] vmovupd xmm0,xmmword ptr [rsp+28h] vmulps xmm0,xmm0,xmmword ptr [rax+8] vmovupd xmmword ptr [rsi],xmm0 vmovupd xmm0,xmmword ptr [rsi] mov rax,1A9DCFC5418h mov rax,qword ptr [rax] vaddps xmm0,xmm0,xmmword ptr [rax+8] vmovupd xmmword ptr [rsi],xmm0 vmovupd xmm0,xmmword ptr [rsi] vxorps xmm1,xmm1,xmm1 mov rax,1A9DCFC5410h mov rax,qword ptr [rax] vmovupd xmm2,xmmword ptr [rax+8] vmaxps xmm0,xmm0,xmm1 vminps xmm0,xmm0,xmm2 vmovupd xmmword ptr [rsi],xmm0 vcvttss2si eax,dword ptr [rsi] mov byte ptr [rdi+2],al vcvttss2si eax,dword ptr [rsi+4] mov byte ptr [rdi+1],al vcvttss2si eax,dword ptr [rsi+8] mov byte ptr [rdi],al vcvttss2si eax,dword ptr [rsi+0Ch] mov byte ptr [rdi+3],al add rsp,38h pop rsi pop rdi ret
This is the codegen for the same method in .NET 7:
push rdi push rsi sub rsp,38h vzeroupper mov rdi,rcx mov rsi,rdx vmovupd xmm0,xmmword ptr [rsi] vmovupd xmmword ptr [rsp+28h],xmm0 mov rcx,7FF887B04688h mov edx,155h call CORINFO_HELP_GETSHARED_NONGCSTATIC_BASE (07FF8E6C4C890h) mov rax,22D8700EF80h mov rax,qword ptr [rax] add rax,8 vmovupd xmm0,xmmword ptr [rsp+28h] vmulps xmm0,xmm0,xmmword ptr [rax] vmovupd xmmword ptr [rsi],xmm0 vmovupd xmm0,xmmword ptr [rsi] mov rdx,22D8700EF88h mov rdx,qword ptr [rdx] vaddps xmm0,xmm0,xmmword ptr [rdx+8] vmovupd xmmword ptr [rsi],xmm0 vxorps xmm0,xmm0,xmm0 vmaxps xmm0,xmm0,xmmword ptr [rsi] vminps xmm0,xmm0,xmmword ptr [rax] vmovupd xmmword ptr [rsi],xmm0 vmovss xmm0,dword ptr [rsi] vcvttss2si eax,xmm0 mov byte ptr [rdi+2],al vmovss xmm0,dword ptr [rsi+4] vcvttss2si eax,xmm0 mov byte ptr [rdi+1],al vmovss xmm0,dword ptr [rsi+8] vcvttss2si eax,xmm0 mov byte ptr [rdi],al vmovss xmm0,dword ptr [rsi+0Ch] vcvttss2si eax,xmm0 mov byte ptr [rdi+3],al add rsp,38h pop rsi pop rdi ret
You'll note that these are basically identical (you can ignore the
GETSHARED_NONGCSTATIC_BASEdifference) except .NET 6 does (simplified):vmovupd xmm0, [vector4] ; read vector4 into xmm0 vxorps xmm1, xmm1, xmm1 ; zero xmm1 vmovupd xmm2, [maxBytes] ; read maxBytes into xmm2 vmaxps xmm0, xmm0, xmm1 ; vector4 = max(vector4, zero) vminps xmm0, xmm0, xmm2 ; vector4 = min(vector4, maxBytes)
But .NET 7 is doing (simplified):
vxorps xmm0, xmm0, xmm0 ; zero xmm0 vmaxps xmm0, xmm0, [vector4] ; vector4 = max(zero, vector4) vminps xmm0, xmm0, [maxBytes] ; vector4 = min(vector4, maxBytes)
This might not seem like much, but it has big impact for
NaNbecausemaxps/minpsreturn the right hand side if either operand isNaN. This means .NET 6 propagates up0while .NET 7 propagates upNaN.I believe this is non-deterministic because it somewhat depends on TieredCompilation and when the method becomes optimized. It more reliably reproduces if
FromVector4is marked "no-inlining" and bothFromVector4/Packare markedAggressiveOptimization.Going to see if I can figure out why the JIT is deciding to swap operands here and will try to get a fix up. In the interim, the simple workaround here should be to change
Pack(ref Vector4)to justPack(Vector4). This should be "better" when the method is inlined (and its being aggressively inlined) but also even when not inlined for non-Windows platforms.Reacted by Scott Williams and Brian PopowI've root caused the bug. There looks to be a bug with
maxpsandminps.@tannergooding: very happy to see that you found the root cause of this. Thanks a lot for working on this issue and providing a fix!
edit:
in the interim, the simple workaround here should be to change Pack(ref Vector4) to just Pack(Vector4)
I will make a PR for that.
- added a commit that references this issue
on Sep 16, 2022 - added a commit that references this issue
on Sep 16, 2022 - added a commit that references this issue
on Sep 16, 2022 - added a commit that references this issue
on Sep 16, 2022 closing this now with #2230 merged
- added a commit that references this issue
on Oct 5, 2022
Prerequisites
DEBUGandRELEASEmodeImageSharp version
Current main branch
Other ImageSharp packages and versions
none
Environment (Operating system, version and so on)
Windows 10
.NET Framework version
Description
The following tests fail on .NET 7.0 Windows only. Introduced in 7.0.100-preview.4.22252.9:
Skew_IsNotBoundToSinglePixelType<Bgra32>(provider: TestPattern100x50[Bgra32], x: 20, y: 10)Skew_IsNotBoundToSinglePixelType<Bgra32>(provider: TestPattern100x50[Bgra32], x: -20, y: -10)The issue seems to be only occurring in Release mode. Also it seems somehow related to the PixelFormat
Bgra32Environment Info
In addition on Mac .NET 7.0 we are seeing the following triggered by a a call to `System.Runtime.Intrinsics.Vector256.Create(Single value)`Environment Info
Steps to Reproduce
Run the tests with .NET 7.0 in Release mode.
Images
No response