Repository navigation
Test failure JIT/Methodical/xxobj/sizeof/sizeof64_Target_64Bit_and_arm_il_r/sizeof64_Target_64Bit_and_arm.cmd #81109
Description
Failed in run: runtime-coreclr outerloop 20230123.1
Failed tests:
R2R-CG2 windows arm Checked @ Windows.11.Arm64.Open
- JIT/Methodical/xxobj/sizeof/sizeof64_Target_64Bit_and_arm_il_r/sizeof64_Target_64Bit_and_arm.cmd
Error message:
Fatal error. Internal CLR error. (0x80131506)
at System.GC.AllocateNewArray(IntPtr, Int32, GC_ALLOC_FLAGS)
at System.Text.StringBuilder.ExpandByABlock(Int32)
at System.Text.StringBuilder.Append(Char, Int32)
at System.Text.StringBuilder.Append(Char)
at System.Diagnostics.StackTrace.ToString(TraceFormat, System.Text.StringBuilder)
at System.Diagnostics.StackTrace.ToString(TraceFormat)
at System.Exception.get_StackTrace()
at System.Exception.ToString()
at Internal.JitInterface.CorInfoImpl.AllocException(System.Exception)
at Internal.JitInterface.CorInfoImpl._resolveToken(IntPtr, IntPtr*, Internal.JitInterface.CORINFO_RESOLVED_TOKEN*)
at Internal.JitInterface.CorInfoImpl.JitCompileMethod(IntPtr ByRef, IntPtr, IntPtr, IntPtr, Internal.JitInterface.CORINFO_METHOD_INFO ByRef, UInt32, IntPtr ByRef, UInt32 ByRef)
at Internal.JitInterface.CorInfoImpl.CompileMethodInternal(ILCompiler.DependencyAnalysis.IMethodNode, Internal.IL.MethodIL)
at Internal.JitInterface.CorInfoImpl.CompileMethod(ILCompiler.DependencyAnalysis.ReadyToRun.MethodWithGCInfo, ILCompiler.Logger)
at ILCompiler.ReadyToRunCodegenCompilation+<>c__DisplayClass46_0.<ComputeDependencyNodeDependencies>g__CompileOneMethod|5(ILCompiler.DependencyAnalysisFramework.DependencyNodeCore`1<ILCompiler.DependencyAnalysis.NodeFactory>, Int32)
at ILCompiler.ReadyToRunCodegenCompilation+<>c__DisplayClass46_0.<ComputeDependencyNodeDependencies>g__CompileOnThread|4(Int32)
at ILCompiler.ReadyToRunCodegenCompilation+<>c__DisplayClass46_0.<ComputeDependencyNodeDependencies>g__CompilationThread|3(System.Object)
at System.Threading.Thread.StartCallback()
Return code: 1
Raw output file: C:hwA74E094BwB27F0950uploads\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_routput.txt
Raw output:
BEGIN EXECUTION
sizeof64_Target_64Bit_and_arm.dll
TestLibrary.dll
2 file(s) copied.
11:39:15.16
Response file: C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll.rsp
C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rIL-CG2sizeof64_Target_64Bit_and_arm.dll
-o:C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll
--targetarch:arm
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:hwA74E094BpSystem..dll
-r:C:hwA74E094BpMicrosoft..dll
-r:C:hwA74E094Bpmscorlib.dll
-r:C:hwA74E094Bp
etstandard.dll
-O
" "dotnet" "C:hwA74E094Bpcrossgen2crossgen2.dll" @"C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll.rsp" -r:C:hwA74E094BwB27F0950eJITMethodicalMethodical_r2IL-CG2*.dll -r:C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rIL-CG2*.dll"
Emitting R2R PE file: C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll
" "dotnet" "C:hwA74E094Bp
2rdump
2rdump.dll" --header --sc --in C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll --out C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll.r2rdump --val"
11:39:16.88
11:39:16.90
Response file: C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rTestLibrary.dll.rsp
C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rIL-CG2TestLibrary.dll
-o:C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rTestLibrary.dll
--targetarch:arm
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:hwA74E094BpSystem..dll
-r:C:hwA74E094BpMicrosoft..dll
-r:C:hwA74E094Bpmscorlib.dll
-r:C:hwA74E094Bp
etstandard.dll
-O
" "dotnet" "C:hwA74E094Bpcrossgen2crossgen2.dll" @"C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rTestLibrary.dll.rsp" -r:C:hw
Stack trace:
at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 132
at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
at Program.<<Main>$>g__TestExecutor7|0_6(<>c__DisplayClass0_0&)
Activity
JulieLeeMSFT commentedon Jan 24, 2023
@markples, it is blocking outerloop. PTAL with a high priority.
JulieLeeMSFT commentedon Jan 24, 2023
@trylek, PTAL. We are seeing the same error message from different test cases. #11360 and #81118.
Edit: corrected the issue number to 11360z
markples commentedon Jan 24, 2023
A meta-question about the testing here - when I look at the devops job - https://dev.azure.com/dnceng-public/public/_build/results?buildId=144935&view=logs&j=454a72be-7065-5174-fb75-00cb418aebaf&t=46d7521e-696b-5674-5766-9cffa0247852 - the log file for this test isn't listed. The test is there in the devops test results integration, and if I go to https://dataexplorer.azure.com/clusters/engsrvprod/databases/engineeringdata and manually find the logs I can see the failure there. Is there a problem with how test merging reports the overall job status back to devops?
Separately, and more directly relevant: do we have definitive instructions on how to repro a crossgen failure like this, especially one with "R2R-CG2 windows arm Checked @ Windows.11.Arm64.Open" that mentions both arm and arm64? I have a bunch of things to try, but I don't know how long it will take until one works and I have a repro to work with.
JulieLeeMSFT commentedon Jan 24, 2023
Moving @AaronRobinsonMSFT ‘s comment to the right issue: #81120 (comment)
trylek commentedon Jan 24, 2023
@markples - reproducing Crossgen2 failures locally should typically amount to building stuff for arm and setting the environment variable
RunCrossgen2to 1 or true (I think it's just checked for being non-empty). For the discrepancy between arm / arm64, I think it's just the same situation as x64 machines being able to execute x86 code by means of the 32-bit subsystem. Please let me know if I missed the point of your comment.JulieLeeMSFT commentedon Jan 26, 2023
@trylek, As Markples mentioned, Send tests to Helix does not show the log file for this failure:
at System.GC.AllocateNewArray(IntPtr, Int32, GC_ALLOC_FLAGS).The page shows 2 other failure logs. Why would it be?
@markwilkie, I grabbed the above error message from here. Where would the sytem get this test failure? The process seems to be different from other test failures cases.
markples commentedon Jan 26, 2023
@JulieLeeMSFT What is the connection to #11360? One of the same tests failed, but I don't see any mention of a failure mode like this one in it.
JulieLeeMSFT commentedon Jan 26, 2023
Error message is the same. #11360 (comment)
19 remaining items
markples commentedon Feb 14, 2023
I think this is change in behavior, isn't it? When I have the test monitor role, I look at the run list for a pipeline, and if recent instances have failed, I open them and look at the stages/jobs. Should I now be looking at the "tests" tab for specific tests and the job list for things like job crashes?
jkoritzinsky commentedon Feb 14, 2023
The "tests" tab will have both crashes and test failures. Individual test failures will have the name of the test, and crashes that take down the whole work item will have a "WorkItemName Work Item Failure" item in addition to any recorded test failures. I've been using this workflow for years for both the runtime and libraries test trees.
trylek commentedon Feb 14, 2023
@jkoritzinsky - I think that Julie's concern I responded to stemmed from the fact that for non-merged tests you see this bit around "Helix work item failure blah blah blah and here is the log" in the "Send to Helix" phase of the runtime test run jobs; apparently we're not seeing the equivalent thing for the merged tests, if it's not about the exit code, what is causing the difference?
BruceForstall commentedon Feb 16, 2023
Another case:
baseservices\invalid_operations\InvalidOperations\InvalidOperations.cmd
R2R-CG2 windows arm Checked @ Windows.11.Arm64.Open
https://dev.azure.com/dnceng-public/public/_build/results?buildId=174073&view=ms.vss-test-web.build-test-results-tab&runId=3460010&paneView=debug&resultId=100024
BruceForstall commentedon Feb 24, 2023
Several new occurrences, e.g.:
CoreMangLib\system\enum\EnumIConvertibleToUint16\EnumIConvertibleToUint16.cmd
R2R-CG2 windows arm64 Checked jitstressregs0x10 @ Windows.11.Arm64.Open
https://dev.azure.com/dnceng-public/public/_build/results?buildId=183267&view=ms.vss-test-web.build-test-results-tab&runId=3560173&paneView=debug&resultId=121383
JulieLeeMSFT commentedon Mar 14, 2023
@trylek another 6 occurences as listed in #83407.
One of them:: runtime-coreclr outerloop 20230312.3
Failed test:
trylek commentedon Mar 14, 2023
@jkotas commented on this yesterday in the issue thread
#77820
Apparently the problem is understood and caused by a race condition that has been fixed since but we need to roll forward to SDK Preview 2 as the LKG version used for executing Crossgen2 to get this fixed completely. As the cause of the crash is now understood, I'll put up a PR fixing the primary cause of the exception - missing reference to
xunitassemblies when running Crossgen2 to build the tests - and that should mitigate the problem. (I didn't want to fix this before the crash is understood as otherwise the fixed references would basically hide this error.)JulieLeeMSFT commentedon Apr 4, 2023
@trylek, You merged #83413. Are we waiting for additional fix to close this isse?
trylek commentedon Apr 4, 2023
@JulieLeeMSFT - my change made Crossgen2 throw much fewer exceptions so we no longer see the occasional exception handling failures on arm64. According to my understanding of @jkotas' explanation the underlying problem is a race condition in exception handling that should be fixed by rolling forward to SDK preview 2 as the LKG version used by the runtime repo. In my runtime repo clone from earlier today,
still yields
so I suspect more work may be needed to fully fix this.
markples commentedon Apr 7, 2023
Here's a direct link to dotnet/runtime main global.json to easily check the sdk version:
https://github.com/dotnet/runtime/blob/main/global.json
markples commentedon Apr 17, 2023
@trylek - we appear to be on preview.3 now. In your comment "much fewer exceptions", did you mean that we're down to the expected number in the tests, or that there are still ones in the same flavor as the ones that you fixed? (In other words, did "more work" only mean waiting for preview.3 to appear so that we can close this now, or is there something else?) Thanks!
trylek commentedon Apr 17, 2023
@markples - I believe this can be closed now. My original mitigation just reduced the number of exceptions internally thrown and caught during Crossgen2 compilation and so reduced the repro rate of this non-deterministic race condition. As we're now at Preview 3 containing the proper fix for the race condition according to JanK's explanation, we should be good now.