Repository navigation
Assert failure: m_alignpad == 0 in libraries tests #70231
Description
Activity
This is failing in the next-object validation.
m_alignpadis 0 in the dump. It means that it got cleared between the assert condition was checked and the crash dump was taken.The most likely explanation is that there is a race condition between
GCHeap::NextObjand the memory clearing for newly allocated objects in the GC that causesGCHeap::NextObjto return next object that was not cleared yet. @dotnet/gc Could you please take a look?Same assert here: #68511
The same assert happened for jitminopts in System.Text.RegularExpressions.Unit.Tests on win-x64:
Same assert failed on System.Buffers.Tests https://dev.azure.com/dnceng/public/_build/results?buildId=1809197&view=ms.vss-test-web.build-test-results-tab&runId=48130508&resultId=197594&paneView=debug in this unrelated PR #70194
- changed the title
[-]Assert failure: m_alignpad == 0 in System.IO.Pipes.Tests[/-][+]Assert failure: m_alignpad == 0 in libraries tests[/+]on Jun 20, 2022 17 remaining items
note: doesn't repro when I change optimization level from
-O2to-O0forcee_wks_core(Checked config)Reacted by Maoni Stephens@Maoni0 has a fix already. I also found a reasonably frequent repro last week - the readytorun\coreroot_determinism\coreroot_determinism coreclr test and run it with her fix over the weekend. Before the fix, it reproed about every 80 iterations. With the fix, 1000 iterations have passed without any repro.
Reacted by Mukund Raghav Sharma (Moko), Maoni Stephens and karb0f0sReacted by Jakob Botsch NielsenThank you very much, @EgorBo! Based on your list of instructions, I was able to fairly quickly repro the assertion failure on my Windows desktop and have confirmed that after:
- Removing the demotion check:
// if ((g == 0) && hp->settings.demotion) // return NULL;//could be racing with another core allocating
- Adding a null check on the GCSafeMethodTable:
Object * nextObj = GCHeapUtilities::GetGCHeap ()->NextObj (this); if ((nextObj != NULL) && (nextObj->GetGCSafeMethodTable() != nullptr) && (nextObj->GetGCSafeMethodTable() != g_pFreeObjectMethodTable))
the assertion failure doesn't occur based on running the tests overnight.
The main question I had was: how do I find out which test did the assertion failure occur for?
Right now, I get a pretty general message that there was an assertion failure for a System.Collections.Test but no indication as to which test it failed on:
=== TEST EXECUTION SUMMARY === System.Collections.Tests Total: 32703, Errors: 0, Failed: 0, Skipped: 0, Time: 26.645s Discovering: System.Collections.Tests (method display = ClassAndMethod, method display options = None) Discovered: System.Collections.Tests (found 5562 of 7453 test cases) Starting: System.Collections.Tests (parallel test collections = on, max threads = 20) Assert failure(PID 24604 [0x0000601c], Thread: 18352 [0x47b0]): m_alignpad == 0 CORECLR! ObjHeader::Validate + 0x43 (0x00007ffa`66f8bcc3) CORECLR! Object::ValidateInner + 0x493 (0x00007ffa`66f12ed3) CORECLR! Object::Validate + 0xA1 (0x00007ffa`66f12a01) CORECLR! OBJECTREF::operator-> + 0x21 (0x00007ffa`66f0c0b1) CORECLR! JIT_ClassProfile32 + 0xC7 (0x00007ffa`671310b7) <no module>! <no symbol> + 0x0 (0x00007ffa`0a65e527) <no module>! <no symbol> + 0x0 (0x000000dd`6cafb030) <no module>! <no symbol> + 0x0 (0x000000dd`6cafac00) <no module>! <no symbol> + 0x0 (0x00000276`e4fd77f0) <no module>! <no symbol> + 0x0 (0x00000276`e543c740) File: C:\runtime\src\coreclr\vm\syncblk.cpp Line: 2955 Image: C:\runtime\artifacts\bin\testhost\net7.0-windows-Release-x64\dotnet.exe Discovering: System.Collections.Tests (method display = ClassAndMethod, method display options = None) Discovered: System.Collections.Tests (found 5562 of 7453 test cases) Starting: System.Collections.Tests (parallel test collections = on, max threads = 20) Finished: System.Collections.TestsIf we know the exact test, we'll have a quicker repro with a more targeted run.
Reacted by Egor Bogatov and karb0f0s@mrsharm in my repro you can append
-verboseafter-notrait category=failingand xunit will print test names
I think it were diffrient tests each run and I wasn't able to reproduce the issue when I was asking xunit to run just one test specifically (via-method xx)Reacted by Mukund Raghav Sharma (Moko)This is still failing in libraries-pgo leg on linux-arm
https://dev.azure.com/dnceng/public/_build/results?buildId=1927773&view=results
This assert can be hit for many different GC holes, GC heap corruptions and GC bugs. It is important to at least capture the stacktrace where the assert is hit, and track different stack traces by separate issues.
I think it is unlikely that the Linux arm crash you have seen is same root cause as this issue. I expect that it is going to have a different stacktrace. Unfortunately, we cannot tell for sure since no dump was collected.
I am closing this as non-actionable. If you see a test failing with this assert, do not re-activate this issue. Instead open a new issue and capture stack trace where the assert is hit in the issue description.
Reacted by Kunal Pathak and Mukund Raghav Sharma (Moko)It seems that it is inconsistent. Tried reproing it but doesn't repro. I will keep an eye and open new issue.
- ghost locked as resolved and limited conversation to collaborators
on Sep 9, 2022
Hit in #70226
Dump and logs: https://dev.azure.com/dnceng/public/_build/results?buildId=1806089&view=ms.vss-test-web.build-test-results-tab&runId=48090090&resultId=197319&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab
Stacktrace: