Skip to content

XUnitLogChecker: Frames are sometimes unresolved #94721

Closed

Description

@carlossanlop

Examples from #93906:

Printing the dumps is not very useful when they look like this:

0  Id: 1ea4.1430 Suspend: 0 Teb: 00894000 Unfrozen
ChildEBP RetAddr  
00b7da5c 76e44240 ntdll!NtWaitForMultipleObjects+0xc
00b7dbf0 7252f3c6 KERNELBASE!WaitForMultipleObjectsEx+0xf0
WARNING: Stack unwind information not available. Following frames may be wrong.
00b7dc74 7252f776 coreclr!GetCLRRuntimeHost+0x145706
00b7ddcc 7253637d coreclr!GetCLRRuntimeHost+0x145ab6
00b7de1c 7252f46b coreclr!GetCLRRuntimeHost+0x14c6bd
00b7de7c 725a7d9b coreclr!GetCLRRuntimeHost+0x1457ab
00b7df54 0b8f1533 coreclr!GetCLRRuntimeHost+0x1be0db
00b7df9c 0b80e2e5 0xb8f1533
00b7e074 0b801784 0xb80e2e5
00b7e15c 0a158358 0xb801784
00b7e240 0123bf82 0xa158358
00b7e27c 728500cb 0x123bf82
00b7e288 7257ddea coreclr!GetCLRRuntimeHost+0x46640b
00b7e6a4 7257df28 coreclr!GetCLRRuntimeHost+0x19412a
00b7e720 7257e5a5 coreclr!GetCLRRuntimeHost+0x194268
00b7e984 723708ed coreclr!GetCLRRuntimeHost+0x1948e5
00b7eaac 723704dc coreclr+0x408ed
00b7eae8 72370599 coreclr+0x404dc
00b7eb3c 723706c3 coreclr+0x40599
00b7eb9c 7236d30d coreclr+0x406c3
00b7eef0 723e8473 coreclr+0x3d30d
00b7f040 729857c2 coreclr+0xb8473
*** WARNING: Unable to verify checksum for hostpolicy.dll
00b7f0a0 72d3edc3 coreclr!coreclr_execute_assembly+0x92
00b7f0d0 72d7b9c9 hostpolicy+0xedc3
00b7f1f0 72d7b691 hostpolicy+0x4b9c9
00b7f22c 72d7cb3e hostpolicy+0x4b691
*** WARNING: Unable to verify checksum for hostfxr.dll
00b7f2d8 72f2425d hostpolicy!corehost_main+0x10e
00b7f3a4 72f2829e hostfxr!hostfxr_set_runtime_property_value+0x912d
00b7f410 72f26ec5 hostfxr!hostfxr_set_runtime_property_value+0xd16e
00b7f4a8 72f23eec hostfxr!hostfxr_set_runtime_property_value+0xbd95
00b7f56c 72f1ab58 hostfxr!hostfxr_set_runtime_property_value+0x8dbc
00b7f614 01363639 hostfxr!hostfxr_main_startupinfo+0x88
00b7f7dc 01363a9d dotnet+0x13639
00b7f830 013674e3 dotnet+0x13a9d
00b7f850 013673b7 dotnet+0x174e3
00b7f8ac 0136725d dotnet+0x173b7
00b7f8b4 01367548 dotnet+0x1725d
00b7f8bc 753a62c4 dotnet+0x17548
00b7f8d0 77e81ca9 kernel32!BaseThreadInitThunk+0x24
00b7f918 77e81c74 ntdll!__RtlUserThreadStart+0x2f
00b7f928 00000000 ntdll!_RtlUserThreadStart+0x1b
...

Activity

  1. ghost added
    untriagedNew issue has not been triaged by the area owner
    on Nov 14, 2023
  2. carlossanlop commented on Nov 14, 2023

    @carlossanlop
    ContributorAuthor

    Another example: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-93906-merge-73d789b084234d60b0/System.IO.Compression.Brotli.Tests/1/console.68d2b2ea.log?helixlogtype=result

    Thread Id: 0x1e
          Child SP               IP Call Site
     0xffa24af9d810 0xffa24a9e22ec libclrjit.so!?? at ??:0:0
     0xffa24af9d890 0xffa24a9d0408 libclrjit.so!?? at ??:0:0
     0xffa24af9d8e0 0xffa24a9d6874 libclrjit.so!?? at ??:0:0
     0xffa24af9d990 0xffa24a9d5430 libclrjit.so!?? at ??:0:0
     0xffa24af9da70 0xffa24a9d7f64 libclrjit.so!?? at ??:0:0
     0xffa24af9e0c0 0xffa24a9e1910 libclrjit.so!?? at ??:0:0
     0xffa24af9e180 0xffe3222f56e0 libcoreclr.so!?? at ??:0:0
     0xffa24af9e1f0 0xffe3222f586c libcoreclr.so!?? at ??:0:0
     0xffa24af9e260 0xffe3222f67a0 libcoreclr.so!?? at ??:0:0
     0xffa24af9eef0 0xffe3223453c8 libcoreclr.so!?? at ??:0:0
     0xffa24af9eff0 0xffe322344bf8 libcoreclr.so!?? at ??:0:0
     0xffa24af9f120 0xffe322344220 libcoreclr.so!?? at ??:0:0
     0xffa24af9f1f0 0xffe3223438c0 libcoreclr.so!?? at ??:0:0
     0xffa24af9f2c0 0xffe322387814 libcoreclr.so!?? at ??:0:0
     0xffa24af9f3e0 0xffe322386a28 libcoreclr.so!?? at ??:0:0
     0xffa24af9f500 0xffe322385dc0 libcoreclr.so!?? at ??:0:0
     0xffa24af9f580 0xffe322385bf8 libcoreclr.so!?? at ??:0:0
     0xffa24af9f5c0 0xffe322380ac0 libcoreclr.so!?? at ??:0:0
     0xffa24af9f740 0xffe322381040 libcoreclr.so!?? at ??:0:0
     0xffa24af9f770 0xffe322385b40 libcoreclr.so!?? at ??:0:0
     0xffa24af9f7b0 0xffe3227ff58c libcoreclr.so!?? at ??:0:0
     0xffa24af9f860 0xffe322c93ce0 ld-musl-aarch64.so.1!
    ----------------------------------
    
  3. hoyosjs commented on Nov 15, 2023

    @hoyosjs
    Member

    This looks like lack of symbols on the test machine (for some of these) and then missing function table data in the dump (this is a known issue and the only workaround is to modify the registry).

  4. added this to the 9.0.0 milestone on Jan 17, 2024
  5. ghost removed
    untriagedNew issue has not been triaged by the area owner
    on Jan 17, 2024
  6. JulieLeeMSFT commented on Jan 17, 2024

    @JulieLeeMSFT
    Member

    If it is a missing symbol, we need to find out why it is missing.

  7. hoyosjs commented on Jan 25, 2024

    @hoyosjs
    Member

    @carlossanlop

    I haven't yet found newer examples on linux, but for windows there's two issues.

    • Nothing installs sos so you'll never get managed stacks. See how runtime tests do it:
      <HelixCorrelationPayload Condition="'$(TestWrapperTargetsWindows)' == 'true'" Include="dotnet-sos">
      <Destination>sos</Destination>
      <Uri>https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools/nuget/v3/flat2/dotnet-sos/$(DotnetSosVersion)/dotnet-sos.$(DotnetSosVersion).nupkg</Uri>
      </HelixCorrelationPayload>
      and then the call to dotnet sos install as part of the workitems command.
    • Nothing sets a symbol path. You need to set _NT_SYMBOL_PATH to point to the folder that has runtime symbols. Like this (with the correct payload path):

    set "_NT_SYMBOL_PATH=E:\Temp\libs-crash\correlation-payload;E:\Temp\libs-crash\correlation-payload\shared\Microsoft.NETCore.App\9.0.0"

  8. JulieLeeMSFT commented on Jan 31, 2024

    @JulieLeeMSFT
    Member

    Reassigning to @carlossanlop.

  9. ericstj commented on Jan 31, 2024

    @ericstj
    Member

    Looks like we missed some additional setup steps required for the xunit symbolizer tool. @hoyosjs is there anything to audit here to make sure we copied all the relevant steps from the runtime test deployment into libs?

  10. ghost added
    in-prThere is an active PR which will close this issue when it is merged
    on Feb 14, 2024
  11. ghost removed
    in-prThere is an active PR which will close this issue when it is merged
    on Feb 27, 2024
  12. locked and limited conversation to collaborators on Mar 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions