Skip to content

[wasm][aot] Link step failing for Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests - wasm-opt getting killed #51961

Closed
#63958
#60701
Closed
[wasm][aot] Link step failing for Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests - wasm-opt getting killed#51961
#63958
#60701
Assignees
vargaz
Labels
arch-wasmWebAssembly architecturearea-Codegen-AOT-monowasm-aot-testWebAssembly AOT Test
Milestone
7.0.0

Description

@radical
radical
opened on Apr 27, 2021

System.Text.Json.Tests

Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests, log

emcc : error : '/datadisks/disk1/work/B7A309CE/p/build/emsdk/upstream/bin/wasm-opt --post-emscripten --no-exit-runtime -O2 --low-memory-unused --zero-filled-memory --strip-producers /datadisks/disk1/work/B7A309CE/w/B09909A9/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/B7A309CE/w/B09909A9/e/wasm_build/obj/wasm/for-build/dotnet.wasm -g --mvp-features --enable-mutable-globals'
   failed (received SIGKILL (-9)) [/datadisks/disk1/work/B7A309CE/w/B09909A9/e/publish/ProxyProjectForAOTOnHelix.proj]

[ 2582.708542] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/helix.service,task=wasm-opt,pid=19587,uid=1000
[ 2582.708558] Out of memory: Killed process 19587 (wasm-opt) total-vm:7377176kB, anon-rss:7254096kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:14340kB oom_score_adj:0
[ 2582.925546] oom_reaper: reaped process 19587 (wasm-opt), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

The binlog should have details of why this failed. The binlog is included in the helix data.

Activity

  1. radical
    added
    arch-wasmWebAssembly architecture
    disabled-testThe test is disabled in source code against the issue
    area-Codegen-AOT-mono
    wasm-aot-testWebAssembly AOT Test
    on Apr 27, 2021
  3. ghost added
    untriagedNew issue has not been triaged by the area owner
    on Apr 27, 2021
  4. lewing
    removed
    untriagedNew issue has not been triaged by the area owner
    on Apr 28, 2021
  5. lewing
    added this to the 6.0.0 milestone on Apr 28, 2021

  6. vargaz commented on May 6, 2021

    @vargaz
    vargaz
    on May 6, 2021
    Contributor

    This probably happens because the code is just too big.

  7. radical commented on Jun 10, 2021

    @radical
    radical
    on Jun 10, 2021
    MemberAuthor

    The current failure:

             cache:INFO:  - ok
          "/datadisks/disk1/work/A6A60972/p/build/emsdk/upstream/bin/wasm-ld" @/datadisks/disk1/work/A6A60972/t/emscripten_fqv4w7nk.rsp
     1>emcc : error : '/datadisks/disk1/work/A6A60972/p/build/emsdk/upstream/bin/wasm-ld @/datadisks/disk1/work/A6A60972/t/emscripten_fqv4w7nk.rsp' failed (-9) [/datadisks/disk1/work/A6A60972/w/A34508DD/e/publish/AOTTestProjectForHelix.proj]
     1>/datadisks/disk1/work/A6A60972/p/build/wasm/WasmApp.Native.targets(278,5): error MSB3073: The command "emcc "@/datadisks/disk1/work/A6A60972/p/build/microsoft.netcore.app.runtime.browser-wasm/runtimes/browser-wasm/native/src/emcc-default.rsp" "@/datadisks/disk1/work/A6A60972/w/A34508DD/e/wasm_build/obj/wasm/emcc-link.rsp"" exited with code 1. [/datadisks/disk1/work/A6A60972/w/A34508DD/e/publish/AOTTestProjectForHelix.proj]
     1>Done Building Project "/datadisks/disk1/work/A6A60972/w/A34508DD/e/publish/AOTTestProjectForHelix.proj" (default targets) -- FAILED.
  8. radical
    changed the title [-][wasm][aot] `wasm-opt` fails for `System.Text.Json.Tests`[/-] [+][wasm][aot] `wasm-ld` fails for `System.Text.Json.Tests`[/+] on Jun 15, 2021
  9. radical
    added
    blocking-aot-tracking-pr
    on Jun 15, 2021

  10. lewing commented on Jun 28, 2021

    @lewing
    lewing
    on Jun 28, 2021
    Member

    This should be fixed by #53603 and the flto changes

  11. lewing
    closed this as completedon Jun 28, 2021
  12. ghost locked as resolved and limited conversation to collaborators on Jul 28, 2021

  13. 27 remaining items

  14. directhex commented on Jan 20, 2022

    @directhex
    directhex
    on Jan 20, 2022
    Contributor

    Okay. Findings:

    • The job in question runs on Helix on a bare D2a_v4 Azure VM (2 cores of AMD Epyc, 8GB RAM)
    • Serializing the build, and using a less-memory-intensive compiler optimization level are about the best we can do to reduce the memory usage, and we already do them
    • The Clang compile stage requires 7.8 gig of RAM. If it's failing on the clang step in CI, then other basic OS stuff on the VM is likely using enough RAM that Clang can't get the 7.8 gig it wants
    • The final step of the build, the call to wasm-opt, requires more memory than Clang - 8.5 gig.
    • Swap exists 😬

  15. vargaz commented on Jan 20, 2022

    @vargaz
    vargaz
    on Jan 20, 2022
    Contributor

    The problem is AOTing roslyn, a linked Microsoft.CodeAnalysis.CSharp.dll still contains 30k methods, AOT creates 80k methods out of this because roslyn contains a lot of valuetypes causing a lot of generic instances to be added, running the aot compiler on this assembly alone takes 40 seconds, the .bc -> .o step takes 10 minutes (!).

  16. vargaz commented on Jan 20, 2022

    @vargaz
    vargaz
    on Jan 20, 2022
    Contributor

    We might want to add some functionality to the build system to exclude certain assemblies like roslyn from AOTing.

  17. radical commented on Jan 20, 2022

    @radical
    radical
    on Jan 20, 2022
    MemberAuthor

    We might want to add some functionality to the build system to exclude certain assemblies like roslyn from AOTing.

    We do have that, as a hack (https://github.com/dotnet/runtime/blob/main/src/tasks/AotCompilerTask/MonoAOTCompiler.cs#L488-L489).

  18. directhex commented on Jan 20, 2022

    @directhex
    directhex
    on Jan 20, 2022
    Contributor

    So what's the play, try to shoehorn the test as-is onto Helix, or try to wire up the AOT skip?

  19. vargaz commented on Jan 20, 2022

    @vargaz
    vargaz
    on Jan 20, 2022
    Contributor

    I'd suggest skipping AOTing for these assemblies, they take forever to AOT which also impacts CI performance even if it works.

  20. directhex commented on Jan 21, 2022

    @directhex
    directhex
    on Jan 21, 2022
    Contributor

    I skipped AOT, and I have no idea how to parse this failure https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-63958-merge-c3e46af019234e5a83/normal-Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests/1/console.0b1eb581.log?sv=2019-07-07&se=2022-02-10T15%3A15%3A41Z&sr=c&sp=rl&sig=uuwgEA0aSi%2BkNV%2BWkslo%2FBLzkvgx4guydgtVeRkyidQ%3D

  21. vargaz
    self-assigned this
    on Jan 21, 2022

  22. directhex commented on Jan 21, 2022

    @directhex
    directhex
    on Jan 21, 2022
    Contributor

    Failure is at

    runtime/src/mono/mono/mini/aot-runtime.c

    Lines 2300 to 2303 in 9c8b804

    if (amodule->out_of_date) {
    mono_trace (G_LOG_LEVEL_DEBUG, MONO_TRACE_AOT, "AOT: Module %s is unusable because a dependency is out-of-date.", assembly->image->name);
    if (mono_aot_only)
    g_error ("Failed to load AOT module '%s' while running in aot-only mode because a dependency cannot be found or it is out of date.\n", found_aot_name);
    which makes sense

  23. vargaz commented on Jan 24, 2022

    @vargaz
    vargaz
    on Jan 24, 2022
    Contributor

    Is there a PR which skips the AOT step for these assemblies ?

  24. vargaz commented on Jan 24, 2022

    @vargaz
    vargaz
    on Jan 24, 2022
    Contributor

    An alternative would be compiling with O1, i.e.:

        <EmccLinkOptimizationFlag>-O1</EmccLinkOptimizationFlag>

    Running with this reduced the total runtime from 16 minutes to 11 minutes on my machine, it might solve the OOM problem as well.

  25. directhex commented on Jan 24, 2022

    @directhex
    directhex
    on Jan 24, 2022 via email
    Contributor

  26. vargaz commented on Jan 25, 2022

    @vargaz
    vargaz
    on Jan 25, 2022
    Contributor

    #64294

  27. radical
    closed this as completedin #63958on Feb 15, 2022

  28. elinor-fung commented on Feb 17, 2022

    @elinor-fung
    elinor-fung
    on Feb 17, 2022
    Member

    @directhex @radical - the latest rolling build (which has #63958) still hit this.

    Still looks to be an OOM, but I don't know how to get more information about the exact cause.

    emcc : error : '/datadisks/disk1/work/A50F091E/p/build/emsdk/upstream/bin/wasm-emscripten-finalize --minimize-wasm-changes -g --dyncalls-i64 --dwarf /datadisks/disk1/work/A50F091E/w/B308094D/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/A50F091E/w/B308094D/e/wasm_build/obj/wasm/for-build/dotnet.wasm --detect-features' failed (received SIGKILL (-9)) [/datadisks/disk1/work/A50F091E/w/B308094D/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/A50F091E/p/build/wasm/WasmApp.Native.targets(404,5): error MSB3073: The command "emcc "@/datadisks/disk1/work/A50F091E/p/build/microsoft.netcore.app.runtime.browser-wasm/runtimes/browser-wasm/native/src/emcc-default.rsp" "@/datadisks/disk1/work/A50F091E/p/build/microsoft.netcore.app.runtime.browser-wasm/runtimes/browser-wasm/native/src/emcc-link.rsp" "@/datadisks/disk1/work/A50F091E/w/B308094D/e/wasm_build/obj/wasm/for-build/emcc-link.rsp"" exited with code 1. [/datadisks/disk1/work/A50F091E/w/B308094D/e/publish/ProxyProjectForAOTOnHelix.proj]

    [ 5962.061349] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/helix.service,task=wasm-emscripten,pid=23875,uid=1000
[ 5962.061364] Out of memory: Killed process 23875 (wasm-emscripten) total-vm:5798616kB, anon-rss:5590440kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:11100kB oom_score_adj:0
[ 5962.258803] oom_reaper: reaped process 23875 (wasm-emscripten), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

arch-wasmWebAssembly architecturearea-Codegen-AOT-monowasm-aot-testWebAssembly AOT Test

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions