Repository navigation
Crashes on startup significantly increased after migration to .NET6 #7335
Description
Android application type
Android for .NET (net6.0-android, etc.)
Affected platform version
VS 2022 17.3
Description
After I ported my Xamarin app to .NET6 and publishing it to the store, I noticed a significant increase in startup crashes reported in the Play Store developer console:
#00 pc 0x0000000000051894 /apex/com.android.runtime/lib64/bionic/libc.so (abort)
#00 pc 0x0000000000029b1c /data/app/~~x4yqp23VTm2FT1FLIxOjlg==/myapp.name-joOsfd1aZ0MWgfx31__DBw==/split_config.arm64_v8a.apk!libmono-android.release.so (xamarin::android::internal::MonodroidRuntime::mono_log_handler(char const*, char const*, char const*, int, void*))
#00 pc 0x00000000002680bc /data/app/~~x4yqp23VTm2FT1FLIxOjlg==/myapp.name-joOsfd1aZ0MWgfx31__DBw==/split_config.arm64_v8a.apk!libmonosgen-2.0.so
#00 pc 0x00000000002681e8 /data/app/~~x4yqp23VTm2FT1FLIxOjlg==/myapp.name-joOsfd1aZ0MWgfx31__DBw==/split_config.arm64_v8a.apk!libmonosgen-2.0.so
#00 pc 0x0000000000161ae8 /data/app/~~x4yqp23VTm2FT1FLIxOjlg==/myapp.name-joOsfd1aZ0MWgfx31__DBw==/split_config.arm64_v8a.apk!libmonosgen-2.0.so
#00 pc 0x00000000001dac28 /data/app/~~x4yqp23VTm2FT1FLIxOjlg==/myapp.name-joOsfd1aZ0MWgfx31__DBw==/split_config.arm64_v8a.apk!libmonosgen-2.0.so
#00 pc 0x00000000001dbe64 /data/app/~~x4yqp23VTm2FT1FLIxOjlg==/myapp.name-joOsfd1aZ0MWgfx31__DBw==/split_config.arm64_v8a.apk!libmonosgen-2.0.so
#00 pc 0x00000000001ee468 /data/app/~~x4yqp23VTm2FT1FLIxOjlg==/myapp.name-joOsfd1aZ0MWgfx31__DBw==/split_config.arm64_v8a.apk!libmonosgen-2.0.so
#00 pc 0x0000000000004bf8
I have also experienced these crashes locally on my own devices, e.g. when launching the app repeatedly while profiling using this script.
The crash seems to appear randomly, that means it can crash even when it previously launched successfully and the app will launch successfully even after the crash happened.
This is the logcat output of when the crash happened locally on my own device:
2022-09-02 06:17:59.197 2727-2727/? E/myapp.name: Unknown bits set in runtime_flags: 0x20000
2022-09-02 06:17:59.206 2727-2727/? E/myapp.name: Not starting debugger since process cannot load the jdwp agent.
2022-09-02 06:17:59.363 604-611/? E/cutils: Nothing there yet; let's create it: /storage/emulated/0/Android/data/myapp.name
2022-09-02 06:17:59.364 604-611/? E/cutils: Nothing there yet; let's create it: /storage/emulated/0/Android/data/myapp.name/cache
2022-09-02 06:18:00.272 2727-3092/? A/libc: Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 3092 (.NET ThreadPool), pid 2727 (myapp.name)
2022-09-02 06:18:00.348 3109-3109/? A/DEBUG: pid: 2727, tid: 3092, name: .NET ThreadPool >>> myapp.name <<<
2022-09-02 06:18:00.544 3109-3109/? A/DEBUG: #01 pc 0000000000029b1c /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmono-android.release.so (offset 0x103d000) (xamarin::android::internal::MonodroidRuntime::mono_log_handler(char const*, char const*, char const*, int, void*)+144) (BuildId: 29c5a3805a0bedee1eede9b6668d7c676aa63371)
2022-09-02 06:18:00.544 3109-3109/? A/DEBUG: #02 pc 00000000002680bc /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b)
2022-09-02 06:18:00.544 3109-3109/? A/DEBUG: #03 pc 00000000002681e8 /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b)
2022-09-02 06:18:00.544 3109-3109/? A/DEBUG: #04 pc 000000000008555c /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (mono_metadata_string_heap+188) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b)
2022-09-02 06:18:00.544 3109-3109/? A/DEBUG: #05 pc 00000000000cade4 /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b)
2022-09-02 06:18:00.544 3109-3109/? A/DEBUG: #06 pc 00000000000cb164 /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b)
2022-09-02 06:18:00.544 3109-3109/? A/DEBUG: #07 pc 0000000000045878 /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b)
2022-09-02 06:18:00.544 3109-3109/? A/DEBUG: #08 pc 00000000000440d0 /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b)
2022-09-02 06:18:00.544 3109-3109/? A/DEBUG: #09 pc 0000000000043d94 /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b)
2022-09-02 06:18:00.544 3109-3109/? A/DEBUG: #10 pc 000000000003db64 /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (mono_class_get_field+44) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b)
2022-09-02 06:18:00.544 3109-3109/? A/DEBUG: #11 pc 00000000001dd374 /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b)
2022-09-02 06:18:00.544 3109-3109/? A/DEBUG: #12 pc 00000000001dfc88 /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b)
2022-09-02 06:18:00.544 3109-3109/? A/DEBUG: #13 pc 00000000001dab24 /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b)
2022-09-02 06:18:00.544 3109-3109/? A/DEBUG: #14 pc 00000000001dbe64 /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b)
2022-09-02 06:18:00.544 3109-3109/? A/DEBUG: #15 pc 00000000001ee468 /data/app/myapp.name-B9t_3dF9i8mDxJEKodZw5w==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (offset 0x109b000) (BuildId: 4a5dd4396e8816b7f69881838bd549285213d53b)
I also found it weird that when I ported multiple of my apps to .NET6, first only one app was affected by this increased crash rate.
Later on I pushed an update to an app that was not affected previously and it then started to show the same problem and the crashes jumped significantly. So maybe the process how the app bundle was built has some influence on this. I use both the stable as well as the Preview version of VS 22 and create the app bundle using the IDE and the "Archive..." menu button
Steps to Reproduce
Unfortunately I have no way to reliably reproduce this crash, but when automating app launches and launching the app many times (e.g. using this) I can occasionally see the crash in the logcat output.
Activity
grendello commentedon Sep 5, 2022
@tipa unfortunately, the log fragments don't include enough information. We can see that it's a failed assertion, most likely int he runtime, but we can't see where it happens and what's the
actual message. We will need a more detailed logcat output, alas. On the devices where you can reproduce the issue, could you make sure these commands are ran:
tipa commentedon Sep 6, 2022
@grendello thanks for the info on how to produced more detailed output. The log.txt contained this part:
Appears to be some problem with AOT and System.Text.Json
tipa commentedon Sep 6, 2022
Running it again produces a similar but different stack trace:
log.txt
Edit: Another few tests and it shows this now:
So all somehow related to
System.Text.Json.Is there any quick fix I can do to prevent these crashes? I am currently using profiled AOT with a custom AOT profile.
AndroidLinkModeis set toFullgrendello commentedon Sep 6, 2022
@tipa in your log I spotted this weird thing:
Note the assembly name:
£D0067CAD9A63E0813759A2BB841051CA73570C0DA2E08E840A8EB45DB6A7A010/System.Private.CoreLib- the part before/should not be there. The runtime fails to load it because I doubt we saw this kind ofname at the build time when we process assemblies. Would you be able to come up with a small repro for this issue? I'd need to examine the generated code.
Edit: the part before
/is normally treated as culture name, which it definitely is not what it is in this instancegrendello commentedon Sep 6, 2022
@tipa also, would you be able to find full log output of this crash? This is likely a failed assertion or
abort()as well, but I'd like to see the message (which should be somewhere above theSIGABRTline)tipa commentedon Sep 6, 2022
Not sure what part exactly I should include in a small repro...
Are you sure the problem is caused by the line with
System.Private.CoreLib? In my other crashes there is no mention of this, e.g. here:I don't have the full output of that particular crash any more, but as I am now able to reproduce it faster, I did another log:
log.txt
grendello commentedon Sep 6, 2022
As sure as I can be without seeing the app source (or at least its
obj/build directory) - the "culture" name is definitely invalid and I doubt it was this particular string at the build time. Something's broken there.The above line and another one:
suggest that it's a problem with the linker, it is probably linking out too much. You might try turning off linking completely, just to see if the app works then.
40 remaining items
Tentative fix for issue dotnet#7335, needs more testing
grendello commentedon Jan 24, 2023
PR #7732 fixes the issue for me. I ran the test app provided by @tipa 5000 times in quick succession (on Pixel 6 Pro) without any SIGABRT being signaled.
Before that fix lands, you might work around it by disabling assembly compression (set the
AndroidEnableAssemblyCompressionMSBuild property toFalse)Commit d236af5 introduced embedded assembly compression,
using the lz4 algorithm, which speeds up startup and reduces final package size.
Assemblies are compressed at build time and, at the same time, pre-allocated buffers for
the decompressed data are allocated in
libxamarin-app.so. The buffers are then passedto the LZ4 APIs, all threads using the same output buffer. The assumption was that we can
do fine without locking as even if overlapped decompression happens, the output data will be
the same and so even if two threads do the same thing at the same time, the data will be valid
at all times, so long as at least one thread completes the decompression.
This assumption proved to be largely true, but it appears that in high concurrency cases it
is possible that the data in decompression buffer differs. My guess is that LZ4 either uses the
output buffer as a scratchpad area when decompressing or that it initializes/modifies the buffer
before writing actual data in it. With overlapped decompression, it may lead to one thread
overwriting valid data previously written by another thread, so that when the latter returns the
buffer it thought to have had valid data may contain certain bytes temporarily overwritten by the
decompression session in the other, still running, thread. It may happen that MonoVM reads the
corrupted data just when it is still invalid (before the still running decompression session actually
writes the valid data), a classic race condition.
To fix this, the decompression block is now protected with a startup-aware mutex. Mutex will be held
only after the initial startup phase is completed, so there should not be much loss of startup
performance.
[monodroid] Prevent overlapped decompression of embedded assemblies (#…
[monodroid] Prevent overlapped decompression of embedded assemblies (d…
[monodroid] Prevent overlapped decompression of embedded assemblies (d…
[monodroid] Prevent overlapped decompression of embedded assemblies (d…
[monodroid] Prevent overlapped decompression of embedded assemblies (#…
[monodroid] Prevent overlapped decompression of embedded assemblies (d…
[monodroid] Prevent overlapped decompression of embedded assemblies (#…