Repository navigation
hot_mode_dev_cycle_macos_target__benchmark fails on ARM M1 mac #85654
Description
Activity
@keyonghan are there any
Runner*.crashfiles in ~/Library/Logs/DiagnosticReports for that user with a timestamp matching up to a failing test?are there any
Runner*.crashfilesOr
flutter_gallery*.crashorFlutter Gallery*.crashfilesThese are the crash list in mac5, but no mentioned ones:
mac5_crash.txt@jmagman any other logs that we can collect to help find the root cause?
Since I couldn't reproduce on my machine, maybe it's something with the Xcode certs?
Is there any way I can get asudo sysdiagnose -Q(the-Qis optional, but so it doesn't take so long to run)? I'm looking for logging related to the sandbox or security.@jmagman generated the system diagnose and shared from drive. This is the link: https://drive.google.com/file/d/1Woc7iW7dmkuCQoLv6482HhyBh8OdSgRr/view?usp=sharing
Thanks @godofredoc.
Crashing with an ad-hoc codesign failure:
default 2021-10-14 05:47:30.277631 -0700 kernel AMFI: '/opt/s/w/ir/x/t/edited_flutter_gallery/build/macos/Build/Products/Debug/flutter_gallery.app/Contents/MacOS/flutter_gallery' is adhoc signed. default 2021-10-14 05:47:30.280885 -0700 amfid Basic requirement validation failed, error: (null) default 2021-10-14 05:47:30.280947 -0700 amfid /opt/s/w/ir/x/t/edited_flutter_gallery/build/macos/Build/Products/Debug/flutter_gallery.app/Contents/MacOS/flutter_gallery signature not valid: -67050Process: flutter_gallery [37965] Path: /opt/s/*/flutter_gallery.app/Contents/MacOS/flutter_gallery Identifier: flutter_gallery Version: 1.0.0 (1) Code Type: X86-64 (Translated) Parent Process: dart [37517] Responsible: Python [98] User ID: 502 Date/Time: 2021-10-14 05:47:31.253 -0700 OS Version: macOS 11.4 (20F71) Report Version: 12 Anonymous UUID: 78CEF10A-3E9D-4960-69A1-5347A00D81DF Time Awake Since Boot: 13000 seconds System Integrity Protection: disabled Crashed Thread: 0 Dispatch queue: com.apple.main-thread Exception Type: EXC_BAD_INSTRUCTION (SIGILL) Exception Codes: 0x0000000000000001, 0x0000000000000000 Exception Note: EXC_CORPSE_NOTIFY Termination Signal: Illegal instruction: 4 Termination Reason: Namespace SIGNAL, Code 0x4 Terminating Process: exc handler [37965] Application Specific Information: dyld: launch, running initializers /usr/lib/libSystem.B.dylib Incoming message euid:502 does not match secinitd uid:0. Application Specific Signatures: Incoming message euid:502 does not match secinitd uid:0. Thread 0 Crashed:: Dispatch queue: com.apple.main-thread 0 libsystem_secinit.dylib 0x00007fff2a698118 _libsecinit_appsandbox.cold.6 + 49 1 libsystem_secinit.dylib 0x00007fff2a6974d4 _libsecinit_appsandbox + 1955 2 libsystem_trace.dylib 0x00007fff200d9de4 _os_activity_initiate_impl + 51 3 libsystem_secinit.dylib 0x00007fff2a696cf0 _libsecinit_initializer + 67 4 libSystem.B.dylib 0x00007fff2a6ac62f libSystem_initializer + 286 5 dyld 0x0000000204eae6c7 ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 535 6 dyld 0x0000000204eaead2 ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 40 7 dyld 0x0000000204ea94b6 ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 492 8 dyld 0x0000000204ea9421 ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 343 9 dyld 0x0000000204ea9421 ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 343 10 dyld 0x0000000204ea9421 ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 343 11 dyld 0x0000000204ea9421 ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 343 12 dyld 0x0000000204ea9421 ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 343 13 dyld 0x0000000204ea9421 ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 343 14 dyld 0x0000000204ea726f ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 191 15 dyld 0x0000000204ea7310 ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 82 16 dyld 0x0000000204e938b1 dyld::initializeMainExecutable() + 199 17 dyld 0x0000000204e99eb2 dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**, unsigned long*) + 8702 18 dyld 0x0000000204e92224 dyldbootstrap::start(dyld3::MachOLoaded const*, int, char const**, dyld3::MachOLoaded const*, unsigned long*) + 450 19 dyld 0x0000000204e92025 _dyld_start + 37 Thread 1:: com.apple.rosetta.exceptionserver 0 runtime 0x00007ffdfff57858 0x7ffdfff56000 + 6232 1 runtime 0x00007ffdfff61d50 0x7ffdfff56000 + 48464 2 runtime 0x00007ffdfff62a20 0x7ffdfff56000 + 51744 Thread 0 crashed with X86 Thread State (64-bit): rax: 0x0000000000000038 rbx: 0x000000030db9f590 rcx: 0xe94993be722a0073 rdx: 0x0000000000000000 rdi: 0x0000000000000000 rsi: 0x00007fff2a698de2 rbp: 0x000000030db9f4d0 rsp: 0x000000030db9f4c0 r8: 0x000000030db9f218 r9: 0x0000000000000000 r10: 0x00007fff2a698de0 r11: 0xffffffffffffffff r12: 0x00007ff804410c2c r13: 0x0000000000000000 r14: 0x00007ff804410c2c r15: 0x000000010cec5510 rip: 0x00007fff2a698118 rfl: 0x0000000000000202Logs show the build settings:
xcrun xcodebuild -workspace /opt/s/w/ir/x/t/edited_flutter_gallery/macos/Runner.xcworkspace -configuration Debug -scheme Runner -derivedDataPath /opt/s/w/ir/x/t/edited_flutter_gallery/build/macos OBJROOT=/opt/s/w/ir/x/t/edited_flutter_gallery/build/macos/Build/Intermediates.noindex SYMROOT=/opt/s/w/ir/x/t/edited_flutter_gallery/build/macos/Build/Products VERBOSE_SCRIPT_LOGGING=YES COMPILER_INDEX_STORE_ENABLE=NO PROVISIONING_PROFILE_SPECIFIER=match Development * DEVELOPMENT_TEAM=S8QB4VV633 CODE_SIGN_STYLE=ManualBut it's not using
S8QB4VV633, it's signing ad-hoc (no cert):[hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stdout: [ ] /usr/bin/codesign --force --sign - --preserve-metadata=identifier,entitlements '/opt/s/w/ir/x/t/edited_flutter_gallery/build/macos/Build/Products/Debug/flutter_gallery.app/Contents/Frameworks/url_launcher_macos.framework' &I don't think this is related to it being an M1 machine, it's some cert/keychain/some other code signing issue.
7 remaining items
I was thinking it was missing the window server when ssh'd in, but in the sysdiagnose (ps.txt) I see:
_windowserver 88 138 1 410c 0.0 0.4 79 0 409231504 67312 - ?? Ss 5:48AM 0:01.87 /System/Library/PrivateFrameworks/SkyLight.framework/Resources/WindowServer -daemonDid you take that sysdiagnose while ssh'd in, or did you log in through the GUI to do it?
I collected them from a VNC session. Do you want me to collect another one over an ssh session?I collected them from a VNC session. Do you want me to collect another one over an ssh session?
Yes, that would be more accurate. I suspect the lack of windowserver is the problem.
If the user is logged in vnc, and then you start an ssh session with the same user, do the tests pass?I did the following:
- Open ssh session in a terminal
- Open VNC to see the UI interactions
- Ran the test from the ssh session and it failed as expected
- Ran the test from a terminal in the vnc session and it succeded
- Collected syslog from the ssh session. Here is the link https://drive.google.com/file/d/1c9IbE3LadUOuHmVn2VQrBznJYX-oSCO3/view?usp=sharing
By the way if you want to run some experimentation you can run the following in mac-4:
# ssh or vnc into the machine using the swarming user cd ~/working sh test.sh | tee output.txt@jmagman seems like we need new entitlements for the application to be able to run over ssh:
The application cannot be opened for an unexpected reason, error=Error Domain=NSOSStatusErrorDomain Code=-10826 "kLSNoLaunchPermissionErr: User doesn't have permission to launch the app (managed networks)" UserInfo={_LSFunction=_LSLaunchWithRunningboard, _LSLine=2436, NSUnderlyingError=0x115f1e630 {Error Domain=RBSRequestErrorDomain Code=5 "Launch failed." UserInfo={NSLocalizedFailureReason=Launch failed., NSUnderlyingError=0x115f1d270 {Error Domain=OSLaunchdErrorDomain Code=125 "Domain does not support specified action" UserInfo={NSLocalizedFailureReason=Domain does not support specified action}}}}}By the way if you want to run some experimentation you can run the following in mac-4:
# ssh or vnc into the machine using the swarming user cd ~/working sh test.sh | tee output.txtI just tried this
sshd into the machine and the test passed?Task result: { "success": true, "data": { "hotReloadInitialDevFSSyncMilliseconds": 246, "hotRestartMillisecondsToFrame": 332, "hotReloadMillisecondsToFrame": 343, "hotReloadDevFSSyncMilliseconds": 31, "hotReloadFlutterReassembleMilliseconds": 309, "hotReloadVMReloadMilliseconds": 0, "hotReloadMillisecondsToFrameAfterChange": 539, "hotReloadDevFSSyncMillisecondsAfterChange": 118, "hotReloadFlutterReassembleMillisecondsAfterChange": 332, "hotReloadVMReloadMillisecondsAfterChange": 85, "hotReloadInitialDevFSSyncAfterRelaunchMilliseconds": 239, "hotReloadMillisecondsToFrameAfterMediumChange": 543, "hotReloadDevFSSyncMillisecondsAfterMediumChange": 44, "hotReloadFlutterReassembleMillisecondsAfterMediumChange": 411, "hotReloadVMReloadMillisecondsAfterMediumChange": 85, "hotReloadMillisecondsToFrameAfterLargeChange": 972, "hotReloadDevFSSyncMillisecondsAfterLargeChange": 259, "hotReloadFlutterReassembleMillisecondsAfterLargeChange": 628, "hotReloadVMReloadMillisecondsAfterLargeChange": 82 }, "detailFiles": [], "benchmarkScoreKeys": [ "hotReloadInitialDevFSSyncMilliseconds", "hotRestartMillisecondsToFrame", "hotReloadMillisecondsToFrame", "hotReloadDevFSSyncMilliseconds", "hotReloadFlutterReassembleMilliseconds", "hotReloadVMReloadMilliseconds", "hotReloadMillisecondsToFrameAfterChange", "hotReloadDevFSSyncMillisecondsAfterChange", "hotReloadFlutterReassembleMillisecondsAfterChange", "hotReloadVMReloadMillisecondsAfterChange", "hotReloadInitialDevFSSyncAfterRelaunchMilliseconds", "hotReloadMillisecondsToFrameAfterMediumChange", "hotReloadDevFSSyncMillisecondsAfterMediumChange", "hotReloadFlutterReassembleMillisecondsAfterMediumChange", "hotReloadVMReloadMillisecondsAfterMediumChange", "hotReloadMillisecondsToFrameAfterLargeChange", "hotReloadDevFSSyncMillisecondsAfterLargeChange", "hotReloadFlutterReassembleMillisecondsAfterLargeChange", "hotReloadVMReloadMillisecondsAfterLargeChange" ], "reason": "success" } ═════╡ ••• Finished task "hot_mode_dev_cycle_macos_target__benchmark" ••• ╞═════ ═══╡ ••• Flaky status for "hot_mode_dev_cycle_macos_target__benchmark" ••• ╞════ Total 1 executions: 1 success flaky: false flutter-devicelab-mac-4:working swarming$It fails consistently for me when running from the ssh terminal. Did you by any chance run it from a terminal inside the vnc session?
Did you by any chance run it from a terminal inside the vnc session?
I used Secure Shell and ssh'd into the bridge and then mac-4.
Do we also need to remove the defaults from
flutter/dev/devicelab/lib/framework/ios.dart
Line 146 in be8d29f
I removed the ios_signing dependency but the tests are still failing:
Do we also need to remove the defaults from
flutter/dev/devicelab/lib/framework/ios.dart
Line 146 in be8d29f
That's not being hit for this test, I'll fix that for #90892 separately.
I'm not sure what the right incantation is needed to get this working with the sandbox. Let's just remove it for now #92065, having some testing in place is better than it being perfect.
This is now passing consistently on Mac M1s.
Thanks @godofredoc!
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.- locked as resolved and limited conversation to collaborators
on Nov 3, 2021
hot_mode_dev_cycle_macos_target__benchmarkis still failing on arm Macs flutter-devicelab-mac-4 and flutter-devicelab-mac-5, though it's passing locally for me.There's still something off about that CI environment?
https://luci-milo.appspot.com/p/flutter/builders/prod/Mac_ios_staging%20hot_mode_dev_cycle_macos_target__benchmark/637