Repository navigation
Mac_arm64_ios hot_mode_dev_cycle_ios__benchmark is 2.04% flaky #190995
Description
Activity
Root Cause Analysis
1. Physical Device Launch Behavior (iOS 17+)
On physical iOS 17+ devices, launching debug JIT apps directly via
devicectl device process launchexits immediately withsignal 11(SIGSEGV) due to OS-level memory protection preventing JIT execution without an attached debugger.2. Xcode Debugger Fallback
The Flutter tool catches the launch exit and falls back to opening and debugging the workspace through Xcode using the JXA automation script (
packages/flutter_tools/bin/xcode_debug.js).3. Symlink Mismatch in
getWorkspaceDocumentIn
xcode_debug.js,getWorkspaceDocumentmatches open Xcode workspaces against the target directory using string comparison:if (filePath === args.projectPath || filePath === args.workspacePath)
On Swarming CI bots, working directories contain symbolic links (e.g.,
/opt/s/w/ir/x/t/...). Xcode resolves symlinks when opening documents, returning canonical paths, whereas the tool passes unresolved paths. Because the paths do not match literally:xcode_debug.jsrepeatedly logsFailed to get workspace.- It eventually reaches the 14-minute timeout:
Timed out waiting for workspace to load.
4. Flakiness Mechanism
After the Xcode fallback times out on the physical device, the benchmark harness falls back to running on
iphonesimulator. The simulator run succeeds, but the overall task is marked flaky due to the initial device failure and timeout.
Implemented Fix
1. Canonicalize Paths in JXA (
xcode_debug.js)Import
Foundationand resolve symlinks in both open document paths and command arguments usingNSURL.URLByResolvingSymlinksInPathbefore comparison:ObjC.import('Foundation'); function resolveSymlink(path) { try { return ObjC.unwrap($.NSURL.fileURLWithPath(path).URLByResolvingSymlinksInPath.path); } catch (e) { return path; } }
2. Log Message Cleanup (
core_devices.dart)Fixed a misleading log message that previously reported
lldb exited with code $statuswhen the monitored process was actuallydevicectl device process launch.Canonicalize Paths in JXA (xcode_debug.js)
I don't think we should change this file without very thorough manual testing. I think the AI is likely wrong.
On physical iOS 17+ devices, launching debug JIT apps directly via devicectl device process launch exits immediately with signal 11 (SIGSEGV) due to OS-level memory protection preventing JIT execution without an attached debugger.
We should look into fixing 1, although, I think the AI's assumptions on why it's failing are also wrong.
[2026-08-11 16:49:48.465518] [STDOUT] stdout: [ +5 ms] executing: script -t 0 /dev/null /usr/bin/arch -arm64e xcrun devicectl device process launch --device 00008030-000A704C3C6B402E --start-stopped --console --environment-variables {"OS_ACTIVITY_DT_MODE": "enable"} io.flutter.examples.gallery --enable-dart-profiling --disable-vm-service-publication --enable-checked-mode --verify-entry-points [2026-08-11 16:49:48.562622] [STDOUT] stdout: [ +97 ms] 16:49:48 Acquired tunnel connection to device. [2026-08-11 16:49:48.573548] [STDOUT] stdout: [ +11 ms] 16:49:48 Enabling developer disk image services. [2026-08-11 16:49:48.620663] [STDOUT] stdout: [ +47 ms] 16:49:48 Acquired usage assertion. [2026-08-11 16:49:48.743845] [STDOUT] stdout: [ +123 ms] Launched application with io.flutter.examples.gallery bundle identifier. [2026-08-11 16:49:48.744042] [STDOUT] stdout: [ ] Waiting for the application to terminate… [2026-08-11 16:49:48.744951] [STDOUT] stdout: [ +1 ms] executing: /usr/bin/arch -arm64e xcrun devicectl device info processes --device 00008030-000A704C3C6B402E --json-output /opt/s/w/ir/x/t/flutter_tools.r2R4hX/core_devices.6YBcGT/core_device_process_list.json [2026-08-11 16:49:48.849724] [STDOUT] stdout: [ +104 ms] App terminated due to signal 11. [2026-08-11 16:49:48.849926] [STDOUT] stdout: [ ] Script started, output file is /dev/null [2026-08-11 16:49:48.850054] [STDOUT] stdout: [ ] Script done, output file is /dev/null [2026-08-11 16:49:48.850229] [STDOUT] stdout: [ ] lldb exited with code 1 [2026-08-11 16:49:48.967489] [STDOUT] stdout: [ +116 ms] 16:49:48 Acquired tunnel connection to device. [2026-08-11 16:49:48.967509] [STDOUT] stdout: 16:49:48 Enabling developer disk image services. [2026-08-11 16:49:48.967512] [STDOUT] stdout: 16:49:48 Acquired usage assertion. [2026-08-11 16:49:48.967515] [STDOUT] stdout: PID Executable Path [2026-08-11 16:49:48.967519] [STDOUT] stdout: --- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ... [2026-08-11 16:49:48.971848] [STDOUT] stdout: 818 /System/Library/CoreServices/ReportCrashIt's appears to be crashing on launch, we likely need to VNC to the bot to get the crash logs to diagnose why it's crashing
Reacted by Ben Konyi[prod pool] flaky ratio for the past (up to) 100 commits between 2026-08-20 and 2026-08-25 is 2.00%. Flaky number: 2; total number: 100.
One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_arm64_ios%20hot_mode_dev_cycle_ios__benchmark/11597
Commit: 40b3290
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_arm64_ios%20hot_mode_dev_cycle_ios__benchmark/11597
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_arm64_ios%20hot_mode_dev_cycle_ios__benchmark/11567Recent test runs:
https://flutter-dashboard.appspot.com/#/build?taskFilter=Mac_arm64_ios%20hot_mode_dev_cycle_ios__benchmarkOkay I was able to get the crash logs and confirmed it is crashing on launch
Exception Type: EXC_BAD_ACCESS (SIGSEGV) Exception Subtype: KERN_INVALID_ADDRESS at 0x0000000000000000 Exception Codes: 0x0000000000000001, 0x0000000000000000 VM Region Info: 0 is not in any region. Bytes before following region: 4341825536 REGION TYPE START - END [ VSIZE] PRT/MAX SHRMOD REGION DETAIL UNUSED SPACE AT START ---> __TEXT 102cb0000-102cfc000 [ 304K] r-x/r-x SM=COW /var/containers/Bundle/Application/C526C1C8-BE3C-4A04-A6CD-E3F0827BAF69/Flutter Gallery.app/Flutter Gallery Termination Reason: SIGNAL 11 Segmentation fault: 11 Terminating Process: exc handler [1258] Triggered by Thread: 0 Thread 0 name: Dispatch queue: com.apple.main-thread Thread 0 Crashed: 0 libswiftCore.dylib 0x18809a66c swift_getObjectType + 40 1 Flutter Gallery 0x102cba790 static URLLauncherPlugin.register(with:) + 172 2 Flutter Gallery 0x102cba9b8 @objc static URLLauncherPlugin.register(with:) + 56 3 Flutter Gallery 0x102cb4208 +[GeneratedPluginRegistrant registerWithRegistry:] + 188 4 Flutter Gallery 0x102cb4060 -[AppDelegate application:didFinishLaunchingWithOptions:] + 96 5 UIKitCore 0x1910c2f84 -[UIApplication _handleDelegateCallbacksWithOptions:isSuspended:restoreState:] + 339 6 UIKitCore 0x1910c1d70 -[UIApplication _callInitializationDelegatesWithActions:forScene:payload:fromOriginatingProcess:] + 3231 7 UIKitCore 0x1910bda14 -[UIApplication _runWithMainScene:transitionContext:completion:] + 819 8 UIKitCore 0x1910bd108 -[_UISceneLifecycleMultiplexer completeApplicationLaunchWithFBSScene:transitionContext:] + 115 9 UIKitCore 0x191131fac _UIScenePerformActionsWithLifecycleActionMask + 111 10 UIKitCore 0x19112dcc0 __101-[_UISceneLifecycleMultiplexer _evalTransitionToSettings:fromSettings:forceExit:withTransitionStore:]_block_invoke + 227 11 UIKitCore 0x1911301f0 -[_UISceneLifecycleMultiplexer _performBlock:withApplicationOfDeactivationReasons:fromReasons:] + 211 12 UIKitCore 0x191132988 -[_UISceneLifecycleMultiplexer _evalTransitionToSettings:fromSettings:forceExit:withTransitionStore:] + 587 13 UIKitCore 0x191130938 -[_UISceneLifecycleMultiplexer uiScene:transitionedFromState:withTransitionContext:] + 243 14 UIKitCore 0x1911349ec __186-[_UIWindowSceneFBSSceneTransitionContextDrivenLifecycleSettingsDiffAction _performActionsForUIScene:withUpdatedFBSScene:settingsDiff:fromSettings:transitionContext:lifecycleActionType:]_block_invoke + 159 15 UIKitCore 0x1911348e4 +[BSAnimationSettings(UIKit) tryAnimatingWithSettings:fromCurrentState:actions:completion:] + 731 16 UIKitCore 0x1911345cc _UISceneSettingsDiffActionPerformChangesWithTransitionContextAndCompletion + 223 17 UIKitCore 0x1911323c8 -[_UIWindowSceneFBSSceneTransitionContextDrivenLifecycleSettingsDiffAction _performActionsForUIScene:withUpdatedFBSScene:settingsDiff:fromSettings:transitionContext:lifecycleActionType:] + 303 18 UIKitCore 0x1911343d0 __64-[UIScene scene:didUpdateWithDiff:transitionContext:completion:]_block_invoke.218 + 647 19 UIKitCore 0x191133cf0 -[UIScene _emitSceneSettingsUpdateResponseForCompletion:afterSceneUpdateWork:] + 211 20 UIKitCore 0x191133b64 -[UIScene scene:didUpdateWithDiff:transitionContext:completion:] + 219 21 UIKitCore 0x1910d8320 -[UIApplication workspace:didCreateScene:withTransitionContext:completion:] + 499 22 UIKitCore 0x1910e33a4 -[UIApplicationSceneClientAgent scene:didInitializeWithEvent:completion:] + 303 23 FrontBoardServices 0x1abf6c438 __95-[FBSScene _callOutQueue_didCreateWithTransitionContext:alternativeCreationCallout:completion:]_block_invoke + 363 24 FrontBoardServices 0x1abf678ac -[FBSScene _callOutQueue_maybeCoalesceClientSettingsUpdates:] + 131 25 FrontBoardServices 0x1abf6c1dc -[FBSScene _callOutQueue_didCreateWithTransitionContext:alternativeCreationCallout:completion:] + 431 26 FrontBoardServices 0x1abfd62c4 __93-[FBSWorkspaceScenesClient _callOutQueue_sendDidCreateForScene:transitionContext:completion:]_block_invoke.349 + 251 27 FrontBoardServices 0x1abf6b520 -[FBSWorkspace _calloutQueue_executeCalloutFromSource:withBlock:] + 175 28 FrontBoardServices 0x1abf6b898 -[FBSWorkspaceScenesClient _callOutQueue_sendDidCreateForScene:transitionContext:completion:] + 471 29 FrontBoardServices 0x1abf6b694 __92-[FBSWorkspaceScenesClient createSceneWithIdentity:parameters:transitionContext:completion:]_block_invoke_2 + 255 30 FrontBoardServices 0x1abf6b520 -[FBSWorkspace _calloutQueue_executeCalloutFromSource:withBlock:] + 175 31 libdispatch.dylib 0x1c57d11e4 _dispatch_client_callout + 15 32 libdispatch.dylib 0x1c57bc694 _dispatch_block_invoke_direct + 283 33 BoardServices 0x1a3538f84 __BSSERVICEMAINRUNLOOPQUEUE_IS_CALLING_OUT_TO_A_BLOCK__ + 51 34 BoardServices 0x1a3538e00 BSServiceMainRunLoopSourceHandler + 223 35 CoreFoundation 0x18b435358 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 27 36 CoreFoundation 0x18b4352cc __CFRunLoopDoSource0 + 171 37 CoreFoundation 0x18b3fa5a8 __CFRunLoopDoSources0 + 231 38 CoreFoundation 0x18b3c41a0 __CFRunLoopRun + 819 39 CoreFoundation 0x18b3c354c _CFRunLoopRunSpecificWithOptions + 531 40 GraphicsServices 0x230b2b498 GSEventRunModal + 119 41 UIKitCore 0x1910bc244 -[UIApplication _run] + 795 42 UIKitCore 0x191027158 UIApplicationMain + 331 43 Flutter Gallery 0x102cb411c main + 96 44 dyld 0x187fd1c1c start + 6927 Thread 1 name: Dispatch queue: com.apple.libtrace.state.block-list Thread 1: 0 libsystem_kernel.dylib 0x23a399120 __ulock_wait + 8 1 libdispatch.dylib 0x1c57b95d8 _dispatch_thread_main_event_wait_slow + 75 2 libdispatch.dylib 0x1c57c764c __DISPATCH_WAIT_FOR_QUEUE__ + 463 3 libdispatch.dylib 0x1c57c7178 _dispatch_sync_f_slow + 139 4 libsystem_trace.dylib 0x1b0d461b8 ___os_state_request_for_self_block_invoke + 371 5 libdispatch.dylib 0x1c57b79a8 _dispatch_call_block_and_release + 31 6 libdispatch.dylib 0x1c57d11e4 _dispatch_client_callout + 15 7 libdispatch.dylib 0x1c57bffb0 _dispatch_lane_serial_drain + 739 8 libdispatch.dylib 0x1c57c0ae4 _dispatch_lane_invoke + 447 9 libdispatch.dylib 0x1c57cadac _dispatch_root_queue_drain_deferred_wlh + 283 10 libdispatch.dylib 0x1c57ca6ac _dispatch_workloop_worker_thread + 719 11 libsystem_pthread.dylib 0x1ea5893b0 _pthread_wqthread + 291 12 libsystem_pthread.dylib 0x1ea5888c0 start_wqthread + 7 Thread 2 name: Dispatch queue: com.apple.UIKit.KeyboardManagement Thread 2: 0 libsystem_kernel.dylib 0x23a399120 __ulock_wait + 8 1 libdispatch.dylib 0x1c57b95d8 _dispatch_thread_main_event_wait_slow + 75 2 libdispatch.dylib 0x1c57c764c __DISPATCH_WAIT_FOR_QUEUE__ + 463 3 libdispatch.dylib 0x1c57c7178 _dispatch_sync_f_slow + 139 4 UIKitCore 0x1923e6c80 __37-[_UIRemoteKeyboards startConnection]_block_invoke.449 + 143 5 CoreFoundation 0x18b3f88f4 __invoking___ + 147 6 CoreFoundation 0x18b3f8778 -[NSInvocation invoke] + 423 7 Foundation 0x188668734 <deduplicated_symbol> + 15 8 Foundation 0x1886620b0 -[NSXPCConnection _decodeAndInvokeReplyBlockWithEvent:sequence:replyInfo:] + 527 9 Foundation 0x188661a0c __88-[NSXPCConnection _sendInvocation:orArguments:count:methodSignature:selector:withProxy:]_block_invoke_5 + 187 10 libxpc.dylib 0x19fdedeac _xpc_connection_reply_callout + 119 11 libxpc.dylib 0x19fddf5c4 _xpc_connection_call_reply_async + 95 12 libdispatch.dylib 0x1c57d1214 <deduplicated_symbol> + 15 13 libdispatch.dylib 0x1c57d5238 _dispatch_mach_msg_async_reply_invoke + 339 14 libdispatch.dylib 0x1c57bfe18 _dispatch_lane_serial_drain + 331 15 libdispatch.dylib 0x1c57c0ae4 _dispatch_lane_invoke + 447 16 libdispatch.dylib 0x1c57cadac _dispatch_root_queue_drain_deferred_wlh + 283 17 libdispatch.dylib 0x1c57ca6ac _dispatch_workloop_worker_thread + 719 18 libsystem_pthread.dylib 0x1ea5893b0 _pthread_wqthread + 291 19 libsystem_pthread.dylib 0x1ea5888c0 start_wqthread + 7 Thread 3: Thread 4: Thread 5: Thread 6: Thread 7 name: com.apple.uikit.eventfetch-thread Thread 7: 0 libsystem_kernel.dylib 0x23a392cd4 mach_msg2_trap + 8 1 libsystem_kernel.dylib 0x23a39630c mach_msg2_internal + 75 2 libsystem_kernel.dylib 0x23a39622c mach_msg_overwrite + 423 3 libsystem_kernel.dylib 0x23a396078 mach_msg + 23 4 CoreFoundation 0x18b3fa344 __CFRunLoopServiceMachPort + 159 5 CoreFoundation 0x18b3c4310 __CFRunLoopRun + 1187 6 CoreFoundation 0x18b3c354c _CFRunLoopRunSpecificWithOptions + 531 7 Foundation 0x188631cf0 -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 211 8 Foundation 0x188631bd8 -[NSRunLoop(NSRunLoop) runUntilDate:] + 63 9 UIKitCore 0x191081a7c -[UIEventFetcher threadMain] + 419 10 Foundation 0x1886b5824 __NSThread__start__ + 731 11 libsystem_pthread.dylib 0x1ea58c438 _pthread_start + 135 12 libsystem_pthread.dylib 0x1ea5888cc thread_start + 7 Thread 8: Thread 9: Thread 10: Thread 11: Thread 0 crashed with ARM Thread State (64-bit): x0: 0x0000000000000000 x1: 0x0000000000000000 x2: 0x0000000102d057d0 x3: 0x00000001600823a0 x4: 0x0000000160082340 x5: 0x0000000000000000 x6: 0x0000000102cfcd00 x7: 0xfffff0003ffff800 x8: 0x00000002028f4000 x9: 0x0000000000000103 x10: 0x000000000100006e x11: 0x0000000188526291 x12: 0x0000000001000033 x13: 0xfffffffffffffff4 x14: 0x0000000188526399 x15: 0x00000001f436b238 x16: 0x000000018809a644 x17: 0x00000001880d7500 x18: 0x0000000000000000 x19: 0x0000000000000000 x20: 0x0000000102d06d60 x21: 0x0000000000000001 x22: 0x0000000000000001 x23: 0x0000000000000000 x24: 0x0000000000000001 x25: 0x00000001f43e3000 x26: 0x00000001f1468000 x27: 0x000000002b870064 x28: 0x0000000000000010 fp: 0x000000016d14cad0 lr: 0x0000000102cba790 sp: 0x000000016d14cac0 pc: 0x000000018809a66c cpsr: 0x40000000 far: 0x0000000000000000 esr: 0x92000006 (Data Abort) byte read Translation faultOkay this crash can happen when you launch a debug build: #149214
However, we purposely pass
--start-stoppedtodevicectlso that it will start in a "suspended state" until a debugger is attached.script -t 0 /dev/null /usr/bin/arch -arm64e xcrun devicectl device process launch --device 00008140-000530AC1107001C --start-stopped --console --environment-variables {"OS_ACTIVITY_DT_MODE": "enable"} io.flutter.examples.gallery --enable-dart-profiling --disable-vm-service-publication --enable-checked-mode --verify-entry-points [2026-08-24 19:08:29.924798] [STDOUT] stdout: [ +65 ms] 19:08:29 Acquired tunnel connection to device. [2026-08-24 19:08:29.931403] [STDOUT] stdout: [ +6 ms] 19:08:29 Enabling developer disk image services. [2026-08-24 19:08:29.963093] [STDOUT] stdout: [ +31 ms] 19:08:29 Acquired usage assertion. [2026-08-24 19:08:30.014878] [STDOUT] stdout: [ +51 ms] Launched application with io.flutter.examples.gallery bundle identifier. [2026-08-24 19:08:30.014890] [STDOUT] stdout: [ ] Waiting for the application to terminate… [2026-08-24 19:08:30.059595] [STDOUT] stdout: [ +43 ms] App terminated due to signal 11. [2026-08-24 19:08:30.059801] [STDOUT] stdout: [ ] Script started, output file is /dev/null [2026-08-24 19:08:30.059858] [STDOUT] stdout: [ ] Script done, output file is /dev/null [2026-08-24 19:08:30.060024] [STDOUT] stdout: [ ] devicectl exited with code 1Gemini speculates it could be because "This issue usually happens because the system reuses an already running/suspended instance of your app"
- added a commit that references this issue
on Aug 27, 2026 [prod pool] flaky ratio for the past (up to) 100 commits between 2026-08-27 and 2026-09-01 is 2.00%. Flaky number: 2; total number: 100.
One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_arm64_ios%20hot_mode_dev_cycle_ios__benchmark/11690
Commit: 3796c36
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_arm64_ios%20hot_mode_dev_cycle_ios__benchmark/11690
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_arm64_ios%20hot_mode_dev_cycle_ios__benchmark/11671Recent test runs:
https://flutter-dashboard.appspot.com/#/build?taskFilter=Mac_arm64_ios%20hot_mode_dev_cycle_ios__benchmarkLooks like recent failure was due to the iOS device not being found - let's wait another week
Reacted by John "codefu" McDole[prod pool] flaky ratio for the past (up to) 100 commits between 2026-09-01 and 2026-09-08 is 1.04%. Flaky number: 1; total number: 96.
One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_arm64_ios%20hot_mode_dev_cycle_ios__benchmark/11840
Commit: 2d39b7d
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_arm64_ios%20hot_mode_dev_cycle_ios__benchmark/11840Recent test runs:
https://flutter-dashboard.appspot.com/#/build?taskFilter=Mac_arm64_ios%20hot_mode_dev_cycle_ios__benchmark[prod pool] flaky ratio for the past (up to) 100 commits between 2026-09-09 and 2026-09-15 is 0.00%. Flaky number: 0; total number: 98.
Closing as no longer flaking
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.- locked as resolved and limited conversation to collaborators
on Oct 5, 2026
The post-submit test builder
Mac_arm64_ios hot_mode_dev_cycle_ios__benchmarkhad a flaky ratio 2.04% for the past (up to) 100 commits, which is above our 2.00% threshold.One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_arm64_ios%20hot_mode_dev_cycle_ios__benchmark/11318
Commit: 537c9d1
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_arm64_ios%20hot_mode_dev_cycle_ios__benchmark/11318
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_arm64_ios%20hot_mode_dev_cycle_ios__benchmark/11249
Recent test runs:
https://flutter-dashboard.appspot.com/#/build?taskFilter=Mac_arm64_ios%20hot_mode_dev_cycle_ios__benchmark
Please follow https://github.com/flutter/flutter/blob/master/docs/infra/Reducing-Test-Flakiness.md#fixing-flaky-tests to fix the flakiness and enable the test back after validating the fix (internal dashboard to validate: go/flutter_test_flakiness).