[Mac-14] macOS xcodebuild test fails with "Test runner never began executing tests after launching"

Testing failed:
[2024-05-29 11:19:23.371464] [STDOUT] stderr: 	plugintest_example (5844) encountered an error (Test runner never began executing tests after launching)
[2024-05-29 11:19:23.371488] [STDOUT] stderr: 
[2024-05-29 11:19:23.371510] [STDOUT] stderr: ** TEST FAILED **

https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac_arm64%20macos_platform_tests%20master%20-%20packages/2/overview
https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac%20plugin_test_macos/1/overview
https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac%20plugin_test_macos/2/overview
https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac%20plugin_test_macos/3/overview

Is this related to #145143?

https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac%20plugin_test_macos/1/overview xcresults zip have a spindump attached, which shows process plugintest_example waiting for an xpc call to secinitd while it's dyld loading:

  Thread 0x1eb63    DispatchQueue 8295628152    499 samples (1-499)    priority 46 (base 46)
  499  start + 1844 (dyld + 24284) [0x18658dedc]
    499  dyld4::prepare(dyld4::APIs&, dyld3::MachOAnalyzer const*) + 3156 (dyld + 28540) [0x18658ef7c]
      499  dyld4::APIs::runAllInitializersForMain() + 84 (dyld + 263004) [0x1865c835c]
        499  dyld4::PrebuiltLoader::runInitializers(dyld4::RuntimeState&) const + 44 (dyld + 159016) [0x1865aed28]
          499  dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 524 (dyld + 118896) [0x1865a5070]
            499  dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 516 (dyld + 388636) [0x1865e6e1c]
              499  dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 192 (dyld + 334128) [0x1865d9930]
                499  dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 300 (dyld + 8956) [0x18658a2fc]
                  499  invocation function for block in dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 496 (dyld + 338332) [0x1865da99c]
                    499  invocation function for block in dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 340 (dyld + 389896) [0x1865e7308]
                      499  invocation function for block in dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const::$_0::operator()() const + 168 (dyld + 135260) [0x1865a905c]
                        499  libSystem_initializer + 280 (libSystem.B.dylib + 5732) [0x193595664]
                          499  _libsecinit_initializer + 80 (libsystem_secinit.dylib + 6236) [0x19357e85c]
                            499  _os_activity_initiate_impl + 64 (libsystem_trace.dylib + 30104) [0x186668598]
                              499  _libsecinit_appsandbox + 592 (libsystem_secinit.dylib + 6912) [0x19357eb00]
                                499  _xpc_pipe_routine + 392 (libxpc.dylib + 200552) [0x186646f68]
                                  499  _xpc_pipe_mach_msg + 56 (libxpc.dylib + 202640) [0x186647790]
                                    499  mach_msg + 24 (libsystem_kernel.dylib + 5496) [0x1868d6578]
                                      499  mach_msg_overwrite + 476 (libsystem_kernel.dylib + 40500) [0x1868dee34]
                                        499  mach_msg2_trap + 8 (libsystem_kernel.dylib + 4596) [0x1868d61f4]
                                         *499  ??? (kernel.release.t8103 + 231580) [0xfffffe000855889c] (blocked by turnstile waiting for secinitd [445] thread 0x1c040 after 2 hops)

man secinitd says:

secinitd is a system daemon that initializes the runtime security policies for processes.

The secinitd process is waiting on -[AppSandboxRequest displaySharingConsentPrompt:] so probably some UI popping up asking for permissions:

  Thread 0x1c040    DispatchQueue "io.flutter.devicelab.plugintestExample"(392)    499 samples (1-499)    priority 46 (base 46)
...
                                  499  __45-[AppSandboxRequest executeRequestWithError:]_block_invoke + 156 (AppSandbox + 36260) [0x19c289da4]
                                    499  -[AppSandboxRequest appsandboxContainerSync:] + 2436 (AppSandbox + 32192) [0x19c288dc0]
                                      499  -[AppSandboxRequest displaySharingConsentPrompt:] + 660 (AppSandbox + 29164) [0x19c2881ec]
                                        499  CFUserNotificationReceiveResponse + 320 (CoreFoundation + 1529516) [0x186aed6ac]

spindump.txt

I was able to reproduce this locally. The pop up is:

I took a sudo spindump and I see the same stacks:

  Thread 0x1d686c    DispatchQueue 8249589112    1001 samples (1-1001)    priority 46 (base 46)
  1001  start + 1844 (dyld + 24284) [0x1839a5edc]
    1001  dyld4::prepare(dyld4::APIs&, dyld3::MachOAnalyzer const*) + 3156 (dyld + 28540) [0x1839a6f7c]
      1001  dyld4::APIs::runAllInitializersForMain() + 84 (dyld + 263004) [0x1839e035c]
        1001  dyld4::PrebuiltLoader::runInitializers(dyld4::RuntimeState&) const + 44 (dyld + 159016) [0x1839c6d28]
...
                                         *1001  ??? (kernel.release.t6000 + 231580) [0xfffffe000857c89c] (blocked by turnstile waiting for secinitd [861] thread 0x1d69a3 after 2 hops)

and

Process:          secinitd [861]
...
                                  1001  __45-[AppSandboxRequest executeRequestWithError:]_block_invoke + 156 (AppSandbox + 36260) [0x1996a1da4]
                                    1001  -[AppSandboxRequest appsandboxContainerSync:] + 2436 (AppSandbox + 32192) [0x1996a0dc0]
                                      1001  -[AppSandboxRequest displaySharingConsentPrompt:] + 660 (AppSandbox + 29164) [0x1996a01ec]
                                        1001  CFUserNotificationReceiveResponse + 320 (CoreFoundation + 1529516) [0x183f056ac]

https://developer.apple.com/documentation/updates/security#June-2023)

App Sandbox now associates your macOS app with its sandbox container using its code signature. The operating system asks the person using your app to grant permission if it tries to access a sandbox container associated with a different app. For more information, see Accessing files from the macOS App Sandbox.

And that link explains why this is happening on a macOS 14 update:

In macOS 14 and later, the operating system uses your app’s code signature to associate it with its sandbox container. If your app tries to access the sandbox container owned by another app, the system asks the person using your app whether to grant access. If the person denies access and your app is already running, then it can’t read or write the files in the other app’s sandbox container. If the person denies access while your app is launching and trying to enter the other app’s sandbox container, your app fails to launch.

The operating system also tracks the association between an app’s code signing identity and its sandbox container for helper tools, including launch agents. If a person denies permission for a launch agent to enter its sandbox container and the app fails to start, launchd starts the launch agent again and the operating system re-requests access.

In 2021 Quinn said:

Xcode 13 includes significant changes in how it handles Info.plist properties and entitlements. The source of truth for some items is now a build setting rather than the property list. In your specific example, in Xcode 12 and earlier the source of truth for the App Sandbox setting was the presence of com.apple.security.app-sandbox in your entitlements file, whereas now it’s the ENABLE_APP_SANDBOX build setting.

https://forums.developer.apple.com/forums/thread/691164?answerId=689678022#689678022

One solution would be to turn off the sandbox in CI:

From the create template we could remove:

flutter/packages/flutter_tools/templates/app_shared/macos.tmpl/Runner/DebugProfile.entitlements

Lines 5 to 6 in 488fb09

<key>com.apple.security.app-sandbox</key>

<true/>

flutter/packages/flutter_tools/templates/app_shared/macos.tmpl/Runner/Release.entitlements

Lines 5 to 6 in 488fb09

<key>com.apple.security.app-sandbox</key>

<true/>
Add ENABLE_APP_SANDBOX=YES to the macOS Xcode project template
In the test runner an environment variable to override that FLUTTER_XCODE_ENABLE_APP_SANDBOX=NO.

I don't think this would be worth a project migration, since the entitlement seems to be working fine, just I don't know of a way to easily override an entitlement key without editing the plist.

Perhaps another option could also be to generate a entitlement file for CI with it disabled and override CODE_SIGN_ENTITLEMENTS. Not sure if it would work, but may be worth a shot

2. Add ENABLE_APP_SANDBOX=YES to the macOS Xcode project template

Hm, on second thought my plan isn't a good idea, I just created a new blank macOS app in Xcode 15.4 and there's an entitlements file with com.apple.security.app-sandbox set to YES, and ENABLE_APP_SANDBOX is not set at all. We're better off hewing to the Xcode defaults.

Actually, codesigning works on these chromium bots, but we're telling it not to codesign to run the tests. Removing these lines might be all we have to do:

flutter/dev/devicelab/bin/tasks/native_ui_tests_macos.dart

Line 34 in 557a959

skipCodesign: true,

flutter/dev/devicelab/lib/tasks/plugin_tests.dart

Line 333 in 557a959

skipCodesign: true,

Disable sandboxing for macOS apps and tests in CI (#149618)

Disable sandboxing for macOS tests in CI (#6866)

Fixed by #149618 and flutter/packages#6866.

Disable sandboxing for macOS apps and tests in CI (flutter#149618)

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

Navigation Menu

Repository navigation

[Mac-14] macOS `xcodebuild test` fails with "Test runner never began executing tests after launching" #149264

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	<key>com.apple.security.app-sandbox</key>
	<true/>

	<key>com.apple.security.app-sandbox</key>
	<true/>

[Mac-14] macOS xcodebuild test fails with "Test runner never began executing tests after launching" #149264

Description

Activity

martinverhey commented on May 29, 2024

jmagman commented on May 30, 2024

jmagman commented on May 30, 2024

jmagman commented on May 30, 2024

jmagman commented on May 30, 2024

vashworth commented on May 30, 2024

jmagman commented on May 30, 2024

jmagman commented on May 30, 2024

vashworth commented on Jun 6, 2024

github-actions commented on Jun 20, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

[Mac-14] macOS `xcodebuild test` fails with "Test runner never began executing tests after launching" #149264