Some macOS tests are flaky and timing out.

The post-submit test builder Mac build_tests_3_4 had a flaky ratio 3.15% for the past 15 days, which is above our 2.00% threshold.

One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20build_tests_3_4/1834
Commit: 1422bfa
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20build_tests_3_4/1834
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20build_tests_3_4/1778
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20build_tests_3_4/1777
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20build_tests_3_4/1773

Succeeded builds (3 most recent):
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20build_tests_3_4/1838
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20build_tests_3_4/1837
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20build_tests_3_4/1836

Please follow https://github.com/flutter/flutter/wiki/Reducing-Test-Flakiness#fixing-flaky-tests to fix the flakiness and enable the test back after validating the fix (internal dashboard to validate: go/flutter_test_flakiness).

@jmagman

Successful runs can take > 20 minutes, so I'm wondering if the 30 minute timeout is too tight. That is the cause for some of the failures.

The other listed failures are happening because it looks like the version of XCode is not right:

[   +9 ms] Found "Xcode 12.0, Build version 12A7209". Xcode 12.0.1 or greater is required to develop for iOS.

/cc @jmagman

@jmagman

The other listed failures are happening because it looks like the version of XCode is not right:
[   +9 ms] Found "Xcode 12.0, Build version 12A7209". Xcode 12.0.1 or greater is required to develop for iOS.
/cc @jmagman

I believe that was a transient issue related to the .ci.yaml work and has already been fixed (example https://ci.chromium.org/p/flutter/builders/prod/Mac%20build_tests_3_4/1778).
#87814

It seems like there are three different issues:

Xcode taking more thant 14 mins to install which was caused by the .ci.yaml migration and already fixed.
Gradle daemon not being stopped at the end of the script causing the step to hang until all the child threads are complete.
xcrun xcodebuild -project ... not finishing the subprocesses correctly.

[   +3 ms] Exit code 1 from: sysctl hw.optional.arm64
[        ] sysctl: unknown oid 'hw.optional.arm64'
[        ] executing: xcrun xcodebuild -version
[  +98 ms] Exit code 0 from: xcrun xcodebuild -version
[        ] Xcode 12.3
           Build version 12C33
[  +35 ms] 💪 Building with sound null safety 💪
[  +17 ms] Running Gradle task 'assembleDebug'...
[  +18 ms] Using gradle from /opt/s/w/ir/x/w/flutter/dev/integration_tests/flutter_gallery/android/gradlew.
[  +30 ms] executing: [/opt/s/w/ir/x/w/flutter/dev/integration_tests/flutter_gallery/android/] /opt/s/w/ir/x/w/flutter/dev/integration_tests/flutter_gallery/android/gradlew -Pverbose=true -Ptarget-platform=android-arm,android-arm64,android-x64 -Ptarget=lib/main.dart -Pdart-obfuscation=false -Ptrack-widget-creation=true -Ptree-shake-icons=false assembleDebug
[ +688 ms] To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/6.7/userguide/gradle_daemon.html.
[ +815 ms] Daemon will be stopped at the end of the build stopping after processing

ound plugin integration_test at /opt/s/w/ir/x/w/flutter/packages/integration_test/ [ +82 ms] Found plugin connectivity at /opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/connectivity-3.0.6/ [ +1 ms] Found plugin connectivity_for_web at /opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/connectivity_for_web-0.4.0/ [ ] Found plugin connectivity_macos at /opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/connectivity_macos-0.2.0/ [ +3 ms] Found plugin device_info at /opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/device_info-2.0.2/ [ +21 ms] Found plugin url_launcher at /opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/url_launcher-6.0.9/ [ ] Found plugin url_launcher_linux at /opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/url_launcher_linux-2.0.1/ [ ] Found plugin url_launcher_macos at /opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/url_launcher_macos-2.0.1/ [ +1 ms] Found plugin url_launcher_web at /opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/url_launcher_web-2.0.2/ [ +1 ms] Found plugin url_launcher_windows at /opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/url_launcher_windows-2.0.1/ [ +1 ms] Found plugin video_player at /opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/video_player-2.1.1/ [ +1 ms] Found plugin video_player_web at /opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/video_player_web-2.0.2/ [ +12 ms] Found plugin integration_test at /opt/s/w/ir/x/w/flutter/packages/integration_test/ [ +7 ms] Generating /opt/s/w/ir/x/w/flutter/dev/integration_tests/flutter_gallery/android/app/src/main/java/io/flutter/plugins/GeneratedPluginRegistrant.java [ +42 ms] executing: sysctl hw.optional.arm64 [ +3 ms] Exit code 1 from: sysctl hw.optional.arm64 [ ] sysctl: unknown oid 'hw.optional.arm64' [ ] executing: xcrun xcodebuild -version [ +100 ms] Exit code 0 from: xcrun xcodebuild -version [ ] Xcode 12.3 Build version 12C33 [ +32 ms] Warning: Building for device with codesigning disabled. You will have to manually codesign before deploying to device. [ +12 ms] executing: [/opt/s/w/ir/x/w/flutter/dev/integration_tests/flutter_gallery/ios/] xcrun xcodebuild -list [+1023 ms] Command line invocation: /opt/s/w/ir/cache/osx_sdk/XCode.app/Contents/Developer/usr/bin/xcodebuild -list

       Information about project "Runner":
           Targets:
               Runner
               GalleryUITests

           Build Configurations:
               Debug
               Release
               Profile

           If no build configuration is specified and -scheme is not passed then "Release" is used.

           Schemes:
               Runner

https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20build_tests_3_4/1834/overview
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20build_tests_3_4/1773/overview

xcrun xcodebuild -project ... not finishing the subprocesses correctly.

I'm not sure it's stuck, #1773 timed out in 30 minutes, and that process was the last one running, but that command was kicked off around the 30 minute mark.

▌16:13:17▐ STARTING ANALYSIS
...
▌16:43:12▐ RUNNING: cd dev/integration_tests/flutter_gallery; ../../../bin/flutter build ios --no-codesign --release -v
...
[        ] executing: [/opt/s/w/ir/x/w/flutter/dev/integration_tests/flutter_gallery/ios/Runner.xcodeproj/] xcrun xcodebuild -project /opt/s/w/ir/x/w/flutter/dev/integration_tests/flutter_gallery/ios/Runner.xcodeproj -scheme Runner -configuration Release -showBuildSettings BUILD_DIR=/opt/s/w/ir/x/w/flutter/dev/integration_tests/flutter_gallery/build/ios

Now that you mention it, this is a hostonly test and it should have a timeout longer than 30 mins. I'm taking another look.

We'll need to open a bug to the chromium labs team to get help triaging this issue.

Current flaky ratio for the past 15 days is 1.16%.
One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20build_tests_3_4/2011
Commit: 965bd1f
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20build_tests_3_4/2011
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20build_tests_3_4/1939

Current flaky ratio for the past 15 days is 1.20%.
One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20build_tests_3_4/2011
Commit: 965bd1f
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20build_tests_3_4/2011
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20build_tests_3_4/1939

Chromium bug:

https://bugs.chromium.org/p/chromium/issues/detail?id=1247735

Current flaky ratio for the past 15 days is 0.79%.
One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20build_tests_3_4/2011
Commit: 965bd1f
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20build_tests_3_4/2011

Fix landed on the mac machines to prevent aggressive logging and now that builds are below 2% flakiness we are closing this bug.

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

Navigation Menu

Repository navigation

Some macOS tests are flaky and timing out. #88286

1 remaining item

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Some macOS tests are flaky and timing out. #88286

Description

Activity

zanderso commented on Aug 16, 2021

jmagman commented on Aug 16, 2021

godofredoc commented on Aug 16, 2021

jmagman commented on Aug 16, 2021

godofredoc commented on Aug 16, 2021

1 remaining item

godofredoc commented on Aug 19, 2021

fluttergithubbot commented on Aug 30, 2021

fluttergithubbot commented on Sep 6, 2021

godofredoc commented on Sep 8, 2021

fluttergithubbot commented on Sep 13, 2021

godofredoc commented on Sep 13, 2021

github-actions commented on Sep 27, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions