Repository navigation
Mac module_test_ios is 2.76% flaky #89175
Description
The post-submit test builder Mac module_test_ios had a flaky ratio 2.76% for the past 15 days, which is above our 2.00% threshold.
One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20module_test_ios/3945
Commit: c5861cf
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20module_test_ios/3945
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20module_test_ios/3930
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20module_test_ios/3913
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20module_test_ios/3903
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20module_test_ios/3898
Succeeded builds (3 most recent):
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20module_test_ios/4007
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20module_test_ios/4006
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20module_test_ios/4005
Please follow https://github.com/flutter/flutter/wiki/Reducing-Test-Flakiness#fixing-flaky-tests to fix the flakiness and enable the test back after validating the fix (internal dashboard to validate: go/flutter_test_flakiness).
Activity
cyanglaz commentedon Sep 1, 2021
Looks like the flakeness is due to lots of timeouts:
[module_test_ios] [STDOUT] stdout: /opt/s/w/ir/x/t/flutter_module_test.er8AG5/hello_host_app/FlutterUITests/FlutterUITests.m:39: error: -[FlutterUITests testFullScreenWarm] : (([app.staticTexts[@"Button tapped 0 times."] waitForExistenceWithTimeout:60.0]) is true) failed[FlutterUITests testFlutterViewWarm] : (([app.staticTexts[@"Button tapped 0 times."] waitForExistenceWithTimeout:60.0]) is true) failedMaybe we should increase the timeout and see what's going on.
zanderso commentedon Sep 6, 2021
Looks like a duplicate of #88286
jmagman commentedon Sep 7, 2021
II don't think this is a duplicate, the test is actually failing, not timing out.
It's failing because the Flutter view isn't loading and the static text really isn't there:
See test results from run: https://storage.googleapis.com/flutter_logs/flutter/c5861cfb56725f492e4a58d3dfdfca868c64dade/module_test_ios/f5624720-7397-42a0-a384-5e85faaf0355/module_test_ios-objc-2021-08-25T10:23:39.878872.zip
fluttergithubbot commentedon Sep 13, 2021
Current flaky ratio for the past 15 days is 2.38%.
One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20module_test_ios/4132
Commit: 53e8511
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20module_test_ios/4132
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20module_test_ios/4099
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20module_test_ios/4094
cyanglaz commentedon Sep 13, 2021
zanderso commentedon Sep 14, 2021
@cyanglaz Are you continuing to investigate this, or are you blocked? Should we check the logs for dropped messages?
31 remaining items
jmagman commentedon Feb 9, 2022
crash.txt
fluttergithubbot commentedon Feb 14, 2022
Current flaky ratio for the past (up to) 100 commits is 1.00%. Flaky number: 1; total number: 100.
One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20module_test_ios/456
Commit: ec58835
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/staging/Mac%20module_test_ios/456
Recent test runs:
https://flutter-dashboard.appspot.com/#/build?taskFilter=Mac%20module_test_ios
zanderso commentedon Feb 14, 2022
Build 456 (2/9) was before the fix flutter-team-archive/engine#31416 from @jason-simmons rolled into the framework #98303 (2/11).
jmagman commentedon Feb 15, 2022
@keyonghan I don't see the reported https://ci.chromium.org/ui/p/flutter/builders/staging/Mac%20module_test_ios/456 flake in the list. That one was on ec58835 on 2/9, but the last flake I see in https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Mac%20module_test_ios%22 under
Flakiness status for recent commitsis d78377f from 2/7. Am I reading it wrong?keyonghan commentedon Feb 15, 2022
The first two charts are for prod runs. As this test has been marked as
bringup: true, it is running in the staging pool. Please take a look at the middle two charts. (by default it shows recent 50 commits. you could see the flake run if you increase the number to say 60).jmagman commentedon Feb 15, 2022
That makes sense, thanks for the explainer!
jmagman commentedon Feb 23, 2022
Not seeing any recent staging flakes in the table, the bot didn't comment with new stats. @keyonghan should I wait for the bot to put out a PR to mark unflaky, or just do it myself?
keyonghan commentedon Feb 23, 2022
The flake bot was shifted to Wed. (flutter/cocoon#1603), and it should be deflaked tomorrow if you can wait:-)
fluttergithubbot commentedon Feb 23, 2022
Current flaky ratio for the past (up to) 100 commits is 0.00%. Flaky number: 0; total number: 100.
keyonghan commentedon Feb 23, 2022
Marking the test as fixed.
The bot didn't deflake the test automatically due to this issue being still open. #99009 to manually deflake.
github-actions commentedon Mar 9, 2022
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.