Various Linux linux_web_engine Mac tests time out on macOS 14

Example:

clip_op_golden_test.dart: Clips using difference [E]                                                                                                                                      
   TimeoutException after 0:00:30.000000: Test timed out after 30 seconds. See https://pub.dev/packages/test#timeouts
   home/chrome-bot/.pub-cache/hosted/pub.dev/test_api-0.6.1/lib/src/backend/invoker.dart 338:9   call
   org-dartlang-sdk:///lib/async/zone.dart 1391:39                                               _rootRun
   org-dartlang-sdk:///lib/async/zone.dart 1300:34                                               run
   home/chrome-bot/.pub-cache/hosted/pub.dev/test_api-0.6.1/lib/src/backend/invoker.dart 336:5   _handleError
   home/chrome-bot/.pub-cache/hosted/pub.dev/test_api-0.6.1/lib/src/backend/invoker.dart 331:8   _handleError
   org-dartlang-sdk:///lib/async/zone.dart 1399:12                                               _rootRun
   org-dartlang-sdk:///lib/async/zone.dart 1300:34                                               run
   home/chrome-bot/.pub-cache/hosted/pub.dev/test_api-0.6.1/lib/src/backend/invoker.dart 290:38  call
   org-dartlang-sdk:///lib/_internal/js_runtime/lib/async_patch.dart 121:9                       call

https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20linux_web_engine/8641/overview
https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20linux_web_engine/23872/overview
https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20linux_web_engine/23874/overview
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20linux_web_engine/8642/overview

Appears to be flakey:
For example, here's safari-dart2js-canvaskit-ui passing on Mac-14: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20Production%20Engine%20Drone/363852/overview
And here's it failing on Mac-14: https://ci.chromium.org/ui/p/flutter/builders/try/Mac%20Engine%20Drone/937707/overview

Temporarily run Linux linux_web_engine Mac tests on Mac-13 only (#53318)

@jmagman

@jmagman Once fixed, flutter-team-archive/engine#53318 will need to be reverted (or made part of the PR that fixes)

Hack to prevent Safari from being backgrounded during unit tests. (#5…

This is still happening.

20% of the last 50 runs of https://ci.chromium.org/ui/p/flutter/builders/luci.flutter.prod/Linux%20linux_web_engine had timeouts in the safari-dart2js test suites.

(example: https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8744844604307291681/+/u/test:_run_suite_safari-dart2js-canvaskit-ui/stdout)

Many of the logs show multiple TimeoutException after 0:00:30.000000: Test timed out after 30 seconds logs for different test cases.

The log then ends with several minutes of repeated gradient_golden_test.dart: Gradients Using a radial gradient on a paint messages from a test that seems to be stuck.

@yjbanov

@yjbanov mentioned that the next attempt would be to see if this test flakily times out on Safari 18 / macOS 15.
You can follow instructions similar to #148860 (chromium bot only, ignore the devicelab/devicelab cert parts).
And then to run just the Safari test it will look similar to #148877.

More details in go/ios-flutter-ci under "How to Upgrade"

If that works and is the only thing that works, we could consider either keeping a few macOS 15 bots to run these tests, which would bifurcate the release pool. Or, we could start #148870 over again and leapfrog macOS 14 straight to macOS 15, if it's more stable for all tests.

cc @eyebrowsoffire

Also, the chromium bots seem to be on macOS 14.7 now, which was released the same day as Safari 18. Maybe it's the default (I can't seem to find this info)? If so it might be worth seeing if it's still flaking on these newer OS bots.

According to https://chromium-swarm.appspot.com/bot?id=mac-696-h526 and https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20Production%20Engine%20Drone/538825/infra we're still running on macOS 13.

Reenable linux_web_engine mac tests on Mac-14 (#162409)

@yjbanov

Looks like this is still occurring. However, the bots that it fails on still are using Safari 17.6.

https://ci.chromium.org/ui/p/flutter/builders/try/Mac%20Engine%20Drone/1540129/overview
https://ci.chromium.org/ui/p/flutter/builders/try/Mac%20Engine%20Drone/1538130/overview

@yjbanov What are your thoughts on next steps?

I can help you VNC to a bot and run tests on it if you think we may be able to debug it that way. We could also upgrade a few bots to macOS 15, but that would bifurcate the pools, so not preferred.

@yjbanov

@yjbanov I also tested this on macOS 15 (Safari 18.3), seems to also have issues there, although different:
https://ci.chromium.org/ui/p/flutter/builders/staging.shadow/Mac%20Engine%20Drone/5/overview

It seems to get stuck on:

window_test.dart: onSemanticsActionEvent delays action until after frame

@yjbanov

@yjbanov and I have been chatting offline. #162919 was added to try and fix/debug the issue more. However, we're now getting this error:

Starting safaridriver on port 54109
 safaridriver failed to start.
 Failed to start safaridriver:
 Error: SessionNotCreatedException (500): Could not create a session: The session timed out while connecting to a Safari instance.
 #0      parseW3cResponse (package:webdriver/src/handler/w3c/utils.dart:81:9)
 #1      W3cSessionHandler.parseCreateResponse (package:webdriver/src/handler/w3c/session.dart:19:21)
 #2      InferSessionHandler.parseCreateResponse (package:webdriver/src/handler/infer_handler.dart:106:34)
 #3      AsyncRequestClient.send (package:webdriver/src/common/request_client.dart:96:32)
 <asynchronous suspension>
 #4      createDriver (package:webdriver/async_core.dart:62:19)
 <asynchronous suspension>
 #5      SafariMacOsEnvironment._startDriverProcess (file:///Volumes/Work/s/w/ir/cache/builder/engine/src/flutter/lib/web_ui/dev/safari_macos.dart:93:19)
 <asynchronous suspension>
 #6      SafariMacOsEnvironment.prepare (file:///Volumes/Work/s/w/ir/cache/builder/engine/src/flutter/lib/web_ui/dev/safari_macos.dart:44:9)
 <asynchronous suspension>
 #7      RunSuiteStep.run (file:///Volumes/Work/s/w/ir/cache/builder/engine/src/flutter/lib/web_ui/dev/steps/run_suite_step.dart:69:5)
 <asynchronous suspension>
 #8      Pipeline.run (file:///Volumes/Work/s/w/ir/cache/builder/engine/src/flutter/lib/web_ui/dev/pipeline.dart:136:9)
 <asynchronous suspension>
 #9      TestCommand.run (file:///Volumes/Work/s/w/ir/cache/builder/engine/src/flutter/lib/web_ui/dev/test_runner.dart:405:7)
 <asynchronous suspension>
 #10     CommandRunner.runCommand (package:args/command_runner.dart:212:13)
 <asynchronous suspension>
 #11     main (file:///Volumes/Work/s/w/ir/cache/builder/engine/src/flutter/lib/web_ui/dev/felt.dart:41:26)
 <asynchronous suspension>

@yjbanov next thoughts on the issue:

Perhaps the bug in safaridriver is that it opens the port before it's ready to create Safari sessions. So we could either wait a little, or keep trying to open a session a few times.
(we currently do not retry this part; we just give up, kill all processes and retry from scratch, but each time we rush to create a session without waiting)

Reenable linux_web_engine mac tests on Mac-14 (flutter#162409)

Another patch to try and make it more stable: #163791

[CP-stable][web] robustify safaridriver launch sequence (#164191)

It looks like this is fixed now.

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

Hack to prevent Safari from being backgrounded during unit tests. (fl…

Navigation Menu

Repository navigation

Various Linux linux_web_engine Mac tests time out on macOS 14 #150023

16 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Various Linux linux_web_engine Mac tests time out on macOS 14 #150023

Description

Activity

vashworth commented on Jun 13, 2024

jason-simmons commented on Jun 20, 2024

16 remaining items

jmagman commented on Nov 14, 2024

jmagman commented on Nov 15, 2024

yjbanov commented on Dec 5, 2024

vashworth commented on Feb 4, 2025

vashworth commented on Feb 7, 2025

vashworth commented on Feb 19, 2025

vashworth commented on Feb 26, 2025

yjbanov commented on Apr 3, 2025

github-actions commented on Apr 17, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions