Repository navigation
Xcode compilation is slow on macOS 14 bots #150642
Description
Activity
- changed the title
[-]Lots of timeouts on various mac bots[/-][+]Xcode compilation is slow on macOS 14 bots in try pool[/+]on Jun 21, 2024 Timeout seems to only be happening in the try pool
https://ci.chromium.org/ui/p/flutter/builders/try/Mac%20build_ios_framework_module_test?limit=50
vs
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20build_ios_framework_module_test?limit=50Timeout seems to only be happening in the try pool https://ci.chromium.org/ui/p/flutter/builders/try/Mac%20build_ios_framework_module_test?limit=50 vs https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20build_ios_framework_module_test?limit=50
Saw this one in prod/post-submit: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20tool_integration_tests_1_4/17133/infra
Possibly related to trying out the Xcode beta in #150405
The slow bots show the Xcode 16 beta xcode_16a5171c in the Xcode caches:
drwxr-xr-x 6 chrome-bot staff 192 Jun 18 10:32 xcode_15a240d drwxr-xr-x 6 chrome-bot staff 192 Jun 18 09:43 xcode_16a5171c drwx------ 5 chrome-bot staff 160 Jun 18 10:36 xcode_runtime_dmg_ios-17-0_15a240dAnd the passing ones don't:
drwxr-xr-x 6 chrome-bot staff 192 Jun 14 10:02 xcode_15a240dThe test logs show they are correctly and exclusively using the 15a240d version of Xcode, so I'm not sure why having the other one installed would matter.
@christopherfujino any idea how to clear the Xcode caches to ensure only 15 is present? go/flutter-infra-xcode#cache-pollution doesn't have explicit instructions. I see go/flutter-luci-playbook#delete-corrupted-mac-caches.
Saw this one in prod/post-submit: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20tool_integration_tests_1_4/17133/infra
Hmm, well that doesn't bode well for my theory...
Maybe we should split these out by builder/test target, as I think Jenn and I are investigating different timeouts
- changed the title
[-]Xcode compilation is slow on macOS 14 bots in try pool[/-][+]Xcode compilation is slow on macOS 14 bots[/+]on Jun 21, 2024 go/flutter-luci-playbook#delete-corrupted-mac-caches makes it sound like you need to go in one by one and rm rm -rf via ssh on each bot :'(
go/flutter-luci-playbook#delete-corrupted-mac-caches makes it sound like you need to go in one by one and rm rm -rf via ssh on each bot :'(
@bdero do you know if--with SALT--you can run a remote command on all the minions at once?
The last 5 timeouts of https://ci.chromium.org/ui/p/flutter/builders/try/Mac%20tool_integration_tests_1_4?limit=100 in the try pool had xcode_16a5171c in the cache.
I wish I knew why this is happening, ideally these installations wouldn't impact each other.
The last 5 timeouts of https://ci.chromium.org/ui/p/flutter/builders/try/Mac%20tool_integration_tests_1_4?limit=100 in the try pool had xcode_16a5171c in the cache.
I wish I knew why this is happening, ideally these installations wouldn't impact each other.
There are two hard problems in computer science: naming things, and caching CI artifacts.
@christopherfujino the xcode_16a5171c bots be removed from the try pool in the meantime?
@christopherfujino there's code in the recipe to remove the cache:
https://cs.opensource.google/flutter/recipes/+/master:recipe_modules/osx_sdk/api.py;l=244-247Would it be easier to change the recipe to run that on test run timeout until all the bots have it cleared up?
Or hard-code removal of xcode_16a5171c?14 remaining items
- added a commit that references this issue
on Jul 17, 2024 - added a commit that references this issue
on Jul 17, 2024 - added a commit that references this issue
on Aug 7, 2024 - added a commit that references this issue
on Sep 2, 2024 This issue is marked P1 but has had no recent status updates.
The P1 label indicates high-priority issues that are at the top of the work list. This is the highest priority level a bug can have if it isn't affecting a top-tier customer or breaking the build. Bugs marked P1 are generally actively being worked on unless the assignee is dealing with a P0 bug (or another P1 bug). Issues at this level should be resolved in a matter of months and should have monthly updates on GitHub.
Please consider where this bug really falls in our current priorities, and label it or assign it accordingly. This allows people to have a clearer picture of what work is actually planned. Thanks!
- moved this from New to Fixed in [DEPRECATED] Infra Ticket Queue
on Feb 4, 2025 - added a commit that references this issue
on Feb 7, 2025 This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.- locked as resolved and limited conversation to collaborators
on Feb 18, 2025
There seems to be a pretty high frequency of timeouts right now on mac ci bots.
Examples:
From #150585
https://github.com/flutter/flutter/pull/150585/checks?check_run_id=26535171974
https://github.com/flutter/flutter/pull/150585/checks?check_run_id=26535172379
From #150614
https://github.com/flutter/flutter/pull/150614/checks?check_run_id=26520636606
https://github.com/flutter/flutter/pull/150614/checks?check_run_id=26520632867
From #150517
https://github.com/flutter/flutter/pull/150517/checks?check_run_id=26528249189
https://github.com/flutter/flutter/pull/150517/checks?check_run_id=26528248334