Repository navigation
Mac presubmit queues are out of SLO #114656
Description
I've had several PRs the past 2 weeks be out of SLO (stuck in queue for 2 hours). I don't send PRs during the peak hours, and I expect this is worse for those contributing during those hours.
Example PR: #114646 (first commit was queued for 2 hours)
Activity
godofredoc commentedon Nov 4, 2022
2h is probably too much but a huge increase in execution|queue time is expected after #113539 where we are explicitly cleaning the local xcode cache
godofredoc commentedon Nov 4, 2022
Setting as P1 seems like mac capacity is at 100% utilization from 6:00AM PST to 5:00PM PST.
Next step we need to identify if this is normal usage or something strange is happening.
godofredoc commentedon Nov 4, 2022
From the data it seems like we've always had peeks of 30M 100% utilization at ~9:00AM PST but we started to have 3+ peeks per day on 10/17/2022. Things got really bad on 10/24/2022 where we started with 100% utilization from 9:00 to 5:00
keyonghan commentedon Nov 4, 2022
The local cache cleanup happened on 10/19 (#113729), the queue time (90th%) was <1min for 10/19-10/24.
The queue time started increasing from 10/25
10/25-10/26: 6.06 min
10/26-10/27: 60.5 min
10/17-10/28: 58.68 min
10/28-11/04: 85.27 min
keyonghan commentedon Nov 4, 2022
One correlated change per the timing is the quota increase (cl/483782210, which was landed 0n 10/25). @godofredoc Are you comfortable with a revert?
godofredoc commentedon Nov 4, 2022
godofredoc commentedon Nov 4, 2022
Ths is
Can you please post daily data for: 10/25, 10/27, 11/03?
keyonghan commentedon Nov 4, 2022
The engine source checkout was not improved with this quota increase CL. It has been taking 5-7 mins before and after. See
https://ci.chromium.org/p/flutter/builders/prod/Mac%20Host%20clang-tidy?cursor=id%3E8799256324311980481&limit=200
https://ci.chromium.org/p/flutter/builders/prod/Mac%20Unopt?cursor=id%3E8799256222795131969&limit=200
https://ci.chromium.org/p/flutter/builders/prod/Mac%20Web%20Engine?cursor=id%3E8799256222304648017&limit=200
https://ci.chromium.org/p/flutter/builders/prod/Mac%20iOS%20clang-tidy?cursor=id%3E8799263756679076865&limit=200
godofredoc commentedon Nov 4, 2022
That is expected, cl/483782210 fixes peek quota whith checkouts taking 40mins in try.
5-7 mins is expected in cold checkouts if we are getting consistent 5-7 mins checkouts then that is a bug that need to be fixed.
keyonghan commentedon Nov 4, 2022
10/25: 6 min
10/27: 58 min
11/03: 100 min
godofredoc commentedon Nov 4, 2022
Are these 90th percentiles? can we also get the 50th?
keyonghan commentedon Nov 4, 2022
Here are the comparison between 90th and 50th, plus the build number:
date. 90th 50th No. of builds
10/24. 19 6 1605
10/25. 6 0 1192
10/26. 60 15 1983
10/27. 58 21 2046
11/03. 100 7 2493
The queue time correlates with the No. of builds
keyonghan commentedon Nov 4, 2022
When did you enable the Mac engine v2 builders?
11 remaining items
Fix boolean json property. (#37403)
Pass process json dicts in ci.yaml. (#2293)
Move gclient var property to ci.yaml (flutter-team-archive#37351)
Fix boolean json property. (flutter-team-archive#37403)
Update engine v2 builds to remove gclient_custom_var support
Move gclient var property to ci.yaml (flutter-team-archive#37351)
Fix boolean json property. (flutter-team-archive#37403)
keyonghan commentedon Nov 23, 2022
Queue time drops to 20mins for past two weeks. Moving to TD for further optimization track.
Remove auto_cleanup of xcode caches. (flutter#114719)
Remove auto_cleanup of xcode caches. (flutter#114719)
github-actions commentedon Mar 17, 2023
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.