Repository navigation
Mac web_tool_tests failing on some machines with malformed cached chrome cipd package #99372
Description
As part of upgrading the Chrome package in #98278, a Chromium cipd package G-lZgp8TuOPH0ndPr62HLAzRTWcFV3RRvFjOYyPZq_4C was uploaded that was malformed somehow that flutter couldn't detect the browser as a valid device. A new one was uploaded today that tests passed with.
The engine roll #99362 failed on a Mac web_tool_tests with a cache hit with the malformed package:
https://ci.chromium.org/ui/p/flutter/builders/try/Mac%20web_tool_tests/12828/overview
[P19159 15:21:32.857 action_plan.go:160 I] In root:
[P19159 15:21:32.857 action_plan.go:172 I] to update:
[P19159 15:21:32.857 action_plan.go:174 I] flutter_internal/browsers/chrome/mac-amd64 (Qe3INUxKmaLOB1bDUu7ZD8otihLlr3k4mFVuFq8R97kC -> G-lZgp8TuOPH0ndPr62HLAzRTWcFV3RRvFjOYyPZq_4C)
[P19159 15:21:32.857 instancecache.go:187 I] Using instance cache at /opt/s/cipd_cache/cache/instances
[P19159 15:21:32.858 instancecache.go:409 I] [fetch] Cache hit for flutter_internal/browsers/chrome/mac-amd64:G-lZgp8TuOPH0ndPr62HLAzRTWcFV3RRvFjOYyPZq_4C
[P19159 15:21:32.860 deployer.go:294 I] [unzip] Deploying flutter_internal/browsers/chrome/mac-amd64:G-lZgp8TuOPH0ndPr62HLAzRTWcFV3RRvFjOYyPZq_4C into /opt/s/w/ir/cache/chrome/chrome(/)
Does build843-m9 need its chrome cache cleared? Do any other builders have the wrong one cached?
Here's one that's passing that used BlBCql9ZngKYkn_tiqUUCnZKtZtbbCnsP0RgmLIn2ZwC https://ci.chromium.org/p/flutter/builders/prod/Mac%20web_tool_tests/5090?
Activity
godofredoc commentedon Mar 2, 2022
This is strange, cipd talks to the server to resolve the hash for the tag and then downloads it or reuse the one from the cache with the correct hash.
godofredoc commentedon Mar 2, 2022
Could it be that it was scheduled before the new cipd package was available?
15:21:07 Tue, Mar 01 2022 PSTjmagman commentedon Mar 2, 2022
Wonder how it got version 98 then since you merged that after the cipd package was fixed.
Feel free to close if this is no longer an issue.
godofredoc commentedon Mar 2, 2022
yeah, just checked and I landed the cipd package update at 1:00:00 PM PST
jmagman commentedon Mar 2, 2022
More recent failures:
https://ci.chromium.org/p/flutter/builders/try/Mac%20web_tool_tests/12857 https://ci.chromium.org/p/flutter/builders/try/Mac%20web_tool_tests/12852 https://ci.chromium.org/p/flutter/builders/try/Mac%20web_tool_tests/12850
https://ci.chromium.org/p/flutter/builders/try/Mac%20web_tool_tests/12849
https://ci.chromium.org/p/flutter/builders/try/Mac%20web_tool_tests
zanderso commentedon Mar 2, 2022
/sub
godofredoc commentedon Mar 2, 2022
@CaseyHillers is this bug on your radar?
CaseyHillers commentedon Mar 2, 2022
This sounds like a common issue occurring in the mac bots. I'm going to do a cleanup of impacted bots, but do tell me if it's a larger impact.
Related issue: #99429
CaseyHillers commentedon Mar 3, 2022
As FYI, my corp machines lacks permissions to access our mac minis. I'm debugging the cert issues, but no progress.
If there's no updates, I'll reach out to the chrome labs team to get help tomorrow morning.
godofredoc commentedon Mar 3, 2022
Cleaned the caches of all the machines with failures.
CaseyHillers commentedon Mar 3, 2022
Based on https://ci.chromium.org/p/flutter/builders/try/Mac%20web_tool_tests - This is having a large impact on our fleet with an 80% failure rate
godofredoc commentedon Mar 3, 2022
Let's add another tag
version:98.1to the cipd package and update the references. Turns out that deleting tags is a bad idea as they are locally cached by cipd.CaseyHillers commentedon Mar 4, 2022
Closing as this seems fixed with the revving chrome.
PRs in https://ci.chromium.org/p/flutter/builders/try/Mac%20web_tool_tests are passing if they're rebased with the latest changes. Please re-open if this is still occurring on new PRs
CaseyHillers commentedon Mar 7, 2022
We should manually wipe the cipd caches of these bots
godofredoc commentedon Mar 18, 2022
Turns out this was caused by web_engine and adhoc tests using an incorrect version of the xcode module. It was fixed here: https://flutter-review.googlesource.com/c/recipes/+/28064
github-actions commentedon Apr 1, 2022
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.