Repository navigation
flutter-devicelab-mac bots have been failing since 12/16 #161402
Description
They are failing in "Prepare iOS device|Dismiss iOS dialogs|Run app to dismiss dialogs"
reproduction
@LouiseHsu is using led to test device lab PR's on CI.
example affected bots
- https://chromium-swarm.appspot.com/bot?id=flutter-devicelab-mac-1
- https://chromium-swarm.appspot.com/bot?id=flutter-devicelab-mac-14
example failures
- https://ci.chromium.org/ui/p/flutter/builders/try/Mac_x64_ios%20hot_mode_dev_cycle_ios__benchmark/47/overview
- https://luci-milo.appspot.com/ui/p/flutter/builders/try/Mac_x64_ios%20hot_mode_dev_cycle_ios__benchmark/46/overview
error logs
Sometimes it says:
GatherProvisioningInputs
CreateBuildDescription
Build description signature: a1e66c9dc7529a0d8971349d7ac5ac98
Build description path: /Users/swarming/Library/Developer/Xcode/DerivedData/infra-dialog-hbrdbwazhkutrwccpcbhibiotqak/Build/Intermediates.noindex/XCBuildData/a1e66c9dc7529a0d8971349d7ac5ac98.xcbuilddata
/opt/s/w/ir/cache/cocoon/cipd_packages/device_doctor/tool/infra-dialog/infra-dialog.xcodeproj: error: Provisioning profile "match Development *" has platforms "watchOS and iOS", which does not match the current platform "macOS". (in target 'infra-dialogUITests' from project 'infra-dialog')
/opt/s/w/ir/cache/cocoon/cipd_packages/device_doctor/tool/infra-dialog/infra-dialog.xcodeproj: error: Provisioning profile "match Development *" doesn't include the com.apple.application-identifier,
com.apple.security.app-sandbox,
com.apple.security.get-task-allow,
com.apple.security.network.client,
com.apple.security.temporary-exception.files.absolute-path.read-only,
com.apple.security.temporary-exception.mach-lookup.global-name,
com.apple.security.temporary-exception.mach-lookup.local-name,
and com.apple.security.temporary-exception.sbpl entitlements. Profile qualification is using entitlement definitions that may be out of date. Connect to network to update. (in target 'infra-dialogUITests' from project 'infra-dialog')
/opt/s/w/ir/cache/cocoon/cipd_packages/device_doctor/tool/infra-dialog/infra-dialog.xcodeproj: error: Provisioning profile "match Development *" doesn't include the currently selected device "flutter-devicelab-mac-14" (identifier F4B0151D-6DFF-5AC6-A28F-CD6072DC35D4). (in target 'infra-dialogUITests' from project 'infra-dialog')
Test session results, code coverage, and logs:
/Users/swarming/Library/Developer/Xcode/DerivedData/infra-dialog-hbrdbwazhkutrwccpcbhibiotqak/Logs/Test/Test-infra-dialog-2024.12.17_10-01-17--0800.xcresult
Testing failed:
Provisioning profile "match Development *" has platforms "watchOS and iOS", which does not match the current platform "macOS".
Provisioning profile "match Development *" doesn't include the com.apple.application-identifier,
Usually it says:
2024-12-17 09:57:12.131 xcodebuild[1931:14509] Writing error result bundle to /var/folders/87/q7tyxqlx5sv13g_xnv41976h0000gp/T/ResultBundle_2024-17-12_09-57-0012.xcresult
xcodebuild: error: Unable to find a destination matching the provided destination specifier:
{ id:00008030-001C498C022B402E }
Available destinations for the "infra-dialog" scheme:
{ platform:macOS, arch:x86_64, variant:Mac Catalyst, id:F4B0151D-6DFF-5AC6-A28F-CD6072DC35D4 }
{ platform:iOS, id:dvtdevice-DVTiPhonePlaceholder-iphoneos:placeholder, name:Any iOS Device }
{ platform:iOS Simulator, id:dvtdevice-DVTiOSDeviceSimulatorPlaceholder-iphonesimulator:placeholder, name:Any iOS Simulator Device }
{ platform:macOS, variant:Mac Catalyst, name:Any Mac }
{ platform:iOS Simulator, id:92FE5F12-D57C-46FD-A575-73838DE2DD02, OS:17.0, name:iPad (10th generation) }
{ platform:iOS Simulator, id:96556E12-0764-434B-A12F-20FCB8FA28BC, OS:18.0, name:iPad (10th generation) }
{ platform:iOS Simulator, id:E742930F-E2C8-4B85-B41D-8812EC0E92C1, OS:17.0, name:iPad Air (5th generation) }
{ platform:iOS Simulator, id:E51B66FB-D85E-4EF5-B19D-66B4008EEFC0, OS:18.0, name:iPad Air (5th generation) }
{ platform:iOS Simulator, id:F9E91713-2B70-4504-900C-4E5E65766D67, OS:17.0, name:iPad Pro (11-inch) (4th generation) }
{ platform:iOS Simulator, id:17A8CF74-68C6-48E6-844C-A6C52A1A6DB6, OS:18.0, name:iPad Pro (11-inch) (4th generation) }
{ platform:iOS Simulator, id:8C0C729F-4CB8-4BDC-A82D-329C51275B3D, OS:17.0, name:iPad Pro (12.9-inch) (6th generation) }
{ platform:iOS Simulator, id:8A79CEF4-2146-4BB5-8946-93C5223C57C3, OS:18.0, name:iPad Pro (12.9-inch) (6th generation) }
{ platform:iOS Simulator, id:2D9AC777-9142-416D-8000-BCCBA0E9319D, OS:17.0, name:iPad mini (6th generation) }
{ platform:iOS Simulator, id:75D79BB3-2CA0-44EC-9301-EF383590EA9D, OS:18.0, name:iPad mini (6th generation) }
{ platform:iOS Simulator, id:2A24430E-D906-4983-98A4-A0D10E2BFFB5, OS:17.0, name:iPhone 14 }
{ platform:iOS Simulator, id:2572052B-E8A6-4E7E-899A-0015DAE7942F, OS:17.0, name:iPhone 14 Plus }
{ platform:iOS Simulator, id:D5F5DC29-AD5F-4CC1-B4E3-4F60DD392BAC, OS:17.0, name:iPhone 14 Pro }
{ platform:iOS Simulator, id:F081F76D-AD23-40C1-BEB1-ADAF280B3BE8, OS:17.0, name:iPhone 14 Pro Max }
{ platform:iOS Simulator, id:D668C823-10C0-4095-B86F-1F0D9E6E1012, OS:17.0, name:iPhone 15 }
{ platform:iOS Simulator, id:2BF967A0-A6ED-4E41-B703-A2C40CA16680, OS:18.0, name:iPhone 15 }
{ platform:iOS Simulator, id:F245C248-854D-4396-81F0-CB150B780939, OS:17.0, name:iPhone 15 Plus }
{ platform:iOS Simulator, id:C8831D11-4476-4A46-B7ED-75BDE6A801B2, OS:18.0, name:iPhone 15 Plus }
{ platform:iOS Simulator, id:DB9E81C0-FD2E-4806-9CAF-CAC31CC32FD5, OS:17.0, name:iPhone 15 Pro }
{ platform:iOS Simulator, id:EF6DBC83-A1F2-4003-B74E-DBEAD39A868E, OS:18.0, name:iPhone 15 Pro }
{ platform:iOS Simulator, id:971C0588-405C-4F7B-BE6B-28BBDEF6B5A9, OS:17.0, name:iPhone 15 Pro Max }
{ platform:iOS Simulator, id:52D2A073-8E7A-422F-BC71-DA427CEEE134, OS:18.0, name:iPhone 15 Pro Max }
{ platform:iOS Simulator, id:9042CBE5-BE5E-4446-AEDA-3520EB0AB799, OS:17.0, name:iPhone SE (3rd generation) }
{ platform:iOS Simulator, id:C8088DA4-3708-48C7-89DE-2375A26FF165, OS:18.0, name:iPhone SE (3rd generation) }
Ineligible destinations for the "infra-dialog" scheme:
{ platform:iOS, arch:arm64e, id:00008030-001C498C022B402E, name:swarming’s iPhone, error:swarming’s iPhone is not available because it is unpaired Pair with the device in the Xcode Devices Window, and respond to any pairing prompts on the device. }
Activity
zanderso commentedon Jan 9, 2025
@kentnguyen99 Could you check on these in the lab to see if you notice anything wrong with them?
gaaclarke commentedon Jan 9, 2025
I'm guessing they are popping up a dialog like "do you trust this device". It looks like the script that clicks it isn't working.
LouiseHsu commentedon Jan 9, 2025
Ive also gotten this failure https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8726287631050396369/+/u/Prepare_iOS_device/Dismiss_iOS_dialogs/Run_app_to_dismiss_dialogs/stdout
I think this is the failure that isnt clicking the script
kentnguyen99 commentedon Jan 10, 2025
@zanderso,
Both mac-1 and mac-14 are up and running fine, don't see anything wrong on both machines. Do you want wipe them out and re-install fresh? Please let me know.
kentnguyen99 commentedon Jan 10, 2025
@zanderso, Can you submit small job and test on mac-1 and mac-14? I would like to see what errors show in the logs.
Thanks.
LouiseHsu commentedon Jan 10, 2025
@kentnguyen99 here for mac 14 https://ci.chromium.org/b/8726173123142495505
here for mac 1 https://ci.chromium.org/b/8726173077128537361
kentnguyen99 commentedon Jan 10, 2025
@christopherfujino Can you take a look of mac-1 and mac-14? Both machines could not download file from the Github and got errors below. Thanks.
100 256 100 256 0 0 3048 0 --:--:-- --:--:-- --:--:-- 3084
[/opt/s/w/ir/x/w/rc/tmpamfvcl77/flutter sdk/bin/cache/dart-sdk-darwin-x64.zip]
End-of-central-directory signature not found. Either this file is not
a zipfile, or it constitutes one disk of a multi-part archive. In the
latter case the central directory and zipfile comment will be found on
the last disk(s) of this archive.
unzip: cannot find zipfile directory in one of /opt/s/w/ir/x/w/rc/tmpamfvcl77/flutter sdk/bin/cache/dart-sdk-darwin-x64.zip or
/opt/s/w/ir/x/w/rc/tmpamfvcl77/flutter sdk/bin/cache/dart-sdk-darwin-x64.zip.zip, and cannot find /opt/s/w/ir/x/w/rc/tmpamfvcl77/flutter sdk/bin/cache/dart-sdk-darwin-x64.zip.ZIP, period.
It appears that the downloaded file is corrupt; please try again.
If this problem persists, please report the problem at:
https://github.com/flutter/flutter/issues/new?template=1_activation.yml
zanderso commentedon Jan 10, 2025
I think the download failure might be an unrelated flake. Most/all of the other failures I can see on those bots are the ones noted above.
Based on the error messages, I wonder if there's an issue with the provisioning profiles installed on these macs, or whether the app to dismiss dialogs can't run because it is built wrong? I wonder about that because I also see the error:
@kentnguyen99 Instead of a deep investigation of exactly how these machines might be corrupted, is it possible to just re-image them?
kentnguyen99 commentedon Jan 10, 2025
Okay, will re-image both machines and then will go from there.
kentnguyen99 commentedon Jan 11, 2025
@zanderso, Re-image OS for mac-14 has been completed; However, machine could not connecting the phone device due error below. Who should I talk regarding Device doctor tool App? I ran into this issue before and Gofredo fixed last time.
christopherfujino commentedon Jan 13, 2025
There's a SALT state to install idevice_id, but rather than investigate why it failed, i will manually run it on these bots.
12 remaining items
christopherfujino commentedon Jan 17, 2025
Acknowledged. Let me try to VNC in and run a test at the same time...
christopherfujino commentedon Jan 18, 2025
I actually wasn't able to get a VNC client working on linux, so I think I'll have to try this next week in the lab.
christopherfujino commentedon Jan 21, 2025
Ok, I went to the lab, and sat at flutter-devicelab-mac-14 when I triggered this build via LED: https://ci.chromium.org/ui/p/flutter/builders/try.shadow/Mac_x64_ios%20integration_test_test_ios/10/infra
Here's the tail of the logs:
Right after this, the host machine died and rebooted.
kentnguyen99 commentedon Feb 10, 2025
Hi @vashworth
When you have a moment.. Can you take a look both Bots Mac-1 and Mac-14?
Both Bots were not working and failed every times it runs. So, we decided to reimage MacOS on Bots but after reimagined MacOS and problem still there. Can you take a look of both machines and find anything wrong with machines?
vashworth commentedon Feb 11, 2025
Looks like the device on Mac-1 is currently unpaired:
To pair device:
Looks like Mac-14 is having some certificate issues
From my research, this usually indicates you need to set the Trust factor to "System Defaults" instead of "Always Trust". However, when I look at the certs in the keychain, they all look set correctly. Also looks like all the correct certs are there. I'm not sure why this would be failing with this error. @cbracken any ideas?
kentnguyen99 commentedon Feb 12, 2025
@LouiseHsu,
Can you give a run test on both mac-1 and mac-14? Just made some changes on both phone devices. Thanks.
cbracken commentedon Feb 12, 2025
This could happen in cases where our (valid) cert is there, but a cert along the certificate chain back to the root is missing. e.g. an expired Apple cert?
Our cert should be issued by "Apple Worldwide Developer Relations Certification Authority". If you look in the login keychain, do you see a valid certificate for "Apple Worldwide Developer Relations Certification Authority" -- one with an expiry date in the future?
vashworth commentedon Feb 12, 2025
Okay found the answer, the "Apple Worldwide Developer Relations Certificate Authority" needs to be in both the
loginandsystemkeychain. Just theloginkeychain is not enough.vashworth commentedon Feb 13, 2025
Instructions for future reference:
Systemkeychain on the left.AppleWWDRCAG3.cerfrom the Downloads folder to the Keychain Access window.vashworth commentedon Feb 13, 2025
Mac-1 and 14 should be good to go now. Closing.
github-actions commentedon Feb 27, 2025
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.