Repository navigation
Add Mac M1 ARM machines to the prod pool #87508
Description
Activity
\cc @keyonghan as this is relevant to your design doc for rebasing benchmarks.
@jmagman Mac M1s are ready to get moved to prod, which tests/benchmarks we want to run on them and which names should we use?
E.g. in the past we used something like catalina_ but we have had to rename them and lose historical data. Ideally we should be able to completely detach the test name from the benchmark and be able to select the benchmark based on properties<host_os, phone_os, etc>.
Let me know if this list needs to be pared down.
hot_mode_dev_cycle_macos_target__benchmark
flutter_gallery_ios__compile
hello_world_ios__compile
native_ui_tests_macos
native_ui_tests_ios
module_test_ios
build_ios_framework_module_test
plugin_lint_mac
run_release_test
integration_test_test
hello_world_android__compile
ios_app_with_extensions_test
ios_content_validation_test
macos_chrome_dev_mode
build_aar_module_test@zanderso are there others you can think of? Do you have opinions about the names?
Reacted by eggflyFrom benchmark point view, this should not be blocked.
We can view trending charts based ontask nameby simply filteringarch: m1.This is also our team wanted. Please make this last step into infra to support macOS M1 platform on native ARM machine code, looking forward to it. 💪
Reacted by Kyle Seongwoo Jun, Emir Causevic, J4ckTh3R1qp3r, Yiju Guo and mlhwNow tests are running in staging pool, but most tests hit pod install error. @jmagman any insight on how to workaround the issue?
[hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stdout: Don't forget to include the Crash Report log file under [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stdout: DiagnosticReports directory in bug reports. [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stdout: [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stdout: [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: [+3181 ms] Exception: Error running pod install [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stdout: [ ] "flutter run" took 5,518ms. [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: [ +1 ms] [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: #0 throwToolExit (package:flutter_tools/src/base/common.dart:10:3) [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: #1 RunCommand.runCommand (package:flutter_tools/src/commands/run.dart:688:9) [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: <asynchronous suspension> [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: #2 FlutterCommand.run.<anonymous closure> (package:flutter_tools/src/runner/flutter_command.dart:1165:27) [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: <asynchronous suspension> [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: #3 AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19) [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: <asynchronous suspension> [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: #4 CommandRunner.runCommand (package:args/command_runner.dart:209:13) [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: <asynchronous suspension> [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: #5 FlutterCommandRunner.runCommand.<anonymous closure> (package:flutter_tools/src/runner/flutter_command_runner.dart:281:9) [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: <asynchronous suspension> [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: #6 AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19) [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: <asynchronous suspension> [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: #7 FlutterCommandRunner.runCommand (package:flutter_tools/src/runner/flutter_command_runner.dart:229:5) [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: <asynchronous suspension> [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: #8 run.<anonymous closure>.<anonymous closure> (package:flutter_tools/runner.dart:62:9) [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: <asynchronous suspension> [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: #9 AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19) [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: <asynchronous suspension> [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: #10 main (package:flutter_tools/executable.dart:94:3) [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr: <asynchronous suspension> [hot_mode_dev_cycle_macos_target__benchmark] [STDOUT] stderr:It's hitting something like #70796. On those machines, before the gem bundle install, we'll have to first run:
arch -x86_64 sudo gem install ffiHmm, the tool should be printing that, but I don't see it #94385.
See also https://stackoverflow.com/a/65334677.
Tried installing
ffibefore the gem bundle install, it still failed: https://luci-milo.appspot.com/raw/build/logs.chromium.org/flutter/led/keyonghan_google.com/196359aeabbf1620525eac7f6dc32cd7b142767ab762aeb7546e1504bb0aad42/+/build.protoThen tried uninstall coacopods and reinstall it following https://stackoverflow.com/questions/66644365/cocoapods-on-m1-apple-silicon-fails-with-ffi-wrong-architecture, it failed again: https://luci-milo.appspot.com/raw/build/logs.chromium.org/flutter/led/keyonghan_google.com/43613c2bda68ffbcd0e4679d0a985bd912dd897013a663541a1e389b811da1f7/+/build.proto
BTW: replaced
-x86_64with-arm64for our case.BTW: replaced
-x86_64with-arm64for our case.I think that's the problem, confusingly it needs to be
-x86_64in our case.If this is needed for arm machines I'd suggest that we add it to the salt state packages for M1.
BTW: replaced
-x86_64with-arm64for our case.I think that's the problem, confusingly it needs to be
-x86_64in our case.Switched to
-arm64when hitting below error with-x86_64(a new build example):arch: posix_spawnp: gem: Bad CPU type in executableBTW: from the salt side (how we provision devicelab hosts), we have been using
-arm64.23 remaining items
It turns out that we have been using intel built python (#98057) to run the swarming server, which schedules/runs our builds.
At this moment, I have manually brought up mac41 and mac42 in staging to run and pass these tests. But we need to enable these changes in our
saltso that it can provision all bots with same python+pyobjc. As of now, these two bots may get back to old non-working python version if a salt refresh happens.Working on a CL to push the salt change.
Could we move the passing ones over to prod?
I prefer to hold on this before we have a stable working python on the bots.
@keyonghan can we also add
Mac tool_integration_tests_*andMac tool_tests_commandstoMac_arm64?
/cc @christopherfujino@keyonghan can we also add
Mac tool_integration_tests_*andMac tool_tests_commandstoMac_arm64?@christopherfujino do you agree with adding these shards? You expressed concerns that only a subset of these probably need to run on arm64, and you didn't want the extra noise of double the flaky reports.
@keyonghan can we also add
Mac tool_integration_tests_*andMac tool_tests_commandstoMac_arm64?@christopherfujino do you agree with adding these shards? You expressed concerns that only a subset of these probably need to run on arm64, and you didn't want the extra noise of double the flaky reports.
started an internal chat thread to discuss
There are still 6 Mac/ios tests failing consistently:
-
Mac_arm64_staging native_ui_tests_macos: Platform unit tests failed (Mac arm64 testnative_ui_tests_macosfails withPlatform unit tests failed#101721) -
Mac_arm64_staging module_test_ios: Platform unit tests failed (Mac_arm64_staging module_test_iosfails withPlatform unit tests failed#101722) - No android SDK found for
Mac_arm64_staging plugin_lint_mac,Mac_arm64_staging run_release_test,Mac_arm64_staging integration_test_test, andMac_arm64_staging hello_world_android__compile.
/cc @jmagman could you help take a look at the first two?
-
https://chrome-internal-review.googlesource.com/c/infradata/config/+/4673717 to add half M1/ios test beds to prod, with which we can migrate those passing tests. /cc @jmagman (somehow I cannot add you to the CL as a reviewer. Maybe because you have never touched that repo before)
Reacted by Jenn Magder and Chase Kanipe@keyonghan some are failing with the Ruby ffi error again:
@keyonghan what happened with your attempts to install x86 version of the
ffigem? That should resolve theffierror.
#87508 (comment)@keyonghan what happened with your attempts to install x86 version of the ffi gem? That should resolve the ffi error.
The workaround to use
ffiandrosettadoes not help. Using the customizedruby, here are the latest run results (three tests are still failing):- Mac_arm64_staging module_test_ios fails with Platform unit tests failed,
Mac_arm64_staging module_test_iosfails withPlatform unit tests failed#101722 - Mac arm64 test native_ui_tests_macos fails with Platform unit tests failed, Mac arm64 test
native_ui_tests_macosfails withPlatform unit tests failed#101721 - Mac_arm64_staging plugin_lint_mac failed with xcodebuild failure, Mac_arm64_staging plugin_lint_mac failed with
Undefined symbols for architecture arm64#100638
- Mac_arm64_staging module_test_ios fails with Platform unit tests failed,
All requested tests have been available and running in prod. To sum up:
Enabled arm64 android tests:
- hello_world_android__compile
- integration_test_test
- run_release_test
Enabled arm64 ios tests:
- build_ios_framework_module_test
- module_test_ios
- plugin_lint_mac
- flutter_gallery_ios__compile
- hello_world_ios__compile
- hot_mode_dev_cycle_macos_target__benchmark
- ios_app_with_extensions_test
- ios_content_validation_test
- macos_chrome_dev_mode
- run_release_test_macos
Marking this as fixed.
Reacted by mono — Masayuki OnoThis thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.- locked as resolved and limited conversation to collaborators
on Jun 30, 2022
flutter-devicelab-mac-4 and flutter-devicelab-mac-5 are now in the staging pool as of #79430. Promote to the prod pool once all existing tests pass on those machines.
https://chromium-swarm.appspot.com/bot?id=flutter-devicelab-mac-4
https://chromium-swarm.appspot.com/bot?id=flutter-devicelab-mac-5
Blocked by #85654.