Repository navigation
Mac_arm64 tool_host_cross_arch_tests is failing with recipe crash #113231
Description
The recipe has crashed at point 'Uncaught exception'!
Traceback (most recent call last):
File "/Volumes/Work/s/w/ir/kitchen-checkout/recipe_engine/recipe_engine/internal/engine.py", line 585, in run_steps
raw_result = recipe_obj.run_steps(api, engine)
File "/Volumes/Work/s/w/ir/kitchen-checkout/recipe_engine/recipe_engine/internal/recipe_deps.py", line 936, in run_steps
recipe_result = invoke_with_properties(
File "/Volumes/Work/s/w/ir/kitchen-checkout/recipe_engine/recipe_engine/internal/property_invoker.py", line 90, in invoke_with_properties
return _invoke_with_properties(callable_obj, all_props, environ, prop_defs,
This test is marked bringup:true due to #112130, but was previously successfully passing in staging.
Activity
jmagman commentedon Oct 10, 2022
cc oncall @drewroengoogle
drewroengoogle commentedon Oct 13, 2022
It seems like the
bundlerpath isn't being recognized at all, but I'm unsure what changed between the time this recipe was passing and failing, as the bundler version seems to be the same, and the recipe hasn't been modified during that timeframe. I can try taking this on.drewroengoogle commentedon Oct 18, 2022
Unassigning after discussion with @godofredoc about whether this is an infra issue.
jmagman commentedon Oct 19, 2022
@drewroengoogle do you know who is working on this? Is
/Volumes/Work/s/w/ir/cache/ruby/bin/bundlemissing? We want to add more Mac_arm64 tests #110126drewroengoogle commentedon Oct 20, 2022
@jmagman Thanks for following up, I meant to yesterday but got sidetracked.
I sent a message in the LUCI chat this morning and CC'ed you, as I'm not sure the best person to take a look at this. From my search it doesn't seem like there was any difference between the last successful run and the first failing run, the recipe, recipe properties, and ruby CIPD package did not change around that time at all.
godofredoc commentedon Oct 20, 2022
I took a quick look, removing the installation of ruby makes the gems steps to succeed but all the tests fail.
E.g. for x64 GEM_HOME is set to a generic folder where blundler and bundle are installed https://cs.opensource.google/flutter/recipes/+/main:recipe_modules/flutter_deps/api.py;l=300 and only after the gems path has been set and gems have been installed the path is changes to the ruby specific location https://cs.opensource.google/flutter/recipes/+/main:recipe_modules/flutter_deps/api.py;l=326
In arm64 GEM_HOME is set directly to <gem_dir>/ruby/3.1.0 when the bundler and bundle are located at <gem_dir>
Another potential problem is that ruby binaries from cipd are installed at the end of the path and most likely never used is ruby is already installed in the system or the code is running from an xcode context. https://cs.opensource.google/flutter/recipes/+/main:recipe_modules/flutter_deps/api.py;l=282
keyonghan commentedon Oct 20, 2022
Compared the failed build with succeeded one, the env.
dirpath root was switched from/optto/Volumes/Work, which caused issues. Followed up in the chat to see why such changes in our bots.keyonghan commentedon Oct 20, 2022
Chromium team reinstalled our bots on 10/07 morning, which is aligned with our incident here. It is suspected some hard coded paths somewhere.
godofredoc commentedon Oct 20, 2022
Apart from the what actually caused this problem. We need to make this installation hermetic and resilience to changes in the environment.
keyonghan commentedon Oct 20, 2022
Yeah, I agree. When creating the ruby version, a prefix path is needed to make it work. I believe that's the culprit here.
keyonghan commentedon Oct 31, 2022
Have spent quite some hours to work on the portable version, but still no luck to make it work.
Referenced https://github.com/stephan-nordnes-eriksen/ruby_ship, which seems promising to meet our purpose. However this still depends on openssl pre-installed in the bot, which will make the ruby doesn't work again if the root dir changes.
I am running out of ideas.. Anyone with experience on ruby may have a better idea on how to make it work? /cc @jmagman
One possible workaround is build/deploy openssl and ruby on the runtime, but it will add 5+ mins on each run.
jmagman commentedon Nov 2, 2022
Were you able to figure out why
Install gemsis working for other arm builders, likehttps://ci.chromium.org/p/flutter/builders/prod/Mac_arm64_ios%20build_ios_framework_module_test/2149
https://ci.chromium.org/p/flutter/builders/prod/Mac_arm64_ios%20flutter_gallery_ios__compile/3432
godofredoc commentedon Nov 2, 2022
Devicelab vs chromium machines?
keyonghan commentedon Nov 2, 2022
Yeah, these are devicelab testbeds where the root dirs are not changed (still /opt/s), whereas the chromium ones have been updated to
/Volumes/Work/s. The existing ruby was compiled based on/opt/s, so the devicelab ones are still valid.godofredoc commentedon Dec 14, 2022
@yusuf-goog FYI
godofredoc commentedon Jan 13, 2023
This is fixed now with ruby hermetic packages and https://flutter-review.googlesource.com/c/recipes/+/38088
github-actions commentedon Mar 4, 2023
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.