Repository navigation
✔️ Ideas to reduce flutter build apk flakes on CI #159000
Description
Working theory: This is caused by #51421.
A lot of credit goes both to @gmackall and @reidbaker for offline conversations that led to this issue.
As of today (2024-11-15), different integration tests (in particular, tooling integration tests, if that ends up mattering) are having somewhat significant and currently undiagnosed problems with tests that call (and block on) flutter build apk.
For example:
- Windows tool integration tests are flaky: "Java heap space" #157640
flutter build "apk" succeeds without librariesfromnative_assets_without_cbuild_assemble_test.dartis slow and flaky #158560
Three possible things happen that end up leading to failures:
- The test times out (taking longer than 15m) or the shard times out (taking longer than 30-60m)
- The test fails due to a Java Heap/OOM
- The test fails due to (hypothesis) the OS killing Gradle due to low memory
Some things to try:
- Try creating a
bringup: trueshard where failures happen somewhat consistently to try fixes - Try adding
--verboseto get information about the version of Gradle/Gradle settings being used - Remove the
no-daemonfrom recipes which doesn't appear to be having any effect/is confusing - Try using jps to list active Java processes and memory usage between/after tests.
- Try running with double the amount of RAM to see if higher memory lowers flakes significantly
/cc @andrewkolos @bkonyi.
Metadata
Metadata
Assignees
Labels
High-priority issues at the top of the work listIssue is blocked by another issueTests that sometimes, but not always, incorrectly passTechnical debt, code quality, testing, etc.For the attention of Android platform teamFor the attention of Infrastructure teamAndroid applications specificallyOwned by Flutter Tool team
Activity
Switch `flutter_build_apk_health_tests` to use a subset of current te…
matanlurey commentedon Nov 18, 2024
Unfortunately my
bringup: trueshard has ... yet to fail.I am not sure if its because the underlying issue has been resolved (and I could unskip the tests), if I'm missing some sort of condition that leads to these tests timing out (stale/corrupt caches), or more tests need to be run in a certain order to get into this bad state :-/
reidbaker commentedon Nov 18, 2024
Is the current state that your bringup tests are a clone of an existing shard/subshard?
matanlurey commentedon Nov 18, 2024
No, because unfortunately that is impossible (#158972).
gmackall commentedon Nov 18, 2024
Do you know what the flake rate of the existing shard it copied is? Should we have expected to see one for certain by now?
Perhaps there is some difference between staging and prod bots?
matanlurey commentedon Nov 18, 2024
It was relatively high (6%) before I added the skip statements, though that did include timeouts that were a result of the shard being too large, so it is possible it just hasn't happened yet. I suppose I could use LED to try and "speed things up" :P
matanlurey commentedon Nov 18, 2024
My current working theory is:
flutter build apkafter leaky process tests are more likely to OOM/fail... so, my next guess is solving #51421 is directly related to solving this issue (and that there is likely nothing directly we can or should do to change the Gradle configuration).
gmackall commentedon Nov 20, 2024
Not directly related to building apks, but another instance of something randomly taking much longer than expected:
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20build_android_host_app_with_module_source/235/overview
flutter doctortaking over 5 minutes and causing a timeoutLooks like there is an issue already #157513, not android test specific
matanlurey commentedon Nov 22, 2024
I haven't observed a flake ever since the process cleanup, so I am closing for now.
Remove now unused `apk-health-tests`. Can always re-add in the future. (
github-actions commentedon Dec 6, 2024
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.