Repository navigation
Linux analyzer_benchmark is 9.09% flaky #161306
Description
The post-submit test builder Linux analyzer_benchmark had a flaky ratio 9.09% for the past (up to) 100 commits, which is above our 2.00% threshold.
One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20analyzer_benchmark/11268
Commit: b5df290
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20analyzer_benchmark/11268
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20analyzer_benchmark/11257
Recent test runs:
https://flutter-dashboard.appspot.com/#/build?taskFilter=Linux%20analyzer_benchmark
Please follow https://github.com/flutter/flutter/blob/master/docs/infra/Reducing-Test-Flakiness.md#fixing-flaky-tests to fix the flakiness and enable the test back after validating the fix (internal dashboard to validate: go/flutter_test_flakiness).
Activity
matanlurey commentedon Jan 9, 2025
I'll take a look as Andrew is OOO (sick).
Marks Linux analyzer_benchmark to be flaky (#161307)
Move the analyzer_benchmark to Mac arm64 devicelab bots
zanderso commentedon Jan 10, 2025
@jason-simmons from chat:
and
The script that runs the benchmarks is at https://github.com/flutter/flutter/blob/master/dev/devicelab/lib/tasks/analysis.dart.
An interesting one is the
mega_gallerybenchmark that creates copies of the flutter_gallery until it reaches 64K lines of code, and then benchmarks the analysis time a few times. I suspect this is trying to simulate analysis times for a real large app.https://github.com/flutter/flutter/blob/master/dev/tools/mega_gallery.dart
cc @jensjoha @mraleph
jason-simmons commentedon Jan 10, 2025
To execute the most memory intensive part of this benchmark, run:
Tried bisecting and found that there was a major regression in analysis server memory usage starting with https://dart.googlesource.com/sdk/+/d83aa005eee28233367a693bc83ba59e8504ba9f
For the
flutter analyze --flutter-repocommand above, after that commit the peak VmRSS of the analysis server increases from ~4GB to ~7GB.@scheglov
scheglov commentedon Jan 10, 2025
This is sad. IIRC, the change on Golem for the analysis server analyzing itself was not so dramatic.
Does it block you? I could try to look if it is feasible to restore this optimization.
scheglov commentedon Jan 11, 2025
https://dart-review.googlesource.com/c/sdk/+/403924 looks promising in Golem, about 4.2% less memory.
Not a huge difference, but as I mentioned, it was not huge in that other CL too.
jason-simmons commentedon Jan 11, 2025
The https://dart-review.googlesource.com/c/sdk/+/403924 patch helps significantly.
With that change, the peak memory usage of
flutter analyze --flutter-repois reduced to 4.6GB.scheglov commentedon Jan 11, 2025
@jason-simmons That's great to hear! 4.6GB is a significant improvement. Apparently the effect on Flutter applications is much bigger than to applications that use smaller libraries. I'll get that patch landed once it is reviewed. Thanks for testing!
mraleph commentedon Jan 13, 2025
@srawlins @scheglov could somebody look into why Golem numbers don't directly align with Flutter numbers? I would like to make sure that Golem numbers are representative of the real world so that we could catch regressions like this.
Move the analyzer_benchmark to Mac arm64 devicelab bots
Elements. Restore lazy reading ClassElement members.
srawlins commentedon Jan 13, 2025
I can try to look into this.
Move the analyzer_benchmark to Mac arm64 devicelab bots (#161405)
mraleph commentedon Jan 14, 2025
@srawlins
I have chatted with @a-siva and we suspect that we have not caught this regression on Golem because flutter-analyzer target is broken by monorepo migration. AFAIK @aam is working on fixing Golem so maybe we don't actually have a problem with benchmarks, it's just that the Flutter benchmark was not running at that moment.
aam commentedon Jan 14, 2025
FlutterAnalyze on golem is back up and running, but it doesn't capture memory consumption, only runtime performance(which shows 3.56% runtime slowdown
between [vm] Remove BaseIsolate as it serves no purposes anymore. and Elements. Add Element2.fragments - such a large gap due to failing runs due to monorepo).
matanlurey commentedon Jan 14, 2025
I am going to close this specific bug (the flakiness) and have created sub-issues for the Dart team to resolve:
Move the analyzer_benchmark to Mac arm64 devicelab bots (flutter#161405)
github-actions commentedon Jan 28, 2025
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.