Repository navigation
Regression in some microbenchmarks_ios benchmarks. #68890
Description
Affects the following benchmarks:
stock_animation_close_first_frame_average
stock_animation_open_first_frame_average
stock_animation_subsequent_frame_average
First occurred at #67781, but could be later; it continued to rise. After that, it's dropped a little. Marking P1.
cc @jmagman , originator of that commit.
Activity
jmagman commentedon Oct 23, 2020
hm, I don't see how #67781 could be related, it introduced a new flag to a command those benchmarks aren't touching. Though the graph does indeed look like it started there.
justinmc commentedon Oct 28, 2020
Another nearby PR was #66785, which I reviewed, but it similarly seems unrelated. It only added a pass through parameter to text editing widgets.
What is it that these benchmarks measure exactly?
There is a lot of NNBD work in the vicinity of this change (relevant commits page), but I also wouldn't think that would cause this. CC @goderbauer @jonahwilliams for nearby PRs.
goderbauer commentedon Oct 28, 2020
Nothing in that range looks suspicious in the framework - especially for something that would only affect the ios version of these benchmarks (Android seems unaffected).
Did we ever fix the problem where random iOS pop-ups mess with our benchmark data? Could that be the problem here?
goderbauer commentedon Oct 28, 2020
Also: In more recent runs, the benchmark seems to be bi-modal: https://flutter-flutter-perf.skia.org/e/?begin=1602198589&end=1603867624&formulas=norm(shortcut(%22X1fa8ce79a6a882d2eacbb51a27b2fec2%22))&requestType=0&xbaroffset=22131
Some runs are close to what they were before this regression, others are still high.
goderbauer commentedon Oct 28, 2020
@godofredoc confirmed that those get automatically dismissed now.
liyuqian commentedon Nov 2, 2020
Reducing priority as this benchmark recently dropped and increased again on some seemingly unrelated commits. There doesn't seem to be anything actionable other than trying to reproduce it locally. If we can't reproduce locally, something might be happening in our infra that caused these benchmark jumps.
godofredoc commentedon Nov 2, 2020
Did those metrics dropped last Thursday after 3:00 PM?
liyuqian commentedon Nov 3, 2020
One drop occurred around 291ee94 which landed last Tuesday 9am.
Another drop occurred around 3e41962 which landed last Friday 1pm.
This is what I'm looking at: https://flutter-flutter-perf.skia.org/e/?begin=1602548289&end=1604386684&keys=X1fa8ce79a6a882d2eacbb51a27b2fec2&requestType=0&xbaroffset=22131
kf6gpe commentedon Nov 9, 2020
triage: The number's gotten slightly better, but the midpoint's still above the old value. I'm going to leave this open for now.
kf6gpe commentedon Nov 16, 2020
I don't know if this is a bona fide regression or not --- it's bouncing around slightly higher than the old values on average if I eyeball it, but that's a bad measure.
@liyuqian What do you think?
liyuqian commentedon Nov 16, 2020
By focusing on just microbenchmarks_ios stock_animation_subsequent_frame_average, I can see it to be around 290ns before the regression, and now the metric has gradually reduced to a level between 297ns to 340ns. It's much less than the peak regression around 500ns to 600ns.
I'm reducing the priority because the number now is only slightly higher, and the micro benchmarks usually don't have a big impact on the end-user's experience directly. Let's hope that the downward trend will continue and we can probably close this soon.
Hixie commentedon Mar 10, 2021
Looks like we dropped the ball on this so I'm just going to close it, too much time has passd now to really do anything.
github-actions commentedon Aug 4, 2021
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.