Skip to content

Disable tests using gauge - #50367

Merged
dnfield merged 1 commit into
flutter:masterfrom
dnfield:disable_test
Feb 7, 2020
Merged

dnfield merged 1 commit into
flutter:masterfrom
dnfield:disable_test

Conversation

@dnfield

@dnfield dnfield commented Feb 7, 2020

Copy link
Copy Markdown
Contributor

This test is flaking at an unacceptable rate, and is incompatible with certain versions of iOS and certain versions of Xcode that we need to be able to test on.

It is also preventing another test from detecting regresions on an iPhoneX, which is what the target was original set up for.

/cc @digiter @godofredoc @keyonghan

issue tracking this: #45435

See also: #49787

I plan to land this TBR on red to make infra less flaky.

@Hixie

Hixie commented Mar 17, 2020

Copy link
Copy Markdown
Contributor

I just closed a bug that was complaining about this test being flaky. We should probably just delete the files since nobody is championing them.

@dnfield

dnfield commented Mar 17, 2020

Copy link
Copy Markdown
Contributor Author

#45435 is the bug tracking re-enabling. @gaaclarke @liyuqian and I all looked at making it work at some point but it's not trivial and it's probably not sustainable in its current form.

@flar

flar commented Mar 18, 2020 •

Copy link
Copy Markdown
Contributor

The post_backdrop_filter tests were representing a concern from a developer. The CPU/GPU measurements helped us track the state of the fix that helped drastically reduce the energy usage for their app. The fact that we have no replacement for that measurement is of big concern. The energy usage could be tracked manually using Xcode and its energy graphing tools, but that can't be automated the same way that gauge was doing for us until it got broken by the upgrade to Xcode 11.

@liyuqian has more details on the developers who have been running into that problem.

The fix that was being tested for regression by these benchmarks is flutter-team-archive/engine#14104

@dnfield

dnfield commented Mar 18, 2020

Copy link
Copy Markdown
Contributor Author

I don't think anyone is opposed to testing this. We just need tests that are reliable enough to not close the tree and do not require the devicelab to stay on a fixed version of iOS we can't keep it on.

@liyuqian

Copy link
Copy Markdown
Contributor

I'm pushing the work to get the measurement tool work again, and our test to be reliable and not flaky. Please stay tuned :)

@flar

flar commented Mar 23, 2020

Copy link
Copy Markdown
Contributor

Originally I thought this was just getting rid of the CPU/GPU measurements of these benchmarks, but it looks like they were deleted entirely. Note that the benchmarks measure both the standard ms/frame factors and they also separately produce the CPU/GPU numbers. We can disable their dependency on the flutter gauge mechanism independently of their other role as a general benchmark.

Was deleting them in their entirety the intended goal here, or just the dependency on gauge?

@dnfield

dnfield commented Mar 23, 2020

Copy link
Copy Markdown
Contributor Author

Just the dependency on gauge. It's fine to add them back if they don't depend on that.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

c: contributor-productivity Team-specific productivity, code health, technical debt.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants