allow golden tests to specify epsilon tolerance - #77703

Hey @lukepighetti thanks for the contribution! I looked at doing this in #76545 but found some issues with it. Changing the signature of compare is a pretty big breaking change. It also does not seem like the best way to approach setting a tolerance level like this for golden file tests. If a user has a LOT of golden file tests, they'll have to set a tolerance level for every single call to matchesGoldenFile, which feels a bit tedious. There is some further discussion here: #76545 (comment)
What we did do what expose the diff from ComparisonResult so users can extend or implement their own GoldenFileComparator, and override compare to factor in a tolerance limit of their choosing. We did that in #77014

I am planning on adding some sample code that shows how to do that, I think flutter/cocoon creates their own comparator where we'll be doing something similar. :)

Closing due to #76337 (comment) and above comment, thanks Kate!

It looks like the easiest path forward is to copy-paste LocalFileComparator and change

this line

if (!result.passed) {
  final String error = await generateFailureOutput(result, golden, basedir);
  throw FlutterError(error);
}

to this line

final acceptableDiffPercent = 0.01;

if (!result.diffPercent < acceptableDiffPercent) {
  final String error = await generateFailureOutput(result, golden, basedir);
  throw FlutterError(error);
}

is that about right?

After #77825, it's possible to do it this way: flutter/cocoon#1106 :)

@Piinks Gotcha, that's super helpful. How does one feed CocoonFileComparator to the test runner? Any reason why it wasn't added to the framework as something like TolerancedFileComparator? Can it be fed to individual tests?

How does one feed CocoonFileComparator to the test runner?

There are a couple of ways to do it. The flutter/cocoon one changes the golden comparator at the invocation of the comparison.
Here: https://github.com/flutter/cocoon/blob/7fe6eb26a53d2baf9049dd83f52aeb43d7ca72f8/app_flutter/test/utils/golden.dart#L27

But the flutter_goldens package does this differently, using the flutter_test_config.dart,
here:

and here:

Any reason why it wasn't added to the framework as something like TolerancedFileComparator?

So, we don't really want to make this easy to do. 😜 We don't want to recommend relying on setting a tolerance level for golden files. Adding a TolerancedFileComparator kind of enables 'tests that don't really matter'. If it can fail some of the time, and that's ok, then why have the test? Plus, setting a tolerance does not actually solve anything. Eventually, the golden files will drift with each subtle change until they finally hit that tolerance level. It is basically just making today's failure a future failure.
Typically when we have a flaky test, we look at fixing the test rather than making it possible for the test to pass through on a failure, because sometimes, you'll end up missing a real bug. For now, we've implemented this in flutter/cocoon to unblock engine rolls while we investigate other options.

Can it be fed to individual tests?

I believe the cocoon implementation affords a unique comparator for each test, while the flutter_goldens approach creates a comparator for each test file. You can use either one and modify to your preference. :)

It's true that the desire for golden tolerance is due to something that I would consider broken (if you run goldens on different operating systems you get different results), but we've needed relief from this for the last three years so I'm very happy to be able to make use of it until that issue can be resolved. Thanks for your efforts in this area and I look forward to using golden tests again soon with toleranced comparisons.