Skip to content

Expose the diff from ComparisonResult - #77014

Merged
fluttergithubbot merged 3 commits into
flutter:masterfrom
Piinks:comparisonDiff
Mar 2, 2021
Merged

fluttergithubbot merged 3 commits into
flutter:masterfrom
Piinks:comparisonDiff

Conversation

@Piinks

@Piinks Piinks commented Mar 1, 2021

Copy link
Copy Markdown
Contributor

This exposes the diff percentage of a ComparisonResult, which will enable sub-classed LocalFileComparators to set a tolerance level for golden file testing.

Pre-launch Checklist

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I read the Tree Hygiene wiki page, which explains my responsibilities.
  • I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
  • I signed the CLA.
  • I listed at least one issue that this PR fixes in the description above.
  • I updated/added relevant documentation (doc comments with ///).
  • I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt.
  • All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

@Piinks Piinks added a: tests "flutter test", flutter_test, or one of our tests framework flutter/packages/flutter repository. See also f: labels. labels Mar 1, 2021
@google-cla google-cla Bot added the cla: yes label Mar 1, 2021
@Piinks
Piinks requested a review from CaseyHillers March 1, 2021 23:04
Comment thread packages/flutter_test/lib/src/_goldens_io.dart Outdated
Comment thread packages/flutter_test/lib/src/_goldens_io.dart Outdated
Comment thread packages/flutter_test/lib/src/_goldens_io.dart Outdated
@burhanrashid52

Copy link
Copy Markdown

Can anyone help me to understand how do we set diff tolerance levels for the golden test ? I am not sure how I will use these api in my tests.
Currently I’ve golden test matcher like this.

await expectLater(find.byType(MyTestWidget),
    matchesGoldenFile('my_test_widget.png'));

How do I set a diff tolerance level for this test ?
@Piinks

@felix-barz-brickmakers

Copy link
Copy Markdown

anyone?

@CaseyHillers

Copy link
Copy Markdown
Contributor

Here's an example with a comparator where you can customize the tolerance.

class GoldenDiffComparator extends LocalFileComparator {
  GoldenDiffComparator(String testFile, {double tolerance = 0.005,}) : super(Uri.parse(testFile));

  @override
  Future<bool> compare(Uint8List imageBytes, Uri golden) async {
    final ComparisonResult result = await GoldenFileComparator.compareLists(
      imageBytes,
      await getGoldenBytes(golden),
    );

    if (!result.passed && result.diffPercent > _kGoldenDiffTolerance) {
      final String error = await generateFailureOutput(result, golden, basedir);
      throw FlutterError(error);
    }
    if (!result.passed) {
      log('A tolerable difference of ${result.diffPercent * 100}% was found when comparing $golden.');
    }
    return result.passed || result.diffPercent <= _kGoldenDiffTolerance;
  }
}

Then in the test you can call it like

goldenFileComparator = GoldenDiffComparator(path.join(
  (goldenFileComparator as LocalFileComparator).basedir.toString(),
  goldenFileKey,
));

expectLater(actual, matchesGoldenFile(goldenPath), reason: reason, skip: skip || !Platform.isLinux);

@fzyzcjy

fzyzcjy commented Jun 18, 2022

Copy link
Copy Markdown
Contributor

Hi, may I know what is goldenFileKey?

@fzyzcjy

fzyzcjy commented Jun 18, 2022

Copy link
Copy Markdown
Contributor

Seems that the code above does not work for macos. The following works:

MyLocalFileComparator(
      Uri.file(path.join(path.fromUri((goldenFileComparator as LocalFileComparator).basedir), 'just_arbitrary_string.dart')));

@ueman ueman mentioned this pull request Sep 29, 2023
diffPercent: diffPercent,
error: 'Pixel test failed, '
'${((pixelDiffCount/totalPixels) * 100).toStringAsFixed(2)}% '
'${(diffPercent * 100).toStringAsFixed(2)}% '

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Piinks When this is multiplied by 100, it looks like this is not a percentage to begin with, meaning it's 0-1 instead of 0-100, right?

Just was confused by some downstream code multiplying by 100, which I found unexpected for something called "percent" already.

As we probably can not change/rename this at this point, would you be fine with commenting the range on the property itself to make this clear? Or is my confusion unfounded?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Piinks 👋🏻
Like @tp , I found it unexpected that we’re multiplying a value that’s already a percentage. Could you clarify this when you have a moment? Thanks in advance!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

a: tests "flutter test", flutter_test, or one of our tests framework flutter/packages/flutter repository. See also f: labels.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants