Support golden file image diffing in testWidgets

After #16758 lands, we'll ostensibly have the ability to capture a screenshot in a widget test running via flutter_tester and compare it to a golden file, as follows:

testWidgets('test', (WidgetTester tester) async {
  await tester.pumpWidget(new MyApp());

  // Get the bytes of the screenshot
  RenderRepaintBoundary repaintRenderBoundary = tester
      .allRenderObjects
      .firstWhere((RenderObject object) => object is RenderRepaintBoundary);
  ui.Image image = await repaintRenderBoundary.toImage();
  ByteData bytes = await image.toByteData(format: ui.ImageByteFormat.png);

  // Get the bytes of the golden file
  File golden = new ui.File('/path/to/golden.png');
  List<int> goldenBytes = await golden.readAsBytes();

  // Fail the test if the screenshot isn't within tolerance of golden
  ...
});

But doing so causes the following exception due to flutter_test's use of FakeAsync:

  TimeoutException after 0:00:05.000000: Test timed out after 5 seconds.
  dart:async  _startMicrotaskLoop

A hacked up attempt to call _fakeAsync.flushTimers() ends up yielding the same exception when we await that call.

@Hixie

@Hixie thoughts?

we should just have a "tester.compareToGolden" or something that jumps through the appropriate hoops.

Within Google, there's an existing system set up to do golden file comparisons (it saves results to a location where they can be visualized, etc.). It has roughly the following API:

typedef Future<List<int>> ScreenshotTaker();

class GoldenFileComparisonService {
  /// User-defined function for taking a screenshot.
  ScreenshotTaker takeScreenshot;

  /// Makes a diff request against a golden image and publishes the result.
  Future<Null> diffScreenshot(String key);
}

Then test code is expected to hook into the API like so:

void main() {
  GoldenFileComparisonService service;

  setUp(() {
    service = new GoldenFileComparisonService();
  });

  testWidgets('matches golden file', (WidgetTester tester) async {
    service.takeScreenshot = () async {
      // Use `tester` to get image bytes
    };

    await service.diffScreenshot('imageKey.png');
  });
}

So at the heart of it, the service expects two things:

We have a method that can return Future<List<int>> for the screenshot bytes
We can safely await arbitrary futures in our test code

We potentially have some amount of influence over that API, but we do want to delegate to the service to do the comparison, because it exports its findings in such a way that they can be visualized in test results (providing a suite of UI tools for visualizing the differences and approving the test files as the new goldens).

@Hixie

Discussed with @Hixie - here's what we came up with...

We introduce the notion of a static image comparison "backend" into WidgetTester:

/// Compares [testBytes] against the golden file identified by [goldenKey].
///
/// The returned future will complete with a boolean value that indicates
/// whether [testBytes] match the golden bytes within the tolerance defined by
/// the comparator. In the case of failure, the comparator may also throw a
/// [TestFailure] if it wants to control the failure message.
typedef Future<bool> GoldenComparator(Uint8List testBytes, String goldenKey);

class WidgetTester {
  /// Compares rasterized images bytes against a golden image file.
  ///
  /// This comparator is used as the backend for [matchesGoldenFile].
  ///
  /// The default comparator will treat the golden key as a relative path from
  /// the test file's directory, load the golden file's bytes from disk, and
  /// perform a byte-for-byte comparison, returning true only if there's an
  /// exact match
  static GoldenComparator compareAgainstGolden;
}

As referenced in the comments for #1, we introduce a new matchesGoldenFile(String key) top-level constant in package:flutter_test. It'll be an AsyncMatcher as follows:

Matcher matchesGoldenFile(String key) =>
    new _MatchesGoldenFile(key);

class _MatchesGoldenFile extends AsyncMatcher {
  _MatchesGoldenFile(this.key);

  final String key;

  @override
  Future<String> matchAsync(dynamic item) async {
    // Do whatever magic is needed to run this async code in the correct zone
    // <assert that [item] is a finder>
    // Uint8List bytes = <get image bytes for [item]>
    // Invoke [compareAgainstGolden], passing [bytes] and [key]
    // If comparator threw [TestFailure], catch it and return the exception's message
    // Otherwise, if comparator returned true, return null
    // Otherwise (if comparator returned false), return generic error message
  }
}

Users of the new matcher would use it like so:

testWidgets('test name', (WidgetTester tester) async {
  Finder finder = ...
  expect(finder, matchesGoldenFile('relative/path/to/golden.png'));
});

To enable users to do things like modify the default comparator for all tests in their repo, we add a hook for tests to globally initialize their test environment.

The hook would be a configuration file that, by contract, will be discovered by the Flutter test harness. Starting from the test directory, we'd search the directory ancestry for a file named flutter_test_config.dart. As soon as one is found, we'll stop searching and execute its main() method before invoking the test's main method.

We'll implement this hook here:

flutter/packages/flutter_tools/lib/src/test/flutter_platform.dart

Line 617 in fd7222e

String _generateTestMain({
We add a packages/flutter/test/flutter_test_config.dart that will override the default comparator with one that reads a Git hash from a newly added bin/internal/goldens.version file, checks out a local clone of a newly added github.com/flutter/goldens repository, and reads golden files from that clone.
We add a new --update-goldens flag to flutter test, that will, whenever a golden file match fails, update the golden file with the bytes of the test image. This will be used by developers when they know they've made a UI update, and they wish to automatically replace all golden files with the version produced by the test run.
We add a new hidden --local-goldens flag to flutter_test and plumb the flag through to flutter_test_config.dart, making our local config respect it. This will enable developers to update goldens for Flutter framework changes with the following invocation:
```
$ flutter test --local-goldens=<value> --update-goldens
```
The developer will still then have to land a PR to the flutter/goldens repo and then update bin/internal/goldens.version after their test run. This rigmarole is needed to keep from checking binary golden files into the main Flutter repo.

SGTM.

Rather than --local-goldens how about just putting the checkout in bin/cache? That way everyone has it in a known location.

We'll have to make sure the flutter_test_config.dart logic is protected by a lock to prevent multiple simultaneous checkouts or some such.

~~Currently blocked on dart-lang/sdk#32982~~

Move toImage() impl from RepaintRenderBoundary to OffsetLayer (#17060)

Add `expectLater` to flutter_test (#17063)

Add WidgetTester.runAsync() (#17029)

Fix expectLater to not wrap the delegate in a guard (#17085)

Add basic support for golden image file testing (#17094)

Fix RenderView.paintBounds (#17126)

Make Layer.toImage() accept bounds rather than size (#17129)

Ok, marking this as done. There are still several improvements we should make, being tracked in:

Implement better logic in golden file comparators #17258 -- Implement better logic in golden file comparators
Centralized location for test file output #17259 -- Allow centralized place where tests can output files
Visualize golden image file failures in test output #17260 -- Visualize golden image file failures in test output
Test golden files without shadows #17262 -- Test golden files without shadows

Move toImage() impl from RepaintRenderBoundary to OffsetLayer (flutte…

Add `expectLater` to flutter_test (flutter#17063)

Add WidgetTester.runAsync() (flutter#17029)

Fix expectLater to not wrap the delegate in a guard (flutter#17085)

Add basic support for golden image file testing (flutter#17094)

Fix RenderView.paintBounds (flutter#17126)

Make Layer.toImage() accept bounds rather than size (flutter#17129)

Add support for flutter_test_config.dart (flutter#17141)

Add Flutter-repo-specific golden comparator (flutter#17205)

This this in production already? Is there a command I can run to see the diff between a new snapshot and an old one?

@FrankApiyo

@FrankApiyo Just found this:
https://medium.com/flutter-community/flutter-golden-tests-compare-widgets-with-snapshots-27f83f266cea

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

Navigation Menu

Repository navigation

Support golden file image diffing in `testWidgets` #16859

3 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support golden file image diffing in testWidgets #16859

Description

Activity

tvolkert commented on Apr 22, 2018

Hixie commented on Apr 22, 2018

tvolkert commented on Apr 22, 2018

tvolkert commented on Apr 23, 2018

Hixie commented on Apr 23, 2018

tvolkert commented on Apr 26, 2018

3 remaining items

tvolkert commented on May 3, 2018

FrankApiyo commented on Aug 28, 2019

rknell commented on Nov 25, 2019

github-actions commented on Aug 25, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Support golden file image diffing in `testWidgets` #16859