[integration_test] Missing documentation on how to compare taken screenshots (like in golden tests)

Description

My goal is to write integration tests where I take a screenshot and compare this with a master screenshot, like you do in golden tests. So looked into the documentation on how do this. But I only found information about how to take a screenshot. How can I compare this taken screenshot now the with the master screenshot?

I searched a lot, but couldn’t any information. Until I found the PR for Android integration tests (#84472), where is some documentation about how to compare the screenshots:

final bytes = binding.takeScreenshot('myApp');
await expectLater(bytes, matchesGoldenFile('screenshot.png'));

I also found that you need to use flutter test integartion_test instead of flutter drive --driver=test_driver/integration_test.dart --target=integration_test/app_test.dart. Otherwise, your tests will pass and you will see in the console: "Golden file comparison requested for "screenshot-1.png"; skipping..." (

flutter/packages/flutter_test/lib/src/goldens.dart

Line 279 in 3752fb7

print('Golden file comparison requested for "$golden"; skipping...');

)

For me is also an open question, where should I locate the master screenshot? I'm always getting this error Which: Could not be compared against non-existent file: "goldens/screenshot-1.png") - I tried in integration_test/, test/ and Users.

@blasten

@blasten Can you help me find out where I need to store the screenshot.png (the master screenshot)? I saw you implemented #84472. If I have a working golden integration test, I can add some documentation to the README.md of the integration_test package 👍

When I run flutter test integration_test, I'm getting this error:

══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following TestFailure was thrown running a test:
Expected: one widget whose rasterized image matches golden image "screenshot-1.png"
  Actual: [
            137,
            80,
            78,
            71,
            13,
            10,
            26,
            10,
            0,
            0,
            0,
            13,
            73,
            72,
            68,
            82,
            0,
            0,
            4,
            56,
            0,
            0,
            7,
            236,
            ...
          ]
   Which: Could not be compared against non-existent file: "screenshot-1.png"

When the exception was thrown, this was the stack:
#0      fail (package:test_api/src/expect/expect.dart:137:31)
#1      _expect.<anonymous closure> (package:test_api/src/expect/expect.dart:112:9)
<asynchronous suspension>
<asynchronous suspension>
(elided 6 frames from dart:async and package:stack_trace)

The test description was:
  should show the counter app
════════════════════════════════════════════════════════════════════════════════════════════════════
00:17 +0 -1: Example should show the counter app [E]                                                                                                     
  Test failed. See exception logs above.
  The test description was: should show the counter app

You find a reproducible repository here (Flutter counter app with a simple integration test).

@nilsreichardt

Hey @nilsreichardt, going out on a limb here... did you first run flutter test integration_test --update-goldens?
This flag let's the test framework know that you'd like to create the missing golden files, instead of comparing them. You can at that point see where they've been created - for me, it was in the integration_test folder itself.

@brenodt

Hi @brenodt, of course it did this. Are you able to compare screenshots in integration tests? I have a reproducible repo where it's not working. You can clone it and try it out.

Test passes on my machine when using an iOS simulator.
Instead of pumping the app widget, I had to invoke the main function, though.

import 'package:example/main.dart' as app;
import 'package:flutter_test/flutter_test.dart';
import 'package:integration_test/integration_test.dart';

void main() {
  final binding = IntegrationTestWidgetsFlutterBinding.ensureInitialized();

  group('Example', () {
    testWidgets('should show the counter app', (tester) async {
      app.main();  // I changed this
      await tester.pumpAndSettle();

      await binding.convertFlutterSurfaceToImage();
      await tester.pumpAndSettle();

      final bytes = await binding.takeScreenshot('screenshot-1');
      await expectLater(bytes, matchesGoldenFile('screenshot-1.png'));
    });
  });
}

Maybe the issue you're having relates to using an Android device to run the test?

@nilsreichardt

TLDR taking screenshots & comparing to golden on device seems to be broken.

Running the repository linked by @nilsreichardt:

git clone https://github.com/nilsreichardt/playground.git && cd playground && git checkout af866e3   
flutter clean
flutter create --platforms=ios
flutter test -d <your iphone> integration_test/app_test.dart --update-goldens

will fail with Creation failed, path = '/Users' (OS Error: Operation not permitted, errno = 1). Making the changes suggested by @brenodt fails with the same error. Looks like binding.takeScreenshot is trying to save to a local directory which won't work on a physical device (but is probably fine on the simulator).

After fixing test_driver/integration_test.dart, the following will run/complete successfully on device:

flutter drive --driver=test_driver/integration_test.dart  -d <your iphone> --target=integration_test/app_test.dart

but the console prints:

flutter: Golden file comparison requested for "screenshot-1.png"; skipping...

Again, seems it's looking for that file on device, rather than locally on the host machine, so the golden comparison is never run.

My current solution is to just manually compare the test/golden images in the onScreenshot callback. The drawback is I lose have access to GoldenFileComparator which has a few nifty helper functions (I guess test_driver/integration_test.dart is running outside a Flutter desktop shell?).

Expected functionality: running flutter test integration_test/app_test.dart on a physical device with a call to matchesGoldenFile would dump golden files on the host.

I have the same error in Android Emulator: Which: Could not be compared against non-existent file: "example.png". in IOS everything's ok. No errors. I need to use this function in Android too.

Any updates on this?

Navigation Menu

Repository navigation

[integration_test] Missing documentation on how to compare taken screenshots (like in golden tests) #103222

Description

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[integration_test] Missing documentation on how to compare taken screenshots (like in golden tests) #103222

Description

Description

Activity

nilsreichardt commented on May 6, 2022

brenodt commented on Jul 1, 2022

nilsreichardt commented on Jul 1, 2022

brenodt commented on Jul 1, 2022

nmfisher commented on Oct 12, 2023

sk784 commented on Oct 16, 2023

HaraldWolz commented on Jun 12, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions