flutter_goldens generate different files everytime

Hello, after migrating to flutter 2.5.0 our goldens failed as we noticed that there was a change in the shadows. We then had to recreate all of our goldens. After runing the tests with the new goldens, we still received a 0.0% diff in one of our screens.

Looking further it seems that for that particular widget the shadows are created differently each time, so a test can pass on one run and fail on the other, it's being totally random so far.

We extracted the code causing the issue and created a test repo.

Steps to Reproduce

With flutter 2.5.x Execute flutter test on the provided example repo, see how some test will fail
Execute flutter test --update-goldens to create a fresh set of images
Execute flutter test again, and see how some test will still fail

Findings

We have a function called setupSize to setup the size of the golden image. If we remove it's contents, the issue is fixed.
We have this code

Scaffold(
      body: Stack(
        children: [
       // other children removed for demo
          Positioned(
            top: _defaultFloatingTopPosition - _offset,
            right: 0,
            left: 0,
            child: IgnorePointer(....
        )
)

if we remove the Stack(children[Positioned( the issue is fixed.

Scaffold(
      body: IgnorePointer(...

Expected results:
Golden tests should not fail after updating images

Actual results:
Golden tests fail randomly

Code sample

https://github.com/diegog-sf/goldendemo

Logs

flutter test

00:05 +4 -2: Home golden tests group2 Test7                                                                                                                                                                                      
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following assertion was thrown while running async test code:
Golden "goldens/Test7.png": Pixel test failed, 0.00% diff detected.
Failure feedback can be found at /Users/diego/workspaces/goldendemo/test/failures

When the exception was thrown, this was the stack:
#0      LocalFileComparator.compare (package:flutter_test/src/_goldens_io.dart:101:7)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)

Analyzing goldendemo...                                                 
No issues found! (ran in 1.1s)

[✓] Flutter (Channel stable, 2.5.3, on macOS 12.0.1 21A559 darwin-x64, locale en-MX)
    • Flutter version 2.5.3 at /Users/diego/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 18116933e7 (12 days ago), 2021-10-15 10:46:35 -0700
    • Engine revision d3ea636dc5
    • Dart version 2.14.4

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at /Users/diego/Library/Android/sdk
    • Platform android-31, build-tools 30.0.3
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 13.1, Build version 13A1030d
    • CocoaPods version 1.11.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)

[✓] VS Code (version 1.61.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.27.0

[✓] Connected device (2 available)
    • iPhone 13 Pro (mobile) • 4E63E183-DACE-4D21-A433-55DDA5A59F7E • ios            • com.apple.CoreSimulator.SimRuntime.iOS-15-0 (simulator)
    • Chrome (web)           • chrome                               • web-javascript • Google Chrome 95.0.4638.54

• No issues found!

@diegog-sf

Hi @diegog-sf, Thanks for filing the issue. I can reproduce the issue on the latest stable and the master channel the updated goldens tests are failing. And the tests are failing randomly each time the goldens are updated.

flutter doctor -v

[✓] Flutter (Channel master, 2.6.0-12.0.pre.478, on macOS 12.0.1 21A559 darwin-arm, locale en-GB)
    • Flutter version 2.6.0-12.0.pre.478 at /Users/mahesh/Documents/flutter_master
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision abecef6ed3 (30 hours ago), 2021-10-26 20:31:29 -0700
    • Engine revision 925bae45ee
    • Dart version 2.15.0 (build 2.15.0-250.0.dev)

[✓] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
    • Android SDK at /Users/mahesh/Library/Android/sdk
    • Platform android-31, build-tools 31.0.0
    • ANDROID_HOME = /Users/mahesh/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 13.0)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • CocoaPods version 1.10.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)

[✓] IntelliJ IDEA Community Edition (version 2021.2.1)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin version 60.1.4
    • Dart plugin version 212.5080.8

[✓] VS Code (version 1.59.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.27.0

[✓] Connected device (3 available)
    • sdk gphone64 arm64 (mobile) • emulator-5554 • android-arm64  • Android 12 (API 31) (emulator)
    • macOS (desktop)             • macos         • darwin-arm64   • macOS 12.0.1 21A559 darwin-arm
    • Chrome (web)                • chrome        • web-javascript • Google Chrome 95.0.4638.54

• No issues found!

[✓] Flutter (Channel stable, 2.5.3, on macOS 12.0.1 21A559 darwin-arm, locale en-GB)
    • Flutter version 2.5.3 at /Users/mahesh/Documents/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 18116933e7 (13 days ago), 2021-10-15 10:46:35 -0700
    • Engine revision d3ea636dc5
    • Dart version 2.14.4

[✓] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
    • Android SDK at /Users/mahesh/Library/Android/sdk
    • Platform android-31, build-tools 31.0.0
    • ANDROID_HOME = /Users/mahesh/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 13.0, Build version 13A233
    • CocoaPods version 1.10.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)

[✓] IntelliJ IDEA Community Edition (version 2021.2.1)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin version 60.1.4
    • Dart plugin version 212.5080.8

[✓] VS Code (version 1.59.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.27.0

[✓] Connected device (3 available)
    • sdk gphone64 arm64 (mobile) • emulator-5554 • android-arm64  • Android 12 (API 31) (emulator)
    • macOS (desktop)             • macos         • darwin-arm64   • macOS 12.0.1 21A559 darwin-arm
    • Chrome (web)                • chrome        • web-javascript • Google Chrome 95.0.4638.54

• No issues found!

logs(master)

mahesh@Maheshs-Air-M1 goldendemo % flutterm test     
00:12 +2: Home golden tests group1 Test3                                                                                         
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following assertion was thrown while running async test code:
Golden "goldens/Test3.png": Pixel test failed, 0.00% diff detected.
Failure feedback can be found at /Users/mahesh/Development/triage/ISSUES/goldendemo/test/failures

When the exception was thrown, this was the stack:
#0      LocalFileComparator.compare (package:flutter_test/src/_goldens_io.dart:100:7)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)
════════════════════════════════════════════════════════════════════════════════════════════════════
00:12 +2 -1: Home golden tests group1 Test3 [E]                                                                                  
  Test failed. See exception logs above.
  The test description was: Test3
  
00:14 +4 -1: Home golden tests group2 Test6                                                                                      
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following assertion was thrown while running async test code:
Golden "goldens/Test6.png": Pixel test failed, 0.00% diff detected.
Failure feedback can be found at /Users/mahesh/Development/triage/ISSUES/goldendemo/test/failures

When the exception was thrown, this was the stack:
#0      LocalFileComparator.compare (package:flutter_test/src/_goldens_io.dart:100:7)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)
════════════════════════════════════════════════════════════════════════════════════════════════════
00:14 +4 -2: Home golden tests group2 Test6 [E]                                                                                  
  Test failed. See exception logs above.
  The test description was: Test6
  
00:15 +4 -2: Home golden tests group2 Test7                                                                                      
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following assertion was thrown while running async test code:
Golden "goldens/Test7.png": Pixel test failed, 0.00% diff detected.
Failure feedback can be found at /Users/mahesh/Development/triage/ISSUES/goldendemo/test/failures

When the exception was thrown, this was the stack:
#0      LocalFileComparator.compare (package:flutter_test/src/_goldens_io.dart:100:7)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)
════════════════════════════════════════════════════════════════════════════════════════════════════
00:15 +4 -3: Home golden tests group2 Test7 [E]                                                                                  
  Test failed. See exception logs above.
  The test description was: Test7
  
00:16 +4 -3: Home golden tests group3 Test8                                                                                      
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following assertion was thrown while running async test code:
Golden "goldens/Test8.png": Pixel test failed, 0.00% diff detected.
Failure feedback can be found at /Users/mahesh/Development/triage/ISSUES/goldendemo/test/failures

When the exception was thrown, this was the stack:
#0      LocalFileComparator.compare (package:flutter_test/src/_goldens_io.dart:100:7)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)
════════════════════════════════════════════════════════════════════════════════════════════════════
00:16 +4 -4: Home golden tests group3 Test8 [E]                                                                                  
  Test failed. See exception logs above.
  The test description was: Test8
  
00:17 +4 -4: Home golden tests group3 Test9                                                                                      
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following assertion was thrown while running async test code:
Golden "goldens/Test9.png": Pixel test failed, 0.00% diff detected.
Failure feedback can be found at /Users/mahesh/Development/triage/ISSUES/goldendemo/test/failures

When the exception was thrown, this was the stack:
#0      LocalFileComparator.compare (package:flutter_test/src/_goldens_io.dart:100:7)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)
════════════════════════════════════════════════════════════════════════════════════════════════════
00:17 +4 -5: Home golden tests group3 Test9 [E]                                                                                  
  Test failed. See exception logs above.
  The test description was: Test9
  
00:17 +4 -5: Some tests failed.                                                                                                  
mahesh@Maheshs-Air-M1 goldendemo % flutterm test --update-goldens 
00:04 +9: All tests passed!                                                                                                      
mahesh@Maheshs-Air-M1 goldendemo % flutterm test                  
00:04 +1: Home golden tests group1 Test2                                                                                         
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following assertion was thrown while running async test code:
Golden "goldens/Test2.png": Pixel test failed, 0.00% diff detected.
Failure feedback can be found at /Users/mahesh/Development/triage/ISSUES/goldendemo/test/failures

When the exception was thrown, this was the stack:
#0      LocalFileComparator.compare (package:flutter_test/src/_goldens_io.dart:100:7)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)
════════════════════════════════════════════════════════════════════════════════════════════════════
00:04 +1 -1: Home golden tests group1 Test2 [E]                                                                                  
  Test failed. See exception logs above.
  The test description was: Test2
  
00:06 +3 -1: Home golden tests group2 Test5                                                                                      
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following assertion was thrown while running async test code:
Golden "goldens/Test5.png": Pixel test failed, 0.00% diff detected.
Failure feedback can be found at /Users/mahesh/Development/triage/ISSUES/goldendemo/test/failures

When the exception was thrown, this was the stack:
#0      LocalFileComparator.compare (package:flutter_test/src/_goldens_io.dart:100:7)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)
════════════════════════════════════════════════════════════════════════════════════════════════════
00:06 +3 -2: Home golden tests group2 Test5 [E]                                                                                  
  Test failed. See exception logs above.
  The test description was: Test5
  
00:07 +7 -2: Some tests failed.                                                                                                  
mahesh@Maheshs-Air-M1 goldendemo %

Is this an antialiasing change? Stepping from 2.2.X to 2.5.X specifically is causing the issue to happen.

Hello, I found something interesting.

from the example provided, if I do as below, the problem gets fixed.
top: (_defaultFloatingTopPosition - _offset). roundToDouble(),

Could the problem be because floating points aren’t determinant?

I just upgraded to 2.8.0
I'm now having issues with this simple widget.

 Scaffold(
      floatingActionButton: FloatingActionButton.extended(
        onPressed: () {},
        label: Text('Hello'),
        elevation: 2,
        backgroundColor: Theme.of(context).primaryColor,
      ),
      body: Text('Testing'),
    );

It is reproducible in the example project provided

should we be using debugDisableShadows and override the elevation ourselves? also the documentation of the variable seems wrong

/// Whether to replace all shadows with solid color blocks.
///
/// This is useful when writing golden file tests (see [matchesGoldenFile]) since
/// the rendering of shadows is not guaranteed to be pixel-for-pixel identical from
/// version to version (or even from run to run).
///
/// In those tests, this is usually set to false at the beginning of a test and back
/// to true before the end of the test case.
///
/// If it remains true when the test ends, an exception is thrown to avoid state
/// leaking from one test case to another.
bool debugDisableShadows = false;

Shouldn't it be:

In those tests, this is usually set to true at the beginning of a test and back to false before the end of the test case.

but all tests on flutter do set it to false at the beginning

Navigation Menu

Repository navigation

flutter_goldens generate different files everytime #92590

Steps to Reproduce

Findings

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

flutter_goldens generate different files everytime #92590

Description

Steps to Reproduce

Findings

Activity

maheshj01 commented on Oct 28, 2021

mattsolle commented on Nov 12, 2021

diegog-sf commented on Nov 19, 2021

diegog-sf commented on Dec 9, 2021

diegog-sf commented on Dec 9, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions