The app doesn't stop after failing golden test. + Error in TestGoldenComparatorProcess

Steps to reproduce

Clone the repo. It has a sample golden test with an image that fails the test intentionally.
Launch iOS simulator
Run in the terminal flutter test integration_test/E2Es

Expected results

The test process stops. I should not kill it manually.

Actual results

The test process doesn't exit having the following output:

<< [ERROR:flutter/shell/testing/tester_main.cc(682)] Unhandled exception
<<< Exception: Golden "goldens/failing_counter.png": Pixel test failed, 0.11%, 4496px diff detected.
<<< Failure feedback can be found at /Users/x/Documents/Flutter/golden_tests/integration_test/E2Es/failures
<<< Stack trace: #0      LocalFileComparator.compare (package:flutter_test/src/_goldens_io.dart:108:5)
<<< <asynchronous suspension>
<<< #1      main (file:///var/folders/6n/2pnrqh45205_wq6kvgs8xmq00000gn/T/flutter_tools.40hhzs/flutter_web_platform.TmZB1C/listenerHyn8yD/listener.dart:29:26)
<<< <asynchronous suspension>
<<< 

00:23 +0 -1: loading /Users/x/Documents/Flutter/golden_tests/integration_test/E2Es/ui_test.dart [E]                                                                 
  type 'Null' is not a subtype of type 'Map<String, dynamic>'
  dart:async/stream_impl.dart 1038:17                                _StreamIterator.current
  package:flutter_tools/src/test/test_golden_comparator.dart 315:27  TestGoldenComparatorProcess.getResponse
  ===== asynchronous gap ===========================
  package:flutter_tools/src/test/test_golden_comparator.dart 161:41  TestGoldenComparator._compareGoldens
  ===== asynchronous gap ===========================
  package:flutter_tools/src/test/test_golden_comparator.dart 127:28  TestGoldenComparator.compare
  ===== asynchronous gap ===========================
  package:flutter_tools/src/test/flutter_platform.dart 554:17        FlutterPlatform._listenToVmServiceForGoldens.<fn>

Code sample

import 'package:flutter_test/flutter_test.dart';
import 'package:golden_tests/main.dart';
import 'package:integration_test/integration_test.dart';

const kGoldensFolder = 'goldens';

void main() {
  testWidgets('Counter smoke test', (WidgetTester tester) async {
    await tester.pumpWidget(const MyApp());

    expect(find.text('0'), findsOneWidget);
    expect(find.text('1'), findsNothing);

    await tester.pumpAndSettle(const Duration(seconds: 5));

    await compareScreenshot(kGoldensFolder, 'failing_counter');
  });
}

Future<void> compareScreenshot(String goldensFolder, String key) async {
  List<int> screenshotBytes =
      await IntegrationTestWidgetsFlutterBinding.ensureInitialized()
          .takeScreenshot(key);

  await expectLater(
    screenshotBytes,
    matchesGoldenFile('$goldensFolder/$key.png'),
  );
}

Screenshots or Video

Screenshots

Logs

No response

Flutter Doctor output

Doctor output

[✓] Flutter (Channel stable, 3.35.1, on macOS 15.6 24G84 darwin-arm64, locale en-RU) [334ms]
    • Flutter version 3.35.1 on channel stable at /Users/x/fvm/versions/3.35.1
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 20f8274939 (8 days ago), 2025-08-14 10:53:09 -0700
    • Engine revision 1e9a811bf8
    • Dart version 3.9.0
    • DevTools version 2.48.0
    • Feature flags: enable-web, enable-linux-desktop, enable-macos-desktop, enable-windows-desktop, enable-android, enable-ios, cli-animations, enable-lldb-debugging

[✓] Android toolchain - develop for Android devices (Android SDK version 35.0.0) [1,087ms]
    • Android SDK at /Users/x/Library/Android/sdk/
    • Emulator version 35.2.10.0 (build_id 12414864) (CL:N/A)
    • Platform android-35, build-tools 35.0.0
    • ANDROID_HOME = /Users/x/Library/Android/sdk/
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
      This is the JDK bundled with the latest Android Studio installation on this machine.
      To manually set the JDK path, use: `flutter config --jdk-dir="path/to/jdk"`.
    • Java version OpenJDK Runtime Environment (build 21.0.3+-79915917-b509.11)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 16.4) [685ms]
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 16F6
    • CocoaPods version 1.16.2

[✓] Chrome - develop for the web [103ms]
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2024.2) [103ms]
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 21.0.3+-79915917-b509.11)

[✓] IntelliJ IDEA Ultimate Edition (version 2025.1.3) [102ms]
    • IntelliJ at /Applications/IntelliJ IDEA.app
    • Flutter plugin version 87.0
    • Dart plugin version 251.27623.5

[✓] IntelliJ IDEA Community Edition (version 2024.3.6) [23ms]
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] VS Code (version 1.102.2) [6ms]
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension can be installed from:
      🔨 https://marketplace.visualstudio.com/items?itemName=Dart-Code.flutter

[✓] Connected device (3 available) [5.7s]
    • iPad Pro 11-inch (M4) (mobile) • 19A2CC4F-464F-4482-87E2-0B29AEED90A4 • ios            • com.apple.CoreSimulator.SimRuntime.iOS-18-6 (simulator)
    • macOS (desktop)                • macos                                • darwin-arm64   • macOS 15.6 24G84 darwin-arm64
    • Chrome (web)                   • chrome                               • web-javascript • Google Chrome 139.0.7258.139

[✓] Network resources [682ms]
    • All expected network resources are available.

• No issues found!

@RustamG

Clone the repo.

@RustamG I am unable to access the repo. Maybe it is private.

The test process doesn't exit having the following output:

<< [ERROR:flutter/shell/testing/tester_main.cc(682)] Unhandled exception
<<< Exception: Golden "goldens/failing_counter.png": Pixel test failed, 0.11%, 4496px diff detected.
<<< Failure feedback can be found at /Users/x/Documents/Flutter/golden_tests/integration_test/E2Es/failures
<<< Stack trace: #0      LocalFileComparator.compare (package:flutter_test/src/_goldens_io.dart:108:5)
<<< <asynchronous suspension>
<<< #1      main (file:///var/folders/6n/2pnrqh45205_wq6kvgs8xmq00000gn/T/flutter_tools.40hhzs/flutter_web_platform.TmZB1C/listenerHyn8yD/listener.dart:29:26)
<<< <asynchronous suspension>

Based on this error log, please check if your case is same as #92590 or not.

@darshankawar

@darshankawar My bad. Just made it public.

As for #92590 - it is different. In my case I made the golden image different intentionally to demonstrate another issue - that the app and test don't stop after failure.

Thanks for the update. I was able to replicate the reported behavior on latest sdk versions getting same error.

stable : 3.35.1
master : 3.36.0-1.0.pre-211

Fix bug in test_golden_comparator, add an e2e test. (#174459)

Fix bug in test_golden_comparator, add an e2e test. (flutter#174459)

@matanlurey

@matanlurey
Thank you for the fast work! Much appreciated.

Fix bug in test_golden_comparator, add an e2e test. (flutter#174459)

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

Fix bug in test_golden_comparator, add an e2e test. (flutter#174459)

Fix bug in test_golden_comparator, add an e2e test. (#174459)

Navigation Menu

Repository navigation

The app doesn't stop after failing golden test. + Error in TestGoldenComparatorProcess #174267

Steps to reproduce

Expected results

Actual results

Code sample

Screenshots or Video

Logs

Flutter Doctor output

7 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

The app doesn't stop after failing golden test. + Error in TestGoldenComparatorProcess #174267

Description

Steps to reproduce

Expected results

Actual results

Code sample

Screenshots or Video

Logs

Flutter Doctor output

Activity

darshankawar commented on Aug 25, 2025

RustamG commented on Aug 25, 2025

darshankawar commented on Aug 25, 2025

7 remaining items

RustamG commented on Aug 27, 2025

github-actions commented on Sep 10, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions