Repository navigation
Inconsistent PNG encoding by Image.toByteData() causes false positives in golden file tests #30036
Description
Problem
I have a situation where matchesGoldenFile tests fail when golden files are created on one machine and the tests are run on another. Both are Linux machines with the same Flutter and Dart versions.
Cause
I tracked it down to this line:
This seems to produce a PNG presentation that differs bytewise on different machines even though the images are identical pixelwise.
The matchesGoldenFile tests compare the PNG images bytewise and this causes the test to fail even with pixelwise identical files.
flutter/packages/flutter_test/lib/src/goldens.dart
Lines 178 to 185 in bafe7cb
To reproduce
I created this short code that creates an image and prints its raw hash and png hash.
import 'dart:typed_data';
import 'dart:ui';
import 'package:collection/collection.dart';
void main() async {
final pictureRecorder = PictureRecorder();
Canvas(pictureRecorder).drawCircle(
const Offset(100, 100),
100,
Paint()..color = const Color.fromARGB(255, 255, 0, 0),
);
final picture = pictureRecorder.endRecording();
final image = await picture.toImage(200, 200);
final hashFunc = const ListEquality().hash;
final pngByteData = await image.toByteData(format: ImageByteFormat.png);
final pngHash = hashFunc(Uint8List.view(pngByteData.buffer));
final rawByteData = await image.toByteData(format: ImageByteFormat.rawRgba);
final rawHash = hashFunc(Uint8List.view(rawByteData.buffer));
print('png hash: $pngHash');
print('raw hash: $rawHash');
}Test runs
Machine 1:
❯ flutter test image_test.dart
00:01 +0: loading /home/antti/projects/correct_pitch/image_test.dart
png hash: 1951256462
raw hash: 1641621570
No tests ran.
Machine 2:
❯ flutter test image_test.dart
00:01 +0: loading /home/antti/projects/correct_pitch/image_test.dart
png hash: 1038304024
raw hash: 1641621570
No tests ran.
As you can see, the raw hashes are identical, i.e. the images are identical pixelwise. But the PNG hashes differ, i.e. the bytes in the PNG presentation are not consistent.
flutter doctor -v
Machine 1:
[✓] Flutter (Channel beta, v1.3.8, on Linux, locale en_US.UTF-8)
• Flutter version 1.3.8 at /home/antti/tools/flutter
• Framework revision e5b1ed7a7f (3 weeks ago), 2019-03-06 14:23:37 -0800
• Engine revision f4951df193
• Dart version 2.2.1 (build 2.2.1-dev.0.0 571ea80e11)
[✓] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
• Android SDK at /opt/android-sdk
• Android NDK location not configured (optional; useful for native profiling support)
• Platform android-28, build-tools 28.0.3
• ANDROID_HOME = /opt/android-sdk
• Java binary at: /opt/android-studio/jre/bin/java
• Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1248-b01)
• All Android licenses accepted.
[✓] Android Studio (version 3.3)
• Android Studio at /opt/android-studio
• Flutter plugin version 33.3.1
• Dart plugin version 182.5215
• Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1248-b01)
[!] Connected device
! No devices available
! Doctor found issues in 1 category.
Machine 2:
[✓] Flutter (Channel beta, v1.3.8, on Linux, locale en_US.UTF-8)
• Flutter version 1.3.8 at /home/antti/tools/flutter
• Framework revision e5b1ed7a7f (3 weeks ago), 2019-03-06 14:23:37 -0800
• Engine revision f4951df193
• Dart version 2.2.1 (build 2.2.1-dev.0.0 571ea80e11)
[✓] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
• Android SDK at /home/antti/Android/Sdk
• Android NDK location not configured (optional; useful for native profiling support)
• Platform android-28, build-tools 28.0.3
• ANDROID_HOME = /home/antti/Android/Sdk
• Java binary at: /usr/bin/java
• Java version OpenJDK Runtime Environment (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.4)
• All Android licenses accepted.
[!] Android Studio (not installed)
• Android Studio not found; download from https://developer.android.com/studio/index.html
(or visit https://flutter.io/setup/#android-setup for detailed instructions).
[✓] VS Code (version 1.32.3)
• VS Code at /usr/share/code
• Flutter extension version 2.24.0
[!] Connected device
! No devices available
! Doctor found issues in 2 categories.
Activity
apaatsio commentedon Mar 27, 2019
Here's a real-life example of golden files that are produced by two different computers. The pixels are identical but bytes are not.
Both images are created with
flutter test --update-goldenson the exact same code base with the exact same Flutter and Dart versions as described in the original bug report.Note how the md5sums differ but the images are identical according to the ImageMagick signature.
| golden file |
|
|
|---|---|---|
|machine|machine 1| machine 2|
| md5sum |
810108768907135108c0ba33c0271b3f|017805ffb21c33e85845412900913a89|| ImageMagick signature (
identify -format "%#") |68f86f32fccacc0f9a270a0a90d2ff26 ce0af5ad1b59f1c7ae39d3e93b10e052|68f86f32fccacc0f9a270a0a90d2ff26 ce0af5ad1b59f1c7ae39d3e93b10e052|tp commentedon Jun 19, 2019
We also run into this constantly, where images change but are visually the same.
Though personally I see it also happening on one machine, where the same test produces different outputs, even though I am unable to spot any difference with tooling.
(I don't have a small repro, but could send the files if anyone wants to take a look at the output.)
Edit: Using
magick identify -format "%#" filenameI do get 2 different hashes, even though visually no tool would show a single pixel of difference:bdc3540f6f5ede578979101d7091ecb07003d769a732642b0e13d4f427cb7a47
0a4b730d76d35d51d06226a94122f218a517a47e05e972faee3e8ae27a48d3e
apaatsio commentedon Oct 7, 2019
Closing this issue because by all logic this has been fixed by #38473. Although I can't verify because I don't have access to the original environment anymore to reproduce the issue.
github-actions commentedon Aug 27, 2021
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.