Repository navigation
Running flutter test --update-goldens with an integration test causes FileSystemException #143299
Description
Steps to reproduce
- Add the
integration_testpackage to the project and create an integration test. - Add an
expectLaterwithmatchesGoldenFile('some_file.png')somewhere in the test. - Run
flutter test <path to the test> --update-goldenson an Android Emulator (I didn't test on iOS) - The integration test will run normally but will throw
FileSystemExceptionafter finishing and produce no screenshots.
Expected results
Images for the golden tests should be created and the test run should not throw any exceptions.
If I run the same test but as a widget test instead of integration test, the images get created correctly, but then there's the limitation of not being able to access any native APIs in the test, since it's not being run on a device/emulator.
Since matchesGoldenFile docs don't mention anything about it not working in an integration test environment, I'd presume it should work the same way in widget and integration tests.
Actual results
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following FileSystemException was thrown while running async test code:
Creation failed, path = '/Users' (OS Error: Read-only file system, errno = 30)
When the exception was thrown, this was the stack:
#0 _checkForErrorResponse (dart:io/common.dart:55:9)
#1 _Directory.create.<anonymous closure> (dart:io/directory_impl.dart:114:9)
<asynchronous suspension>
#7 _Directory.create.<anonymous closure>.<anonymous closure> (dart:io/directory_impl.dart:104:54)
<asynchronous suspension>
#8 _Directory.create.<anonymous closure>.<anonymous closure> (dart:io/directory_impl.dart:104:54)
<asynchronous suspension>
#9 _Directory.create.<anonymous closure>.<anonymous closure> (dart:io/directory_impl.dart:104:54)
<asynchronous suspension>
#10 _Directory.create.<anonymous closure>.<anonymous closure> (dart:io/directory_impl.dart:104:54)
<asynchronous suspension>
#11 _Directory.create.<anonymous closure>.<anonymous closure> (dart:io/directory_impl.dart:104:54)
<asynchronous suspension>
#12 _Directory.create.<anonymous closure>.<anonymous closure> (dart:io/directory_impl.dart:104:54)
<asynchronous suspension>
#13 _Directory.create.<anonymous closure>.<anonymous closure> (dart:io/directory_impl.dart:104:54)
<asynchronous suspension>
#14 LocalFileComparator.update (package:flutter_test/src/_goldens_io.dart:109:5)
<asynchronous suspension>
#15 MatchesGoldenFile.matchAsync.<anonymous closure> (package:flutter_test/src/_matchers_io.dart:114:11)
<asynchronous suspension>
#16 LiveTestWidgetsFlutterBinding.runAsync (package:flutter_test/src/binding.dart:1990:14)
<asynchronous suspension>
#17 _expect.<anonymous closure> (package:matcher/src/expect/expect.dart:123:26)
<asynchronous suspension>
<asynchronous suspension>
#19 expectLater.<anonymous closure> (package:flutter_test/src/widget_tester.dart:496:24)
<asynchronous suspension>
#20 main.setBudgetAndQuantityAndGoToCheckout (file:///Users/<username>/<path to the project>/integration_test/guest_blind_order_checkout_flow_test.dart:233:5)
<asynchronous suspension>
#21 main.<anonymous closure> (file:///Users/<username>/<path to the project>/integration_test/guest_blind_order_checkout_flow_test.dart:349:5)
<asynchronous suspension>
#22 testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:168:15)
<asynchronous suspension>
#23 TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:1013:5)
<asynchronous suspension>
<asynchronous suspension>
(elided 7 frames from dart:async and package:stack_trace)
════════════════════════════════════════════════════════════════════════════════════════════════════
Test failed. See exception logs above.
Code sample
Code sample
Any integration test containing `matchesGoldenFile` call.Screenshots or Video
No response
Logs
No response
Flutter Doctor output
Doctor output
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.16.9, on macOS 14.2.1 23C71 darwin-arm64, locale en-US)
[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
[✓] Xcode - develop for iOS and macOS (Xcode 15.2)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2023.1)
[✓] VS Code (version 1.86.0)
[✓] Connected device (5 available)
[✓] Network resources
• No issues found!
Metadata
Metadata
Assignees
Labels
Important issues not at the top of the work list"flutter test", flutter_test, or one of our testsThe flutter/packages/integration_test pluginFound to occur in 3.16Found to occur in 3.20The issue has been confirmed reproducible and is ready to work onAndroid applications specificallyOwned by Android platform teamTriaged by Android platform team
Activity
[-]Running `flutter test --update-goldens` in an integration test causes FileSystemException[/-][+]Running `flutter test --update-goldens` with an integration test causes FileSystemException[/+]dam-ease commentedon Feb 13, 2024
Hello @tbuczkowski.
Thanks for filing this.
From the logs, you seem to be hitting #94197. Can you check though to confirm this is similar/related to your issue?
tbuczkowski commentedon Feb 13, 2024
Hi @dam-ease. Yes, I think they are similar, but refer to different use cases. The issue you linked is more about manually loading some assets during integration tests and this one is about just running golden tests without explicitly accessing any files/assets. So I'm not sure if they qualify as separate issues or should be merged into one?
dam-ease commentedon Feb 13, 2024
Thanks for your response @tbuczkowski.
Could you please share a simple repo that we can run on our end to reproduce your own case?
tbuczkowski commentedon Feb 14, 2024
@dam-ease there you go: https://github.com/tbuczkowski/issue143299
dam-ease commentedon Feb 15, 2024
Thanks for your response @tbuczkowski
Running the sample above, the test runs infinitely on my end.
Can you share your full logs in a
.txtfile?tbuczkowski commentedon Feb 15, 2024
It's just the example test from https://docs.flutter.dev/cookbook/testing/integration/introduction with the line
await expectLater(fab, matchesGoldenFile('fab.png'));added. There's really nothing more to it than what I posted in the first message.dam-ease commentedon Feb 15, 2024
Thanks for your response @tbuczkowski.
I seem to have missed out on a step, I can reproduce this on the latest
masterandstablechannels.Tested this on iOS and couldn't reproduce, this seems to be specific to Android.
Logs
stable, master flutter doctor -v
9 remaining items
deborahdcarlo commentedon Feb 21, 2024
I am running into same problem as @tbuczkowski. Can't golden tests be used on an integration test? I do see the images getting generated if done on a widget test, but not on integration (which is frustrating). Any solution for this problem so far?
dam-ease commentedon Feb 22, 2024
There is an existing issue related to golden tests being used with integration tests #103222.
#103222 (comment) looks like a description the root cause of this issue.
camsim99 commentedon Jul 22, 2024
I ran into the read-only file system issue (
Creation failed, path = '/Users' (OS Error: Read-only file system, errno = 30)), as well, and dug into this a bit because I would like to write an integration test that uses the goldens logic.Seems like the root cause there is that the test is trying to write the golden files (like the images) to the actual device the test is running on, and it's trying to write in a directory (intended for the host machine) that is not writable on the device. I tried using the
path_providerplugin to get the directory that the application can write to (withgetApplicationDocumentsDirectory), and when I modified the Flutter goldens logic to write it here, I was able to see the golden files on the device. However, this also required that I delay the end of the test because the app is automatically deleted (along with its storage) after the test runs. I need to find a way to stop the uninstall after the test run realistically, but I was just trying to see what was going on.Ideally, we find a way to write the golden files to the host machine instead of the device, so I'm going to start looking into this.
matanlurey commentedon Jul 22, 2024
@johnmccutchan and I had a prototype, https://github.com/johnmccutchan/fgs, which did this: it's not in a state where we could land it, and probably needs additional design work and some vetting from platform folks.
I am not sure there is a smaller workaround than what you've already discovered (i.e. making the test aware of the environment). It might be helpful to understand what you are trying to test, and also whether this arc of work will help, or not.
Happy to chat further!
Implement golden-file matching for `integration_test` on Android and …
integration_test: Add gitignore of golden image (#161404)
Enable `goldenFileComparator` fix for on-device integration tests. (#…
github-actions commentedon Feb 27, 2025
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.