Repository navigation
fix(test_api): 0.5.0 breaks flutter test for projects that dont depend on flutter_test #1977
Description
After the release of version 0.5.0 of the test_api package, one may face a weird error message when running flutter test on a project that doesn't have a dependency on flutter_test (which comes from flutters SDK):
Failed to load "/path/to/test/something_test.dart": type 'Null' is not a subtype of type 'Object' in type cast
package:test_api RemoteListener.start.<fn>.<fn>
After adding this to the pubspec file, tests run nomally:
dependency_overrides:
test_api: 0.4.18
Additional context:
Minimal reproducible sample: https://github.com/renancaraujo/test_api_issue
Flutter doctor:
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.7.8, on macOS 12.6 21G115 darwin-arm64, locale en-PT)
[✓] Android toolchain - develop for Android devices (Android SDK version 32.1.0-rc1)
[✓] Xcode - develop for iOS and macOS (Xcode 14.0.1)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2022.1)
[✓] VS Code (version 1.67.0)
[✓] Connected device (2 available)
[✓] HTTP Host Availability
• No issues found!
flutter packages get output:
Running "flutter pub get" in something...
Resolving dependencies... (1.3s)
+ _fe_analyzer_shared 57.0.0
+ analyzer 5.9.0
+ args 2.4.0
+ async 2.11.0
+ boolean_selector 2.1.1
+ collection 1.17.1
+ convert 3.1.1
+ coverage 1.6.3
+ crypto 3.0.2
+ file 6.1.4
+ frontend_server_client 3.2.0
+ glob 2.1.1
+ http_multi_server 3.2.1
+ http_parser 4.0.2
+ io 1.0.4
+ js 0.6.7
+ logging 1.1.1
+ matcher 0.12.15
+ meta 1.9.1
+ mime 1.0.4
+ mocktail 0.3.0
+ node_preamble 2.0.2
+ package_config 2.1.0
+ path 1.8.3
+ pool 1.5.1
+ pub_semver 2.1.3
+ shelf 1.4.0
+ shelf_packages_handler 3.0.1
+ shelf_static 1.1.1
+ shelf_web_socket 1.0.3
+ source_map_stack_trace 2.1.1
+ source_maps 0.10.12
+ source_span 1.9.1
+ stack_trace 1.11.0
+ stream_channel 2.1.1
+ string_scanner 1.2.0
+ term_glyph 1.2.1
+ test 1.24.0
+ test_api 0.5.0
+ test_core 0.5.0
+ typed_data 1.3.1
+ very_good_analysis 3.1.0 (4.0.0+1 available)
+ vm_service 11.3.0
+ watcher 1.0.2
+ web_socket_channel 2.3.0
+ webkit_inspection_protocol 1.2.0
+ yaml 3.1.1
Changed 47 dependencies!
Workrounds 🧠🔨
-
Stop running
flutter teston projects that don't depend onflutter_test. See: Theflutter testcommand should require aflutter_testortest_apidependency (and check for it). flutter/flutter#91107 -
Pin
test_apito0.4.18:
dev_dependencies:
test_api: 0.4.18
Activity
jakemac53 commentedon Mar 23, 2023
Can you provide a full repro example of your setup? Are you trying to run flutter tests or Dart tests from flutter?
jakemac53 commentedon Mar 23, 2023
Some additional info that would be useful:
flutter --versionoutputflutter pub get).renancaraujo commentedon Mar 23, 2023
@jakemac53 added some context to the issue and a reproducible sample.
jakemac53 commentedon Mar 23, 2023
Can you run this test fine with
dart test?renancaraujo commentedon Mar 23, 2023
Yes.
jakemac53 commentedon Mar 23, 2023
cc @natebosch I think probably something changed in the communication channel between the test runner and tests, and without the
flutter_testdependency then the pinned version oftest_apidoesn't take effect, and ultimatelyflutter testis not compatible.jakemac53 commentedon Mar 23, 2023
I have no idea how we want to deal with this type of situation though... it is essentially unsupported that
flutter testworks for non-flutter tests, but I think it is common in practice to do, and it usually works.jakemac53 commentedon Mar 23, 2023
I think that probably the best thing we can recommend is that if you want to run
flutter testyou need to depend onflutter_test, even if you don't actually import it, because you are relying on the flutter test runner, which does require certain versions of the test_* packages.renancaraujo commentedon Mar 23, 2023
I agree, I thought of opening an issue to let people be advised of this problem if they face it in the near future.
jakemac53 commentedon Mar 23, 2023
Happy to leave this open for a bit so people can hopefully discover it and find the workaround.
It might be helpful to that end if you add a "Workarounds" section to the top comment, suggesting either pinning test_api (note I would suggest doing this via regular
dev_dependenciesand notdependency_overrides), adding a dependency onflutter_test, or usingdart testinstead.In general the
flutter_testdependency is preferable - it should be harmless and is more future proof.jakemac53 commentedon Mar 23, 2023
We should also consider filing an issue on the flutter repo, for
flutter testto warn if you don't have that dependency, even if you aren't running flutter tests.jakemac53 commentedon Mar 23, 2023
haha, I already filed this a long time ago :( flutter/flutter#91107
renancaraujo commentedon Mar 23, 2023
Added that workarounds section 👍🏾
natebosch commentedon Mar 23, 2023
Yes, specifically the new
'compiler'field is missing from theflutter testusage.de40c1c#diff-37c08d44d79fe33f695f7173107a674384e7fae4490ce3b14cda343d69d99490R55
I agree that there isn't anything we can do about this from the
test_apilibrary side - the tool needs to have this knowledge and act on it.renancaraujo commentedon Mar 23, 2023
Thanks for the immediate replies
[native_dio_adapter] Fix CI (#1762)
jakemac53 commentedon Mar 27, 2023
Marking this as
needs-infojust so that it automatically gets closed in a little while (I think a couple weeks to a month). It doesn't actually need info though, haha.DanTup commentedon Mar 30, 2023
If this is common and the error comes from
package:test_api, could it check for thenulland throw clearer message? ("Field x is missing from foo, perhaps you're missing a dependency on flutter_test/test_api or should use 'dart test'?")natebosch commentedon Mar 31, 2023
I published a new version of
test_apiwhich is backwards compatible for this field in the communication between tests and the runner. It looks likeflutter testis able to run these tests again with the current published version.We should still add a check in
flutter testthat the project has a dependency onflutter_test, and packages should still prefer to usedart testwhen they don't have that dependency, but for the current package combinations it should work. I'll close this issue now that most users won't need a workaround.Remove test_api pin now that dart-lang/test#1977 has been resolved
Remove test_api pin now that dart-lang/test#1977 has been resolved
alaatrablse commentedon Aug 11, 2023
Thanks, it was a tough problem
eliasyishak commentedon Dec 4, 2023
Hey @natebosch, I'm trying to look into fixing this issue by checking for the
flutter_testdependency, but with your fix to make it backwards compatible, i'm having a hard time reproducing this error. Do you know how I can setup a simple repro to debug my fix?natebosch commentedon Dec 7, 2023
I don't think we should need to see the original error to verify the fix - even when the package versions are aligned in the pub solve and the test would pass I think we should validate the state of
pubspec.yaml.[native_dio_adapter] Fix CI (#1762)
[native_dio_adapter] Fix CI (cfug#1762)