Repository navigation
"Test: Run All Tests" command does not run all tests #3594
Description
Describe the bug
VSCode Test Runner doesn't run all tests, some are left unrun.
To Reproduce
Steps to reproduce the behavior:
- Go to the testing tab
- Click on the button that runs all tests (Run tests)
Expected behavior
All tests should run
Screenshots
Added
Versions (please complete the following information):
- VS Code version: 1.60.2
- Dart extension version: v3.27.0
- Dart/Flutter SDK version:
- Flutter (Channel stable, 2.5.1, on macOS 11.6 20G165 darwin-x64, locale en-CA)
Dart SDK version: 2.14.2 (stable) (Wed Sep 15 12:32:06 2021 +0200) on "macos_x64
"
Activity
lloydrichards commentedon Oct 6, 2021
Im getting the same issue when using the
Dart: Run All TestsWhat I've found is that you now have to use the VSCode test command
Test: Run All Testsbut this is now much much slower as it seems to be launching each time for each test file. For my TDD with over 60 test files this is now a really painful experience. Hopefully this is just a bug and the intended DX is still quick and responsive.old
Dart: Run All Tests-> 2-3secnew
Test: Run All Tests-> 40-60sec[-]VSCode Test Runner doesn't run all tests[/-][+]"Test: Run All Tests" command does not run all tests[/+]DanTup commentedon Oct 6, 2021
Thanks for the report! I overlooked the Test: Run All Tests command. I suspect it's running all discovered tests and therefore only runs those already visible in the tree. I think it may be fixed once #3591 is implemented, but I'll keep this open to track it to be sure.
The reason for Test: Run All Tests being slower might be that VS Code is handing us the full list of tests and they're being run one-at-a-time (or at least, one-file-at-a-time) whereas Dart: Run All Tests might be using a single test process (eg. just running
dart test test). I've opened #3595 to look into that. Thanks!johnnypcc commentedon Oct 6, 2021
@DanTup You're welcome. But even after it discovers all tests, and I try to run them all, they're still not run.
johnnypcc commentedon Oct 6, 2021
Yeah, it's much much slower :(
DanTup commentedon Oct 6, 2021
What's the relative path to one of your test files from the workspace folder? Eg. if you open a test file and find it in the Explorer tree, then right-click it and choose "Copy relative path", what does that look like? Is it outside of the
testfolder, or is thetestfolder not in the root of the folder opened in VS Code?Can you reproduce it in a newly created project?
As a short-term workaround, you can still use the Dart: Run All Tests command for now which should behave the same as before the update (it should find all tests and be faster). The fix in #3595 will make the "Run Tests" button in the new VS Code runner the same speed as long as you're running all tests (eg. haven't hidden any) and are running without debugging. When debugging, or selectively picking tests (eg. excluding some), we need to run them by file, so that will be slower.
johnnypcc commentedon Oct 6, 2021
All my tests, those which were run and those that weren't run, sit together under the test dir, which itself is in the root dir.
Thanks
DanTup commentedon Oct 7, 2021
@johnnypcc is it the same tests that are skipped each time? Are you able to make a small sample project that triggers the issue that you can share?
A log file might be useful too (note: it will include information about your tests, so best to capture in a project that's not sensitive):
You can send the log to me at [email protected] (do not send anything sensitive), or you could try searching through it for some of the filenames that did not run, to see if there's anything obvious there.
DanTup commentedon Oct 7, 2021
I was able to reproduce this and figured out what it is. It happens when we run multiple test suites sequentially - the first one "ends" the test run and subsequent test results are not reported.
I have a fix for this, though I've also improved things so many test runs can run in a single debug session (which will make them much faster - #3599).
I may patch the first fix into a hotfix release, though the other one is tied in with some larger changes and will be in the next major version.
Ensure multi-debug-session test runs are not ended early
DanTup commentedon Oct 8, 2021
v3.27.1 has been published with the fix. Please upgrade (you can run the Extensions: Check for Updates command to force a check) and let me know if you're still having problems. Thanks!
lloydrichards commentedon Oct 12, 2021
thanks so much for the hotfixes, has gone back to the same speed of testing using
Dart: Run All Testswhich is much appreciated. Looking forward to the next release and the other fixes 👍DanTup commentedon Oct 12, 2021
Great, glad to hear! :-)