Repository navigation
The "Test: Run All Tests" command is much slower than "Dart: Run All Tests" #3595
Description
What I've found is that you now have to use the VSCode test command Test: Run All Tests but this is now much much slower as it seems to be launching each time for each test file. For my TDD with over 60 test files this is now a really painful experience. Hopefully this is just a bug and the intended DX is still quick and responsive.
old Dart: Run All Tests -> 2-3sec
new Test: Run All Tests -> 40-60sec
Originally posted by @lloydrichards in #3594 (comment)
Activity
lloydrichards commentedon Oct 6, 2021
Excellent! Thanks so much for the quick response. Will this be a hot fix anytime soon or will we need to wait til v3.28.0 in Nov?
It really slows down my TDD today so if there was a quick workaround or should I just downgrade back to use the old system?
DanTup commentedon Oct 6, 2021
There have been quite a few changes in the test runner with this fix so it'll probably only ship in the next main release. However, I can do a preview build once I've implemented a few more of these test changes (things like automatic discovery of test files/tests without having to open the files) if you're interested in using that (it would be good to have additional eyes testing it).
In the meantime, you can switch to the old test runner without downgrading the whole extension with the
useVsCodeTestRunnersetting. Running the tests from the command palette/CodeLens/etc. should be unaffected too - my understanding is that it's only the Run Tests button at the top of the VS Code test runner that's doing the slow thing.lloydrichards commentedon Oct 6, 2021
would definitely love to help!
DanTup commentedon Oct 26, 2021
There's a beta release of v3.28.0 here that includes all the most recent test fixes, and also automatic test discovery (the tree should populate when you load the project without needing to open/run tests first):
https://github.com/Dart-Code/Dart-Code/releases/tag/v3.28.0-beta.1DanTup commentedon Oct 27, 2021
There were some issues in that beta, there's a new one here:
https://github.com/Dart-Code/Dart-Code/releases/tag/v3.28.0-beta.2
lloydrichards commentedon Dec 10, 2021
I'm not sure what's been updated in the newest version but I'm running into some issues with tests again and figured I would ask in this thread first before making a new issue. Firstly is that the
Dart: Run All Testsseems to have been removed so now I need to instead useTest: Run All Testsbut when I try this, I get a message saying "Unable to find any test folders"Next, I go to the Testing tab and click Run All Tests and get this message: "Integration tests and unit tests cannot be run in a single invocation. Use separate invocations of
flutter testto run integration tests and unit tests."Next, I click Debug Tests and I get the same behaviour as before where it loads a new instance for every test suite causing the tests to run for 4-5min (I have a lot more tests now)
Finally, I go to the Debug tab, select Flutter Test from the dropdown and then Start Debugging, and I get the expected results from running
Dart: Run All Tests, but now when I pressf5to start the device, it instead runs my tests 😅I'm not sure what's going on with the Dart Testing development, but it would be nice to have a workflow that I can keep the same for at least a couple of months at a time. If there was maybe some documentation on how you expect tests to be run in TDD it would be helpful
DanTup commentedon Dec 10, 2021
@lloydrichards sorry for the bother. Moving from our own custom runner to the new built-in runner was quite a big job and introduced a few issues. I'm confident that once the remaining differences and edge case are resolved, things should be much less volatile.
You're right that this command was removed, and Test: Run All Tests is the expected command now. However, there was an issue if you hadn't previously loaded the test runner, VS Code doesn't seem ask us to discover the tests so we produced this error. I've fixed it in #3714 and we'll now automatically discover tests if there is any request to run tests (from anywhere) and we haven't already done so. It'll ship in the next version, but I'm also hoping to make a pre-release build (using VS Code's new functionality) next week that you could switch to.
This sounds like a limitation of
flutter testthat VS code isn't accounting for.Could you file a new issue about this and I'll ensure it's also fixed before the pre-release/next release.I've filed #3727 about this.I believe this has always been the case for debugging. IIRC, it's only when running without a debugger that we're able to run multiple suites in a single session.
I don't quite understand here. If you're selecing a debug configuration from the debig sidebar, pressing F5 will run that config, so if you select a test config and press F5, running tests is expected behaviour. Pressing F5 is the same as clicking the green Debug button at the top of the Debug side bar next to the debug configuration.
There are no real expectations. Things should generally work the way users expect them to. If anything is not working of confusing, it's always worth filing an issue and assuming it's something that can be improved rather than you're doing something wrong :-) I can't promise everything is fixable, but often things don't work the best way just because they haven't had a lot of feedback and were based on my own (very narrow) usage :-)
DanTup commentedon Dec 13, 2021
The fixes noted above (including a fix for running all tests when there are both unit+integration tests) were included in todays pre-release release. You can switch to this in VS Code if you'd like the fixes and/or to help test new features/fixes before they're included in a stable release:
lloydrichards commentedon Dec 13, 2021
thanks so much! will give a shot tomorrow at work :)
lloydrichards commentedon Dec 14, 2021
Have been testing out the Pre-Release and the clicking
Run Testsin the Testing tab does now work and quickly run all tests. however when I use the keyboard shortcut (ctrl + ; A) then I get the same behaviour where it runs each test group in its own debug instance and takes 15min to complete.Is there a different keyboard binding to run the tests from the Testing tab to the
Test: Run All Testsin the command palletDanTup commentedon Dec 14, 2021
@lloydrichards is it possible you're holding down
ctrlwhen you pressAtoo? I asked because Run All Tests and Debug All Tests have similar keybindings, and Debug will be much slower because we can't combine the runs:lloydrichards commentedon Dec 14, 2021
if I press Ctrl + ; A then i get:
and if I press Ctrl + ; Ctrl + A then I get each test as its own debug instance.
I can't find any keybinding that works to run what I click in the Testing tab
DanTup commentedon Dec 15, 2021
Is this different behaviour to running the Test: Run All Tests command? VS Code is handling the shortcut keys and should invoke the same command, so if those two things aren't behaving the same it seems like something is wrong in VS Code (assuming you haven't changed the keybindings).
Assuming you get the same behaviour when running the command too, the second message is perhaps correct - it's trying to run your integration tests and your selected device (in the status bar) is the Chrome device, but integration tests are not supported on web. You'll need to select another device for running the integration tests.
As for "The syntax of the command is incorrect.", that seems like a bug though I can't reproduce. Can you capture a log:
Ctrl + ;thenAto reproduce the issueYou can attach the log to a new issue in this repo describing that issue. Thanks!