Repository navigation
Supress timeout when debugging unit tests. #3687
Description
I think this will be problematic to solve and not entirely with-in dart-codes control but thought I would start the conversation somewhere.
When debugging unit tests with the test package it has a default timeout of 30 seconds.
The problem is that when you are debugging through unit tests you inevitably exceed the 30 second limit.
The result is that to debug a unit test you need to add the @timeout to every unit test which is annoying.
This has a second consequence in that I invariably forget to remove it so the timeout really no longer serves its intended purpose.
So the questions is whether there is someway of supressing the test package's timeout when debugging?
I guess this might be something that needs to be co-ordinated between the test package and vs-code.
Is it even possible for code to detect that it is being debugged?
Edit: I had a google and there is a kDebugMode available in flutter but I would like this to work for dart not just flutter apps.
Activity
DanTup commentedon Nov 24, 2021
There's a flag you can pass to
package:testto override the timeout (--timeout 1h). This could be added in a custom codelens link (either overriding the Run/Debug links, or as an extra link "Run w/o Timeout" etc.).I don't think it should just happen automatically when debugging, as you might just be running with
F5because it's easy, and not really debugging.Perhaps when moving the test codelens into Profiles (#3607) we could add some additional default profiles (like run without timeout, run with coverage, etc.) that would be too spammy in CodeLens, but less so in a drop-down menu? 🤔
DanTup commentedon Nov 24, 2021
I'm not making changes to use Profiles (#3607) as noted above, as the experience wasn't as good as I hoped.
But I think there are still two reasonable solutions to this:
dart.testAdditionalArgs(Support passing args to Dart VM when running tests #3685) at the User or Workspace level to add["--timeout", "1h"]to the test invocationscodeLensfield in launch.json to override the Run/Debug links to have these argsInterested in your opinions on them.
DanTup commentedon Nov 24, 2021
There's a beta of the next release with
dart.testAdditionalArgsat https://github.com/Dart-Code/Dart-Code/releases/tag/v3.29.0-beta.1 if you want to try it out.bsutton commentedon Nov 24, 2021
So a few of thoughts.
I think it would be reasonable to have the debug link (code lens?) automatically disable the timeouts.
Whilst I do occasionally click the debug link to run a set of tests, this is usually by accident as I will normally use the run link or the test flask icon.
For the times that I accidently click the debug button, I'm standing watching the code run so I can act as the time keeper.
If the code is running for too long I will 'try' to pause the code (this doesn't always work but that is another issue) and diagnose the problem (so again I probably don't want the timeout anyway).
So my thoughts are that defaulting the override of timeout on the debug link would provide the best user experience.
I believe that worrying about 2) will result in an reduced overall experience.
Perhaps there could be a config option to disable the behaviour if people really hate but I wouldn't do that until someone complains about it.
So I did some tests using the timeout option in dart.testAdditionalArgs.
I should note that I edited the via the settings gui.
To make it work I had to enter -timeout=1h.
The test I ran already had a @timeout annotation.
It appears that even with the setting the @timeout notation takes precedence.
This is not what I was expecting (but I missed this fact in my original requirements statement).
My preference is that when using the debug link both the default timeout and any library specific timeout are ignored.
When I remove my library @timeout the change appears to work as expected :)
Thanks for the fast turn around on this, I really didn't expect it.
DanTup commentedon Nov 25, 2021
I'm not sure I'd want to assume this always holds true for everyone. I often run with debugging (
F5) and go AFK to make a drink, and wouldn't expect to come back to a set of passing tests if they were actually taking longer than the timeouts. I do think a setting to control this might be reasonable though (and I don't think I'm totally against having it default to on), although doing it with an arbitrary value like 1hr doesn't seem ideal.Yeah, that's the behaviour I'd expect. This is
package:testfunctionality so can't be controlled here. Although if you use factors in timeouts, like@Timeout.factor(2)then I would expect it to be double whatever the default is (so without the flag, 2x 30 seconds, and with the flag, 2x 1 hour). If you need hard-coded timeouts, perhaps using that is a better way.I've filed dart-lang/test#1634 to see if we could get a
--ignore-timeoutsflag or similar that we could use instead that would work better.DanTup commentedon Nov 25, 2021
I've added a new setting:
dart.suppressTestTimeouts. It's an enum ofnever,always,debug. It will inject--timeout 1dinto the toolArgs (although note I found that Flutter test seems to already disable timeouts when running in debug mode, so it's not really doing much there - expect perhaps shortening it from unlimited to 1d).For now, the default is
neverbecause I'd like to ship this as opt-in. I'll consider changing it todebugin a future release.DanTup commentedon Nov 25, 2021
Beta release here -> https://github.com/Dart-Code/Dart-Code/releases/tag/v3.29.0-beta.2
DanTup commentedon Nov 30, 2021
I've opened #3699 about switching to
package:tests's new--ignore-timeoutsflag if the current version oftestsupports it, which will resolve the issue with explicitly timeouts not being ignored by the current implementation.