Repository navigation
Test Runner Viewlet with an Extension API to plug into [Enhancement Request] #9505
Description
In response to Erich Gamma's tweet Krzysztof Cieślak (@Krzysztof-Cieslak)
A test runner viewlet/sidebar with a pluggable API would be a great addition to vscode.
The viewlet should include visual indications of
- the progress of completed tests out of the set of tests being run
- visual indicators or icons showing basic tests states at the least
not run | success | failure | ignored/skipped
The test runner should be able to provide a set of sorting categories and repopulate the viewlet based on the users selection
A query box with the ability to filter the tests based on the names shown in the runner's list at a minimum, with the ability for the underlying runner to populate a set of query filters.
Instead of the VS style of collapsing categories
I think the vscode style used in the debugger viewlet is much nicer 😄
A button could be used to select which test runner you want to use out of the test runner extensions that have been installed
But a dropdown menu would work too
A side button to open a menu could enable the active test runner to populate it with any additional commands specific to its functionality
Some other functionality points -
- When an individual test is selected the result could be rendered in the output panel
- An icon/button to run all tests
- The ability to highlight/select a set of tests and only run those tests
- A basic set of run options, which individual runners might be able to extend
- Tests outputs can contain links to specific points in source code
- Debugging specific tests (would be awesome, but understandable if this is too difficult for an initial implementation)
- Associate test runner extensions with specific languages?
- Test playlists (would be nice, but not especially important)
- Ability to specify a task/trigger that will repopulate the viewlet's list of tests
- Dropdown menu that the test runner can population with platform/environment/runtime configurations to use during test execution.
Activity
Krzysztof-Cieslak commentedon Jul 19, 2016
Few ideas, all just IMHO, may be totally wrong ;)
I believe that test adapters should have their separate extension point just like debuggers.
Also creating nice, lightweight UI (as described above) would be really great, there is enough place on left panel for one more button ;)
From the API point of view I think we would need:
Buttons to run all tests / selected tests / single test / previously not passed tests etc. would all execute function 3. just with appropriate list of tests based on chosen command, selection in UI, and previous result category.
Basic run options could be easily handled by VSCode (as I've described earlier). Getting API for providing custom options / list could be nice but it's definitely not necessary for initial implementation
It's possible to execute start debugging using
vscode.startDebugit's just matter of defining which debugger should be started and with what configurationOpen question, should it be defined by test adapter creator or by end user?
I'd say so - probably just for sake of lazy extension activation. If language supported by test adapter appears in current workspace active test runner.
That's actually really interesting case, for most APIs right now VSCode decide when to recalculate things and call implemented API again.
I'd like to see something similar to debug case where we have dropdown to select different debugging configurations - different configs specified in
launch.json(in case of debuggers, we could have similar file for test adapters)et1975 commentedon Jul 20, 2016
As an alternative, I find TestDriven's unit-testing experience much better than that of built-in VS or as offered by R#. Personally, I don't need a separate test browser, nor a new output window. I'm already looking at the code and if the plugin can receive which line the cursor is on then it can tell wether I want to run the whole suite or the specific test. For everything else there's always a task and since it can also write to the output window the transition is seamless.
egamma commentedon Jul 26, 2016
Thanks for the input and it is great to see the different perspectives.
First, test running is an
inner development loopactivity and is important and fits into Code. However, I was hoping for something more lightweight than a full test explorer. Many test runners like mocha work fine on the command line in the console. Also many of them support continuous test running by watching the files. Something we would not want to replicate.Rather than starting with the maximum number of features that could be added say to a test explorer, what would be the minimum features to still provide a good developer experience. e.g., navigate from failure traces to the code, seeing an indication of the number of the test failures.
Consider the test runners you are using today, would a command line version for them be sufficient?
cloudRoutine commentedon Jul 26, 2016
Erich Gamma (@egamma) I wanted to make a strong opening bid 😉
At a minimum level - navigation to code, the ability to select a subset of tests to run, simple text match based filtering, and showing which tests succeeded, failed, and weren't run.
Krzysztof-Cieslak commentedon Jul 26, 2016
I guess I would like to see following set of feature:
Also, having test viewlet, wouldn't break current command line based workflows, so that's just added value 😉
egamma commentedon Oct 18, 2016
Just adding how some extensions are now supporting unit testing.
alexandrudima fyi
Python
Go
Test at cursor
Test File
Test Package
Test Previous
Generate test skeleton at cursor/file/package
Test coverage in current package
cloudRoutine commentedon Oct 18, 2016
Erich Gamma (@egamma) if several extensions are starting to implement a common set of features to provide functionality that users clearly want, all of which falls within a set of reoccurring abstract constructs, it seems like the perfect opportunity for vscode to provide a standard API to facilitate this so every language extension doesn't need to roll a bespoke implementation 😸
egamma commentedon Oct 19, 2016
Jared Hester (@cloudRoutine) yes, this is exactly the plan.
samuelkdavis commentedon Dec 7, 2016
Any update on this..?
rmunn commentedon Jun 20, 2017
This is the feature that would be the biggest improvement for me in my VS Code experience. Right now I'm running tests with a command-line runner and manually finding the failing test by a Ctrl+F search on the test names. Having a way to jump to the failing test, and (as a bonus) to see the console output of that specific test in the test output window, would improve my VS Code experience quite a bit.
So I'll echo the "Any update on this?" request from last December. This is my most-wished-for feature in VS Code right now, and I think testing is worthy of its own dedicated icon on the Activity Bar, with shortcut Ctrl/Cmd + Shift + T. (Or fallback shortcut Ctrl/Cmd + Shift + U, for "Unit testing", if Ctrl/Cmd + Shift + T is already taken).
forki commentedon Jun 20, 2017
Please please please
12 remaining items
uxsoft commentedon Mar 17, 2018
Hi, we need this!
The existing extensions are a desperation attempt to fix something you should have fixed a long time ago. There needs to be a unified place where I can see all the tests from different plugins for test frameworks. Then a test framework maintainer can just provide a plugin and the test Tab could show an aggregation. Now there is no place where I can see both .net and js tests at the same place. We are definitely not happy with them. This feature seems like a no-brainer and not too difficult to engineer.
rmunn commentedon Mar 26, 2018
If as Erich Gamma (@egamma) points out, there's a lot of variation between the different test runners, then it might make some sense to hold off on implementing this API for a while.
However, I'd like to request one feature be implemented now, at least: adding a new sidebar view (and icon) for unit testing extensions to use. Currently the API only allows extensions to add tree views to the explorer or debug sidebars, which means that unit test extensions like https://github.com/gfrancischini/vscode-unit-test have to either use the debug sidebar (which doesn't feel appropriate) or the explorer sidebar (which is already overloaded, with three panels — open editors, folder view, and bookmarks — opened by default, and that's not counting any other panels that other extensions might add). I'd much rather see a sidebar view, with its own icon, made available for unit test extensions to use. Here's a mockup of what it could look like:
The icon could, of course, be anything: I just picked something that seemed appropriate (in this case, an icon from https://icons8.com/icon/6258/checked with a free license).
As someone else has already mentioned in the thread, unit testing is one of the things that feels fundamental to the concept of "a text editor for programmers". I know you want to avoid sidebar-icon proliferation, but unit testing is one of the things that's so core to the coding experience that I believe it truly deserves its own sidebar icon.
General question: would you prefer me to open a new issue to track this "new sidebar icon for unit testing" idea, or should it be tracked here as a subset of this issue?
rmunn commentedon Mar 26, 2018
#43645 appears to be related to my request for a new sidebar icon for test runner extensions to use.
uxsoft commentedon Mar 26, 2018
Sounds like a good start
DanTup commentedon Mar 26, 2018
Let's not confuse variation with inconsistency. Currently people are left to implement it themselves with no guidelines, so of course they'll all be different.
Visual Studio has a Test window which, while not perfect, has served many languages and frameworks reasonably well. VS Code has a good process for suggesting (and later implementing) proposed APIs to get feedback. I don't see why this wouldn't work well here. Propose something, and let all the people that want this (of which there are many) provide feedback if it won't fit their needs.
hbenl commentedon May 20, 2018
FYI: I have just published a Test Explorer extension that is trying to fill this gap. This extension can be used by other extensions (called Test Adapters) to show and run the tests for a particular testing framework. I have also published a Test Adapter extension for Mocha and I'm working on another one for Jasmine.
The feature set of these extensions is currently fairly basic but I intend to add more features in the coming months and hope that the community will add more adapters.
luabud commentedon Dec 27, 2019
Agreed, and I also find useful to have a "show output" option.
Kradirhamik commentedon Jul 23, 2020
Will there be more news on this please?
connor4312 commentedon Sep 25, 2020
We are starting to investigate this here 👉 #107467
connor4312 commentedon Jan 13, 2021
/duplicate of the linked issue
vscode-triage-bot commentedon Jan 13, 2021
Thanks for creating this issue! We figured it's covering the same as another one we already have. Thus, we closed this one as a duplicate. You can search for existing issues here. See also our issue reporting guidelines.
Happy Coding!