Repository navigation
Support passing args to Dart VM when running tests #3685
Description
Describe the bug
I am attempting to run tests in a project that uses the constructor-tearoffs experiment, and I'm getting errors while running the test from inside VSCode.
I have attempted to use both vmAdditionalArgs in the launch config for that test, as well as dart.vmAdditionalArgs at the workspace level, but neither are passed along to the test.
Example of launch.json:
{
"version": "0.2.0",
"configurations": [
{
"name": "server",
"program": "test/server_test.dart",
"request": "launch",
"type": "dart",
"vmAdditionalArgs": [
"--enable-experiment=constructor-tearoffs",
]
}
]
}Example of my .code-workspace:
{
"folders": [],
"settings": {
"dart.vmAdditionalArgs": [
"--enable-experiment=constructor-tearoffs"
],
}
}I have verified that the VM args do work as expected launching something other than a test.
I have also verified that dart test --enable-experiment=constructor-tearoffs works fine from the commandline.
To Reproduce
- Create a project that uses constructor tear-offs
- Write a test file that references the library containing tear-offs
- Attempt to run using methods described above
Expected behavior
That the tests run without analysis failure.
Versions (please complete the following information):
- VS Code version: v1.62.3
- Dart extension version: v3.28.0
- Dart/Flutter SDK version: v2.15.0-42.0.dev
Activity
DanTup commentedon Nov 24, 2021
vmAdditionalArgsare intended only for running command line apps, not tests. I think we should have something like adartTestAdditionalArgsthough (we already have a Flutter equiv). I'll tweak this issue to be for that.In the meantime, you might be able to use
toolArgsin yourlaunch.jsoninstead ofvmAdditionalArgsto get the same effect.FWIW, the
xxxAdditionalArgsoptions are all settings so can only go in User or Workspace settings, not directly in launch configurations.. in launch configurations, settingtoolArgsandargsdirectly is the right way. The settings are intended for when you want to set this for all instances, without needing to add it to lots of launch configurations (or when you're not usinglaunch.json).[-]Running tests does not include vmAdditionalArgs[/-][+]Add a "dartTestAdditionalArgs" setting[/+][-]Add a "dartTestAdditionalArgs" setting[/-][+]Add a "dart.testAdditionalArgs" setting[/+]shyndman commentedon Nov 24, 2021
toolArgsdoes not have the desired effect.DanTup commentedon Nov 24, 2021
@shyndman can you be more specific? What happens?
FWIW, I'm about to make a beta build of the next release which includes
dart.testAdditionalArgs- although iftoolArgsisn't behaving correctly I'd still like to look at that.DanTup commentedon Nov 24, 2021
Beta is here -> https://github.com/Dart-Code/Dart-Code/releases/tag/v3.29.0-beta.1
shyndman commentedon Nov 24, 2021
I don't have the work in front of me, but as I recall
toolArgsare being provided to the sub-command, not the Dart binary, so the experiment was not enabled.I can verify in a bit.
DanTup commentedon Nov 24, 2021
Oh, I see - I presume the flag you're passing needs to go before
run test:test, but currently we assumetoolArgsare to be passed to the test package.In that case, it's possible
dart.testAdditionalArgsalso won't help you, since they'll also be passed to test.I'm wondering if using
vmAdditionalArgshere may cause issues (for anyone wanting it only passed to running scripts) or if it makes more sense that another setting for vm flags for test. Perhaps the former makes sense, and if thre are flags that need to be passed only to CLI scripts, there should be a specificdart.cliAdditionalArgsor similar.shyndman commentedon Nov 24, 2021
Yup, could be problematic. I don't think you can go wrong adding a test vm flags option.
[-]Add a "dart.testAdditionalArgs" setting[/-][+]Support passing args to Dart VM when running tests[/+]6 remaining items
shyndman commentedon Nov 24, 2021
Oh wow, bizarre. Thanks for the head's up.
DanTup commentedon Nov 24, 2021
Actually, scratch that bit - it was user error. In the nightly (eg. v2.15) it is enabled by default, but only when your pubspec's environment setting is >= 2.15. This makes sense, since you generally need to opt in to language changes with the version.
So it might still catch you out, but the fix is to update your SDK constraint :-)
shyndman commentedon Nov 24, 2021
I just read over all your comments. Thanks for your help. Quick turnaround. :)
DanTup commentedon Nov 25, 2021
New beta release here -> https://github.com/Dart-Code/Dart-Code/releases/tag/v3.29.0-beta.2
dartonly for CLI scripts (to the DAP viatoolArgs)testindart run test:testfor test scripts (to the DAP viatoolArgs)dartin both CLI and test scripts (to the DAP viavmAdditionalArgs)gspencergoog commentedon Apr 14, 2023
@DanTup Is there a way to pass the
--local-engine=host_debug_unoptarg to a flutter test run in VSCode, ideally by just hitting the "Run" button in the code? I've tried all three of the settings above, andtoolArgsin the configuration, and it doesn't appear to be passed, or isn't passed in the right place. I can run it on the command line with something like the line below, but I'd like to be able to set breakpoints and use the debugger:flutter test --local-engine=host_debug_unopt test/services/lifecycle_test.dartAnd how does it know that something is a test? Just if it ends in "_test.dart", or if it's in the "test" dir?
DanTup commentedon Apr 17, 2023
@gspencergoog using
toolArgsshould work fine. You can use"templateFor": "test"to have it apply to the Run/Debug links in thetestfolder (you can set it to an empty string to apply to the whole project).Eg.:
This creates an additional launch config that passes that flag, and sets it as the template for anything in the
testproject (this means the Run/Debug links and the Test Runner will use it when running tests). If I do this and hit the Run/Debug links, I get this output (which matches what I see in a terminal using that command, I guess because I haven't set the engine up locally):Let me know if this doesn't seem to work for you.
To be considered a test file, it must end with
_test.dartand either be:test,integration_testor `test_driverdart.allowTestsOutsideTestFoldersetting totrueIf those conditions are met, we'll use
flutter testinstead offlutter run.gspencergoog commentedon Apr 17, 2023
Hmm. Maybe I'm just not holding it right.
I added the following to my
launch.json:It shows up in my list of "Run and Debug" configurations, but when I run it, it (unsurprisingly) gives the popup message "Set the 'program' value in your launch config (eg 'lib/main.dart') then launch again".
When I select the
scheduler_testconfiguration in the dropdown menu, it runs with the stock engine, not the local one, even though the "Local Engine Test" template above is still defined. Clicking on the "Run" or "Debug" links in the test filemainitself causes it to run without the local engine also.When I just add a
programfield to the "Local Engine Test" JSON template, I get "Setting VM flags failed: Unrecognized flags: local-engine" in the debug output.So, I think I'm missing something fundamental: how should I use this template?
DanTup commentedon Apr 17, 2023
If you don't have an explicit
programin a launch config, it will try to infer it from your open files/workspace. If you had a test file open when you did this, it suggests we're not detecting it correctly.Could you file a new issue and include steps for me to reproduce this, including exactly what codebase you have open, and how you have it open (eg. what folder(s) you have open, or if a
.code-workspacewhat folders are in it)?DanTup commentedon Apr 17, 2023
FWIW, I opened
flutter/packages/flutterin VS Code, created alaunch.jsonfrom your code above, and clicked the "Run" CodeLens, and it did what you wanted 😞So if you file a new issue, please also include a log file captured by:
flutter/packages/flutterfolderF1)RunCodeLens abovetest('Tasks are executed in the right order', () {intest/scheduler/scheduler_test.dartAttach the log to the new issue, or send to me at [email protected] (but still file the issue so we have somewhere to track this issue that isn't on a closed issue triggering notifications to others :-) ).
gspencergoog commentedon Apr 17, 2023
I think I found a config that works:
{ "name": "Local Engine Test", "type": "dart", "request": "launch", "toolArgs": ["--local-engine=host_debug_unopt"], "cwd": "${workspaceRoot}/packages/flutter", "program": "test/scheduler/scheduler_test.dart", },Although why it works and the other didn't, I'm not sure (I'm pretty sure I tried this already and it didn't work).
I'll try and repro exactly what happened before and file an issue if I can. I wasn't aware that which file I have open would affect it the arguments a configuration runs with, so maybe that's what was confusing things.
One thing I think I've observed (but can't repro right now) is that the
--local-engineflag may be caching itself, so that running from the command line might cache the local engine setting for the next run from the IDE, which would definitely be confusing. I'll file that too if I can repro it.DanTup commentedon Apr 17, 2023
To be clear, it shouldn't affect the arguments. Basically if there's no
programin the launch config being used, we will try to inferprogrambefore doing anything else. It's possible theprogramwe infer will change some things (eg. pressingF5with no launch.json at all in a Flutter project will useflutter run, in a Flutter test will useflutter test, and in a Dart project will use dart run`).The rules for what we infer are a little complex, but generally we assume you want to run the current file, and if that doesn't appear runnable (eg. it's not in
test,tool,bin) we'll try some defaults based on project type.The extension doesn't have any visibility of what you're doing at the command line, so if this is happening it could be happening in Flutter? The steps above for capturing a log might be useful though, the log file should contain info about the debug adapter we're launching, as well as all of the arguments (after the extension has added its own) being sent in the
"launch"request. Do file new issues (or ping me) if you see issues you can't explain that could be extension-related though.DanTup commentedon Apr 17, 2023
I just noticed that based on this you probably have the Flutter repo root open. This means in my original example, this was wrong:
It would have had to be:
Since it's a relative path from the workspace root. With just
"test"it wouldn't have applied as the template for the CodeLens links inside the folder you wanted.gspencergoog commentedon Apr 17, 2023
Yes, you're correct, I have the Flutter root repo open.
Yes! that seems to fix things, and clicking "Run" with that change causes the test to run with the local engine. Thanks!
I'm assuming that was what the issue was before so I'm not going to try and repro and file, unless you'd like a documentation issue to document it.
gspencergoog commentedon Apr 17, 2023
Yes, my assumption is that it's in the flutter tool somehow, or maybe even specific to Linux builds. But maybe I've just been confused by not understanding exactly how the command arguments are determined when I click "Run".
DanTup commentedon Apr 17, 2023
There are some existing "docs" about this here that might be useful:
https://dartcode.org/docs/launch-configuration/
Using the log steps above you should be able to see what arguments are going through to the debug adapter that might be useful for debugging things like that. Search for
"launch"and you'll find something like this (although it'll be on one line, so paste into another file, set language to json, then format):