When using variant parameter of testWidgets: No tests match regular expression.... when running individual tests #86659

Activity

1. DanTup commented on Jul 19, 2021

   

   DanTup

   on Jul 19, 2021

   Contributor

   More actions

   This happens because in VS Code when we run a single test, we put its name in the command line like this:

   flutter test --machine --name "^Counter increments smoke test$" test/widget_test.dart
   

   The --name argument is an exact match for the name of the test so that it doesn't match extra tests, for example if you had:

   test('Counter increments', () {});
   
   test('Counter increments when foo', () {});

   Without the $ on the end of the name, trying to run the first test would actually run both.

   When using variant, the name of the test is suffixed (in this case "Counter increments smoke test (TargetPlatform.windows)") and therefore no longer matches the pattern. The VS Code extension doesn't have enough information in the Outline data to know the variant exists, so it's hard to amend the expression.

   @jonahwilliams any thoughts on this? Is there a way that --name could take variation into account? Or could we perhaps make the name slightly more specific so my regex could account for it without being too greedy (for example "Counter increments smoke test (variant: TargetPlatform.windows)")?
2. jonahwilliams commented on Jul 19, 2021

   

   jonahwilliams

   on Jul 19, 2021

   Contributor

   More actions

   I believe @gspencergoog added the variant feature, I would follow up with him
3. gspencergoog commented on Jul 19, 2021

   

   gspencergoog

   on Jul 19, 2021

   Contributor

   More actions

   We could make the output include "variant:", sure.

   I would point out that the IntelliJ plugin just executes all tests that contain the given test name, so there is a difference in functionality between the two plugins. I'm not commenting on which way is better, just that there's a difference (they both seem to have their merits to me).

   It is sometimes useful to run only a particular variant, so you wouldn't want to make the regex too restrictive: I should be able to enter 'Counter increments (variant: TargetPlatform.windows)' and run just the Windows variant of the test.
4. DanTup commented on Jul 19, 2021

   

   DanTup

   on Jul 19, 2021

   Last edited by DanTup

   Contributor

   More actions

   I would point out that the IntelliJ plugin just executes all tests that contain the given test name, so there is a difference in functionality between the two plugins. I'm not commenting on which way is better, just that there's a difference (they both seem to have their merits to me).

   It might make sense to align them. That's what VS Code used to do, but it felt like a bug that if the user clicked "Run" next to one, we might run other tests too. @stevemessick do you think it makes sense to update IntelliJ? I'm not familiar with the UI for IntelliJ so I don't know if it makes more sense there than in VS Code. In VS Code we're invoking i from these Run/Debug links above the testWidgets call:

   

   That we're using the name of the test behind the scenes seems like a detail the user is hidden from, so running multiple looks strange.

   I should be able to enter 'Counter increments (variant: TargetPlatform.windows)' and run just the Windows variant of the test

   If we added a variant: prefix that would all still work in the cli. For VS Code there isn't anywhere you could choose to run one specific variant though, so clicking the Run/Debug links shown above would have to run all variants (described by the variant` argument).

   So if we can add a variant: prefix in the test name, my plan would be to add ( \(variant: .*\))? to the end of the regex to match tests with exactly this name, but optionally with that variant suffix:

   flutter test --name "^Counter increments smoke test( \(variant: .*\))?$" test/widget_test.dart
   

   Does this sound reasonable? I can file a PR to add the text and ensure it doesn't break anything.
5. gspencergoog commented on Jul 19, 2021

   

   gspencergoog

   on Jul 19, 2021

   Contributor

   More actions

   Sounds reasonable to me.
6. stevemessick commented on Jul 19, 2021

   

   stevemessick

   on Jul 19, 2021

   Member

   More actions

   Filed flutter/flutter-intellij#5650
7. TahaTesser

   added

   in triagePresently being triaged by the triage teamPresently being triaged by the triage team

   a: tests"flutter test", flutter_test, or one of our tests"flutter test", flutter_test, or one of our tests

   d: intellijDevTools related - IntelliJ/flutter issues (most should be tracked in flutter/flutter-intellij)DevTools related - IntelliJ/flutter issues (most should be tracked in flutter/flutter-intellij)

   passed first triage

   frameworkflutter/packages/flutter repository. See also f: labels.flutter/packages/flutter repository. See also f: labels.

   and removed

   in triagePresently being triaged by the triage teamPresently being triaged by the triage team

   d: intellijDevTools related - IntelliJ/flutter issues (most should be tracked in flutter/flutter-intellij)DevTools related - IntelliJ/flutter issues (most should be tracked in flutter/flutter-intellij)

   on Jul 20, 2021
8. DanTup commented on Jul 20, 2021

   

   DanTup

   on Jul 20, 2021

   Contributor

   More actions

   #86701 has landed in Flutter and Dart-Code/Dart-Code#3482 in the VS Code extension. The combination of these resolves this issue for VS Code (you'll need releases of both for this to start working). IntelliJ doesn't have this issue since its regex is less strict (though flutter/flutter-intellij#5650 will make this more aligned with VS Code's behaviour).
9. DanTup

   closed this as completedon Jul 20, 2021
10. github-actions commented on Aug 3, 2021

    

    github-actionsbot

    on Aug 3, 2021

    More actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.
11. github-actions

    locked as resolved and limited conversation to collaborators on Aug 3, 2021