Skip to content

Add an "auto" mode for dart.testInvocationMode that tries to pick the best mode automatically #5716

Closed

Description

@DanTup

We have a setting to run tests by line (instead of by name), but it's opt-in.

We could solve some issues by automatically defaulting to using lines if we think that running by-name will fail. For example #5707 where the group() call is not inside main() is problematic because the caller might have wrapped us in a group() and we won't include that in the computed full name.

Similarly, we have some workarounds for dynamic test names (test('foo $bar'...) to replace variables with .*, but running by line might be a cleaner way to do this.

When checking my config, it seems I may have been using by-line for a long time not realising (and forgetting that it wasn't default) so it might be pretty solid.

Some things we should do:

  • Change the default to undefined/"auto" instead of "name" so we can still force running all tests by-name in case there are any problems when we automatically use lines
  • Verify behaviour with tests that come from results (like pkg:test_reflective_loader) because we don't currently track their locations and might do the wrong thing (although, if we minimize where we default to by-line for, that might minimize this)

Probably candidates to run by-line by default:

  • Any variables/dynamic names (we may want to use the presence of quotes in the Outline name for this?)
  • Test nodes outside of the main top level function

Activity

  1. added
    is enhancementAn enhancement or improvement that should be listed in release notes but is not a bug fix.
    on Sep 22, 2025
  2. added this to the v3.120.0 milestone on Sep 22, 2025
  3. added
    in testingRelates to test execution of Dart/Flutter tests for end users
    on Sep 22, 2025
  4. modified the milestones: v3.120.0, v3.122.0 on Sep 29, 2025
  5. modified the milestones: v3.122.0, v3.124.0 on Oct 8, 2025
  6. DanTup commented on Nov 20, 2025

    @DanTup
    MemberAuthor

    This doesn't play too nice with unsaved files, so we need to do some more work here.

  7. modified the milestones: v3.124.0, v3.126.0 on Nov 20, 2025
  8. modified the milestones: v3.126.0, v3.128.0 on Dec 18, 2025
  9. modified the milestones: v3.128.0, On Deck on Jan 21, 2026
  10. modified the milestones: On Deck, v3.144.0 on Sep 21, 2026
  11. self-assigned this
    on Sep 21, 2026
  12. changed the title [-]Default to running tests by line when they might otherwise fail[/-] [+]Add an "auto" mode for dart.testInvocationMode that tries to pick the best mode automatically[/+] on Sep 22, 2026
  13. DanTup commented on Sep 22, 2026

    @DanTup
    MemberAuthor

    The issue with unsaved files was not due to the edits, but rather that saving the file can cause formatting, which can change the lines.

    So, if a file is unsaved, we will always use "name" if you selected "auto", and for now we are not changing the default, it's just a better option than "line".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

in testingRelates to test execution of Dart/Flutter tests for end usersis enhancementAn enhancement or improvement that should be listed in release notes but is not a bug fix.

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions