Stale test markers for dynamic tests come back when using experimentalTestTracking and modifying the test

Describe the behaviour
With the new experimentalTestTracking set to true, I see a weird behaviour in test_reflective_loader test files once I've tried running a debug session that did not complete any tests.

To Reproduce
Steps to reproduce the behavior:

Go to pkg/analysis_server/lib/src/services/correction/dart/add_digit_separators.dart
Add a breakpoint at the first line inside _AddDigitSeparators.compute (at the time of writing is 69)
Go to pkg/analysis_server/test/src/services/correction/assist/add_digit_separators_test.dart
Disable that breakpoint on the breakpoints tab temporarily
Run the full test file
Enable the breakpoint and run the tests
Once it breaks, stop the debugging session
Go back to the test file and see no gutter icons
Add a new line or something like that (some change, I think even whitespaces might do) and see the gutter icons show up
Removing that change will make them go away
Saving the file (even if you delete the change you made - not undoing) will show the gutter icons again

P.S.: With that flag set to false, the gutter icons simply don't show up until you run the tests again (nobody is tracking the tests' positions).

Expected behavior
I'm not sure if this can be fixed here or up on VS Code or what, but I expect it either to never show the gutter icons or to keep showing them.

Screenshots

20251002_121507.mp4

Logs:

Dart-Code-Log-2025-09-04 12-30-26.txt

Slightly simpler repro:

import 'package:test/test.dart';
import 'package:test_reflective_loader/test_reflective_loader.dart';

void main() {
  defineReflectiveSuite(() {
    defineReflectiveTests(MyTest);
  });
}

@reflectiveTest
class MyTest {
  void test_one() {
    expect(1, 1);
  }

  void test_two() {
    expect(1, 1);
  }

  void test_three() {
    expect(1, 1);
  }
}

Run all tests
Add breakpoint into test_two
Run all tests and hit breakpoint
Stop the debugging session

The test results for tests after the one we stopped on vanish.

Edit:

The reason this happens is that we think this test has been deleted, because you run the tests and it was no longer in the results:

Removing potentially-deleted MyTest | test_three

Need to rethink this...

It also doesn't explain why the tests come back though. Is it because the tracker is restoring them?

I'm splitting this into two issues.

This issue tracks that the icons came back when you modified the file. This is a bug - it's stale data being restored because we didn't clean up the tracking properly when we removed the nodes the first time.
I've opened Don't remove non-outline nodes that don't show up in test runs #5739 to see if we can handle dynamic tests that were not in the set of results you just ran. Currently (without the new tracking), we don't know if you deleted a test, or skipped it (via @soloTest or from stopping early). We assume deleted, since you can always get things back by re-running the whole file. With the new tracking, we might be able to improve that.

Navigation Menu

Repository navigation

Stale test markers for dynamic tests come back when using `experimentalTestTracking` and modifying the test #5734

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Stale test markers for dynamic tests come back when using experimentalTestTracking and modifying the test #5734

Description

Activity

DanTup commented on Oct 7, 2025

DanTup commented on Oct 7, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Stale test markers for dynamic tests come back when using `experimentalTestTracking` and modifying the test #5734