experimentalTestTracking with @soloTest

Describe the bug
The new experimentalTestTracking has a problem when you remove @soloTest annotations (maybe the position with them is wrong from the start).

To Reproduce
Steps to reproduce the behavior:

Go to pkg/analysis_server/test/src/services/correction/assist/add_digit_separators_test.dart
Add @soloTest before any test besides the first one
Run the test
Remove the annotation (see the gutter icon is one line above)
Run the whole file, see the position is still one line above

Expected behavior
For the gutter icon to be at the line that declares the method return type

Screenshots

Can you point me where should I look into if I wanted to fix this myself? Would it be on test_reflective_loader?

I'm not sure what the issue is yet, so I'm not certain where the fix was. My feeling is that it's in Dart-Code in the position tracking code, or updating the position after you re-run the tests.

When we run tests, we run them with a json reporter which provides us the location, and when we get a new test location we try to match it up to an existing test in our model before creating a new one.

I feel like there might be two issues here:

When you remove the @soloTest, I feel like we should've collapsed the position we were tracking better (to the start of the test).. I'm actually surprised we didn't just delete the marked entirely (because the start position was removed), even though that wouldn't be a great experience either
When you re-run the tests, we should match up the new test result with the existing node in the test model and update its location

I was going to try and look at both the issues you raised in this area shortly, so if you're mostly interested in a fix than fixing it, I'll still do that. If you're interested in trying to fix I'm happy to leave it though (or try to figure out enough to give you better pointers).

I see. I feel like there are two (maybe three if you separate what you said) problems here:

Dart-Code is not handling what you said above correctly (those two things)
test_reflective_loader showing the offset of the MethodDeclaration as a whole (which includes metadata and such) and not the MethodDeclaration.name.offset, which might help here like 90% of the time

I'd be happy to take a look at test_reflective_loader and see if I can make that work. Currently, if you have a really big method name and a return type, the offset is of the return type, which IMO, is not great UX (I remember some tests had this, but I know none at the top of my head, so a simple stub repro below from a new CL I'm working on).

test_reflective_loader showing the offset of the MethodDeclaration as a whole (which includes metadata and such) and not the MethodDeclaration.name.offset, which might help here like 90% of the time

While that sounds like a fair change, test_reflective_loader is using Mirrors and I'm not sure if you can get at this.

See:

Oh, I see. Then, at least the Mirrors docs are missing the bit on the annotations. I'll get more familiar with how Mirrors work and see if that is really the case. I'll open an issue/PR there then. Thanks!

The docs say:

If the reflectee is a library, class, typedef, function or type variable with associated metadata, the returned location gives the position of the first metadata declaration associated with the reflectee.

I think function here means functions_+methods (since methods aren't otherwise mentioned).

Oops, I skipped that part and read this first:

If the reflectee is a function with a declared return type, the returned location gives the position of the function's return type at the reflectee's point of declaration. Otherwise. the returned location gives the position of the function's name at the reflectee's point of declaration.

It seems fine then, thanks!

Remove the annotation (see the gutter icon is one line above)

I just tested but couldn't actually repro this. If I delete the text that includes the start of @soloTest then the marked disappears:

disappears.mp4

And if I keep that position in-tact and delete after it, the marker remains where you'd expect:

good.mp4

Can you confirm exactly what you're deleting when you see it go to the wrong place?

Currently we delete the tracked range if it's inside an area that gets deleted (for ex. you might have deleted the whole test, in which case we don't want the marker to jump to the end/start of what you deleted). Tracking the test name would certainly reduce the weirdness, but I don't know if we can do that given the mirrors (except maybe with some heuristics in Dart-Code that try to ignore comments and annotation lines 🤔).

Had you already run this test previously? Maybe this only happens on "new tests" (not previously tracked)?

20251007_142825.mp4

This is from a completely clean tracking (restarted the extension host).

Hmm, I still can't repro. Can you confirm exactly what you're doing? This is what I tried:

Added @soloTest to that test
Ran with F5 so the gutter icon showed up
Put my text cursor at the end of the @soloTest and held <shift> then pressed <up>
Pressed <delete>

tests.mp4

At the end I re-ran the whole file with Ctrl+F5 just to see everything showed up again as expected.

Not sure what the difference is between F5 and Ctrl + F5, but I've done exactly as you said, and the gutter icon stays on line 67 for me once I delete the @soloTest (I've probably done the same but with Backspace instead of Delete and just F5 instead of CTRL + F5 originally, but I've tested the same as you and no change).

Debug logs here

@FMorschel

@FMorschel sorry for the delay... I took a look at the log when you posted this and couldn't figure it out, and I'm looking again and still can't.

Are you able to check you definitely have dart.experimentalTestTracking: true for this project? When I try to repro, the marker disappears (as I would expect), but for you it does not - and I can see that the code to remove it will only run for the experimental tracking.

Can you also try running "Clear test results" and reloading first, just in case there's anything weird with previous data hanging around.

Thanks!

Are you able to check you definitely have dart.experimentalTestTracking: true for this project? When I try to repro, the marker disappears (as I would expect), but for you it does not - and I can see that the code to remove it will only run for the experimental tracking.

Oops, sorry! I must've accidentally set this to false on my JSON by mistake when I wanted to change some other flag.

I've made sure I have this set to true on my profile (which I don't edit much), so this doesn't happen again.

This is all WAI now. Thanks!

Got it, thanks for confirming!

FWIW, I published a pre-release today that removes that flag and replaces it with a temporary flag dynamicTestTracking which defaults to true (so basically it's enabled by default now, and that flag is just an emergency opt-out, which will be removed in a future version).

Navigation Menu

Repository navigation

`experimentalTestTracking` with `@soloTest` #5735

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

experimentalTestTracking with @soloTest #5735

Description

Activity

FMorschel commented on Oct 7, 2025

DanTup commented on Oct 7, 2025

FMorschel commented on Oct 7, 2025

DanTup commented on Oct 7, 2025

FMorschel commented on Oct 7, 2025

DanTup commented on Oct 7, 2025

FMorschel commented on Oct 7, 2025

DanTup commented on Oct 7, 2025

FMorschel commented on Oct 7, 2025

DanTup commented on Oct 8, 2025

FMorschel commented on Oct 8, 2025

DanTup commented on Oct 29, 2025

FMorschel commented on Oct 29, 2025

DanTup commented on Oct 29, 2025

DanTup commented on Oct 29, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`experimentalTestTracking` with `@soloTest` #5735