Repository navigation
Improve the summary bubble for failed tests when expected/actual values are simple/single-line #5390
Description
Expected behavior
When a test failed, it shows this red tag in the editor with the expected result. While the idea is really good as the tag would highlight the test that failed, the text the shows is completely useless and redundant as I can actually see what's the expect result on the file. What I can not see in the file however, the actual result that I got from the test execution. I need to to click the red tag which opens a sort of a popup that shows details about the unit.
Screenshots
I think it would be more useful if the red tag showed the actual result instead of the expected one. I looked through preferences to see if i could configure it but I did not find anything. What you do you think?
And sorry if this is unrelated to this repo or if it is not the right place.
Thanks
Activity
DanTup commentedon Jan 13, 2025
The red text here is actually just VS Code showing only the first line from the failure message:
If you click on it, you'll also see the complete message:
I don't think there's much we can do in the extension here, because we just provide the failure message from
package:testto VS Code. In the example here it's the Expected text, but in many cases it may be something different - for example:I don't think there's really a good fix here besides
package:testchanging it's output to try and make the first line of the failure message more of a single-line summary... but for a single editor that might be a bit weird... @jakemac53 - any opinions/thoughts?jakemac53 commentedon Jan 13, 2025
Hmm, I don't think we would change the output just for this, at least not based on just one report. Is this a new VsCode feature? If we have lots of people running into this then we could consider it.
I also don't know if this is part of the reporter or the matcher, if its in the reporter we could have a custom one for VsCode.
DanTup commentedon Jan 13, 2025
It's new that it's displayed like this, yep. It used to just immediately pop open the panel (shown in my middle screenshot), but now it starts collapsed and shows just the first line.
I do prefer the new look, but it does assume the first line of the failure message makes sense on its own.
Based on https://github.com/dart-lang/test/blob/f364fc8291d668d85c702a5b9f9a4f2e5c1ade0e/pkgs/matcher/lib/src/equals_matcher.dart#L64, I presume it's the matcher?
jakemac53 commentedon Jan 13, 2025
Yeah looks like it's the matcher then, which is harder to deal with. It also means there are probably many other matchers that would have to be updated (although equals is definitely the most common one).
jakemac53 commentedon Jan 13, 2025
It is also a bit unfortunate that the location of the message is just the top of the test and not the actual line that failed? It might be worth working with VsCode to improve this more generally.
DanTup commentedon Jan 13, 2025
That's actually on us - see dart-lang/test#1851
Another thing we discussed in the past is passing back expected/actual values specifically - see dart-lang/test#2123. Maybe that would help here too, because we could in theory inject our own first line constructed from those in this case?
DanTup commentedon Feb 19, 2025
For now, I've decided to handle the simple cases here by extracting them from the error message if they are present and on a single line in the form:
In this case, we extract the expected/actual values and provide them to VS Code directly, and it chooses how to render them, for example:
If either value is multiline, we won't do this, because we can't tell where
Actualends and other text (like "Which is different.") starts. If in the future we can get structured actual/expected values frompkg:test(dart-lang/test#2123) we could include those (in which case, VS Code can show an inline diff of the mismatch).[-]failed tests should show actual result rather than expected [/-][+]Improve the summary bubble for failed tests when expected/actual values are simple/single-line[/+]Extract simple expected/actual values from test messages for improved…