Repository navigation
Use the analysis server for semantic highlighting #2202
Description
On the insiders channel, VS Code now provides apis for semantic token highlighting. This would allow Dart Code to enhance the static grammar-based highlighter with highlighting information from the analyzer.
In particular, the extension could use it to:
- be more accurate at semantic highlighting (members starting with an uppercase, named
get/set, complex string literals, ...) - highlight non-Dart files via analysis server plugins
To my knowledge, LSP doesn't support syntax highlighting (yet). So I think this would have to be implemented using the DAS protocol and a contributor for now. I've started some work on this and I hope I can open a PR next week.
Current issues
- Not available in a stable VS Code release
- The analysis server doesn't highlight
nullandthrowtokens, https://dart-review.googlesource.com/c/sdk/+/131071
Activity
DanTup commentedon Aug 27, 2020
@simolus3 I've uploaded a build of Dart-Code that uses the preview LSP client that includes semantic tokens:
https://github.com/Dart-Code/Dart-Code/releases/tag/v3.14.0-dev.semantic-tokens
You can download the vsix and install using the Extensions: Install from VSIX command in VS Code.
I have (temporarily) disabled the built-in syntax for that version to aid testing. This means files will be uncoloured until the server provides highlights (though when we ship for real, the original Dart syntax will be included to avoid a delay in providing some colours while the server is doing initial analysis etc.).
An easy test that it's working in Dart files is to create a variable starting with an uppercase letter - this was incorrectly coloured as a type in the original grammar.
The latest server code is here:
https://github.com/DanTup/sdk/tree/lsp-3-16-semantic-tokens
It fixes multiline tokens, but not yet overlapping ones. For Dart, I might create a custom highlight_computer for LSP since the mapping is quite complicated (esp. for multiline), though for plugins we'd either need to continue to map, or introduce an LSP-specific tokens API for plugins (we don't currently have any LSP-specific plugin APIs, but I suspect we'll want to start at some point).
I'm undecided about when to start trying to land things in the SDK repo yet. I was planning to wait until the LSP spec is final, but I may consider putting it behind a flag or something to avoid building up a huge review.
10 remaining items
DanTup commentedon Dec 10, 2020
@simolus3 LSP v3.16 is due to be finalised imminently (hurrah!). I have started preparing to land changes in the server:
https://dart-review.googlesource.com/c/sdk/+/175722/
Once LSP 3.16 is final (and the next version of VS Code) I'll make a new preview build of Dart-Code with the new client that will work with the server changes.
I've included a basic plugin test in that CL, though if there are cases you think it doesn't cover, please let me know (or feel free to contribute changes).
simolus3 commentedon Dec 10, 2020
Awesome, thanks for letting me know! I took a look at your CL and I think your test cases cover what I'm interested in.
[Analyzer] Add LSP support for textDocument/semanticTokens/full
DanTup commentedon Dec 15, 2020
Fixed by dart-lang/sdk@cb2ede5.
In order to get the fix you'll need Dart-Code v3.18.0 (a preview release should be available later today/tomorrow), be using LSP Preview, and have a Dart SDK from after yesterday (the most recent nightly works).
@simolus3 once the Dart-Code preview is available later, please give this a try with your plugin and let me know if you hit any issues. Thanks!
DanTup commentedon Dec 15, 2020
Preview build is here: https://github.com/Dart-Code/Dart-Code/releases/tag/v3.18.0-alpha.1
ericdallo commentedon Dec 15, 2020
Cool @DanTup, I'll test it with
lsp-dartsince already have support for semantic highlighting, thank you!DanTup commentedon Dec 16, 2020
FYI in case you see this - there was a bug in where overlapping tokens might not be processed correctly (due to an unstable sort) that could result in some tokens being dropped (the blue
imports):I've got a fix at https://dart-review.googlesource.com/c/sdk/+/176446/ though it won't be in the nightly until tomorrows (assuming I get it landed today).
DanTup commentedon Dec 20, 2020
FYI - I've disabled Semantic Tokens in the server while I get a fix for a race condition in the code that handles splitting multiline tokens. Hoping to get a fix soon.
DanTup commentedon Dec 22, 2020
Due to the issue above, I'm reverting the change to support LSP v3.16 from the upcoming release and will instead aim to ship it in the next one. You can still use the preview build for it (and I can build. anew preview build after the release that re-enables it), though you'll want a version of the SDK with all the fixes to avoid the issue mentioned above.
DanTup commentedon Jan 19, 2021
The issues in the server were fixed and landed, and the Dart-Code change to re-enabled them landed in a11cba2.