analyzer takes many seconds for Snippets+Overrides completions in files that are part of "large" libraries

Autocomplete is much faster for me when I use the old analyzer protocol:

Dart.Analyzer.1.mp4

This is a proprietary codebase so unfortunately I'm not able to share it. A few things about this codebase which may be relevant:

This is a web application which uses dart2js to compile to HTML + JS.
This codebase is spread over ~20 packages. There is a main package which imports all the sub-packages.
The main library has a file which imports 33 packages, including 8 from the Dart SDK. It has 49 part declarations for files within the repository. Most other files don't have imports, and instead use part of to inherit the global scope provided by the main file. Most of the other packages in this project do the same.

Please let me know if there is anything else I can share about this codebase.

I've included the info above about imports because the new analyzer protocol seems slowest when trying to autocomplete in the global scope. As you can see in the video, it's much faster once there's a narrower context.

SDK: v2.19 (from Flutter 3.7.0)
OS: Windows 11
Specs: 8th generation i7, 32gb ram

@Secondflight can you confirm a few things:

what version of the SDK are you using?
is this a Flutter or pure Dart project?
what is your "dart.autoImportCompletions" setting set to, and does toggling it to the opposite seem to affect the difference in timings?
do the items in the completion list seem to be approximately the same, or is there a big difference in the items suggested?
are you able to reproduce the poor performance (or a significant difference) on a public project (for example opening the root folder of the flutter/futter repo?). For me, if I open flutter/flutter and wait for initial analysis to complete, code completion in packages/flutter/lib/material.dart at the very end of the file appears in less than 1 second, so your video does seem very slow (although I'm not using your project and I don't know if there may be something specific triggering this)

Thanks!

@DanTup

Hi @DanTup, thanks for the quick response!

I'm using v2.19 of the SDK.
This is a pure-Dart project.
"dart.autoImportCompletions" is set to false in the video, but it doesn't seem to make a difference here.
I'm not sure I understand this question, are you asking if some completions are slower than others? It seems like autocomplete is slowest when I'm in the global scope. As you can see from the video, both the old and new analyzer protocols are fine when doing autocomplete within a class.
- (Edit: maybe not fine, when compared to Flutter below, all our autocompletes feel sluggish.)
I just pulled flutter, and it's much faster. This is with the new protocol:

Dart.Analyzer.2.mp4

I'm wondering if this has something to do with how we're abusing imports in this project. Most packages in our project are constructed almost entirely with part / part of.

I can test this theory by constructing a large project using part statements. I don't have time now, but I'll reply here when I get the chance to do that.

If you're able to reproduce it with something shareable that would really help.

Something else that might be useful:

open your project with it set to LSP (the slow way)
run the Dart: Open Analyzer Diagnostics command
click Code Completion on the left
back in the editor, invoke the slow code completion
refresh the diagnostics page and click on the time for the slow completion

This should give you a page like the below, which might at least show which part of completion is taking the time:

Okay, makes sense. I'll see what I can do.

Thanks for those steps, I think I got something interesting. First off, it looks like I was wrong - disabling dart.autoImportCompletions cuts the time from ~10s to ~5s.

Here's the output with dart.autoImportCompletions set to true. DartCompletionManager - NotImportedContributor and getSnippets both take nearly 5 seconds:

Click to expand

(name: request, elapsed: 0:00:09.859963, elapsedSelf: 0:00:00.000454)
  (name: computeSuggestions, elapsed: 0:00:05.014003, elapsedSelf: 0:00:00.000392)
    (name: DartCompletionManager - ArgListContributor, elapsed: 0:00:00.000892, elapsedSelf: 0:00:00.000892)
    (name: DartCompletionManager - ClosureContributor, elapsed: 0:00:00.001651, elapsedSelf: 0:00:00.001651)
    (name: DartCompletionManager - CombinatorContributor, elapsed: 0:00:00.001243, elapsedSelf: 0:00:00.001243)
    (name: DartCompletionManager - EnumConstantConstructorContributor, elapsed: 0:00:00.002426, elapsedSelf: 0:00:00.002426)
    (name: DartCompletionManager - ExtensionMemberContributor, elapsed: 0:00:00.003492, elapsedSelf: 0:00:00.003492)
    (name: DartCompletionManager - FieldFormalContributor, elapsed: 0:00:00.001411, elapsedSelf: 0:00:00.001411)
    (name: DartCompletionManager - KeywordContributor, elapsed: 0:00:00.001788, elapsedSelf: 0:00:00.001788)
    (name: DartCompletionManager - LabelContributor, elapsed: 0:00:00.000027, elapsedSelf: 0:00:00.000027)
    (name: DartCompletionManager - LibraryMemberContributor, elapsed: 0:00:00.000006, elapsedSelf: 0:00:00.000006)
    (name: DartCompletionManager - LibraryPrefixContributor, elapsed: 0:00:00.000235, elapsedSelf: 0:00:00.000235)
    (name: DartCompletionManager - LocalLibraryContributor, elapsed: 0:00:00.003534, elapsedSelf: 0:00:00.003534)
    (name: DartCompletionManager - LocalReferenceContributor, elapsed: 0:00:00.001672, elapsedSelf: 0:00:00.001672)
    (name: DartCompletionManager - NamedConstructorContributor, elapsed: 0:00:00.000004, elapsedSelf: 0:00:00.000004)
    (name: DartCompletionManager - OverrideContributor, elapsed: 0:00:00.000004, elapsedSelf: 0:00:00.000004)
    (name: DartCompletionManager - RecordLiteralContributor, elapsed: 0:00:00.000010, elapsedSelf: 0:00:00.000010)
    (name: DartCompletionManager - RedirectingContributor, elapsed: 0:00:00.000004, elapsedSelf: 0:00:00.000004)
    (name: DartCompletionManager - StaticMemberContributor, elapsed: 0:00:00.000004, elapsedSelf: 0:00:00.000004)
    (name: DartCompletionManager - SuperFormalContributor, elapsed: 0:00:00.000003, elapsedSelf: 0:00:00.000003)
    (name: DartCompletionManager - TypeMemberContributor, elapsed: 0:00:00.000003, elapsedSelf: 0:00:00.000003)
    (name: DartCompletionManager - UriContributor, elapsed: 0:00:00.000005, elapsedSelf: 0:00:00.000005)
    (name: DartCompletionManager - VariableNameContributor, elapsed: 0:00:00.000003, elapsedSelf: 0:00:00.000003)
    (name: DartCompletionManager - ImportedReferenceContributor, elapsed: 0:00:00.009575, elapsedSelf: 0:00:00.009575)
    (name: DartCompletionManager - NotImportedContributor, elapsed: 0:00:04.985619, elapsedSelf: 0:00:04.985619)
  (name: buildSuggestions, elapsed: 0:00:00.003575, elapsedSelf: 0:00:00.003575)
  (name: mapSuggestions, elapsed: 0:00:00.001993, elapsedSelf: 0:00:00.001993)
  (name: getSnippets, elapsed: 0:00:04.839938, elapsedSelf: 0:00:04.839938)

Disabling dart.autoImportCompletions leaves just getSnippets - the rest are very short:

Click to expand

(name: request, elapsed: 0:00:04.754602, elapsedSelf: 0:00:00.000234)
  (name: computeSuggestions, elapsed: 0:00:00.007088, elapsedSelf: 0:00:00.000200)
    (name: DartCompletionManager - ArgListContributor, elapsed: 0:00:00.000497, elapsedSelf: 0:00:00.000497)
    (name: DartCompletionManager - ClosureContributor, elapsed: 0:00:00.000912, elapsedSelf: 0:00:00.000912)
    (name: DartCompletionManager - CombinatorContributor, elapsed: 0:00:00.000004, elapsedSelf: 0:00:00.000004)
    (name: DartCompletionManager - EnumConstantConstructorContributor, elapsed: 0:00:00.000003, elapsedSelf: 0:00:00.000003)
    (name: DartCompletionManager - ExtensionMemberContributor, elapsed: 0:00:00.000095, elapsedSelf: 0:00:00.000095)
    (name: DartCompletionManager - FieldFormalContributor, elapsed: 0:00:00.000001, elapsedSelf: 0:00:00.000001)
    (name: DartCompletionManager - KeywordContributor, elapsed: 0:00:00.000101, elapsedSelf: 0:00:00.000101)
    (name: DartCompletionManager - LabelContributor, elapsed: 0:00:00.000002, elapsedSelf: 0:00:00.000002)
    (name: DartCompletionManager - LibraryMemberContributor, elapsed: 0:00:00.000001, elapsedSelf: 0:00:00.000001)
    (name: DartCompletionManager - LibraryPrefixContributor, elapsed: 0:00:00.000117, elapsedSelf: 0:00:00.000117)
    (name: DartCompletionManager - LocalLibraryContributor, elapsed: 0:00:00.001357, elapsedSelf: 0:00:00.001357)
    (name: DartCompletionManager - LocalReferenceContributor, elapsed: 0:00:00.000527, elapsedSelf: 0:00:00.000527)
    (name: DartCompletionManager - NamedConstructorContributor, elapsed: 0:00:00.000002, elapsedSelf: 0:00:00.000002)
    (name: DartCompletionManager - OverrideContributor, elapsed: 0:00:00.000002, elapsedSelf: 0:00:00.000002)
    (name: DartCompletionManager - RecordLiteralContributor, elapsed: 0:00:00.000004, elapsedSelf: 0:00:00.000004)
    (name: DartCompletionManager - RedirectingContributor, elapsed: 0:00:00.000001, elapsedSelf: 0:00:00.000001)
    (name: DartCompletionManager - StaticMemberContributor, elapsed: 0:00:00.000001, elapsedSelf: 0:00:00.000001)
    (name: DartCompletionManager - SuperFormalContributor, elapsed: 0:00:00.000001, elapsedSelf: 0:00:00.000001)
    (name: DartCompletionManager - TypeMemberContributor, elapsed: 0:00:00.000001, elapsedSelf: 0:00:00.000001)
    (name: DartCompletionManager - UriContributor, elapsed: 0:00:00.000002, elapsedSelf: 0:00:00.000002)
    (name: DartCompletionManager - VariableNameContributor, elapsed: 0:00:00.000001, elapsedSelf: 0:00:00.000001)
    (name: DartCompletionManager - ImportedReferenceContributor, elapsed: 0:00:00.003256, elapsedSelf: 0:00:00.003256)
  (name: buildSuggestions, elapsed: 0:00:00.000656, elapsedSelf: 0:00:00.000656)
  (name: mapSuggestions, elapsed: 0:00:00.000489, elapsedSelf: 0:00:00.000489)
  (name: getSnippets, elapsed: 0:00:04.746135, elapsedSelf: 0:00:04.746135)

Disabling dart.enableServerSnippets fixes the other 5s. It's still noticeably slow compared to the Flutter codebase, but most completions happen within 1s.

Oh, that's interesting to know. Are you able to construct a repo you can share? If we're taking 5s to do those snippets, we absolutely need to fix that.

Edit: Missed the other comment about NotImportedCompletions. Sounds like we also need to improve that. Anything you can do to get us a repro that has these sorts of timings would really help us track down why it's slow for you.

I can't do it today, but I'll do it ASAP - probably early next week. I'll ping you when I get some more time to spend on it.

Great, thanks!

@DanTup

Hey @DanTup,

I've spent a few hours trying to reproduce the issue, but haven't gotten anywhere yet. I've recreated a similar project structure, but the suggestions in my repro project are still really fast.

Just writing here to say I'm still working on it. Please let me know if you have any suggestions on how I might trace the issue.

@Secondflight I actually started looking at this today too. I think I understand how snippets can be slow, but I would expect it only to be the very first request (it triggers some expensive work to discover all available files). Can you confirm whether you see it adding on 5s only the first time it's invoked, or all?

As for getting a repro - I don't know how big your project is, but I would probably start by making a copy of it, and then start deleting all the bodies of code (you can use throw to avoid having to make functions return the right types) and see whether you can get to something that still reproduces it but might be shareable. I don't know how feasible that is though if your project is massive.

@DanTup

@DanTup It does speed up, but it stabilizes to something that's still pretty slow:

Also, I made the mistake of not recording where I had the 10s - I've been seeing 2s-3s in most function bodies. NotImportedContributor isn't causing issues anymore either which is strange.

Here's the breakdown of that last suggestion. getSnippets is the vast majority:

As for getting a repro - I don't know how big your project is, but I would probably start by making a copy of it, and then start deleting all the bodies of code (you can use throw to avoid having to make functions return the right types) and see whether you can get to something that still reproduces it but might be shareable. I don't know how feasible that is though if your project is massive.

I appreciate this. I probably still can't share the result, but it's a different angle to attack the problem. Our codebase is ~100k lines of Dart plus HTML and CSS, so it's not small but not unmanageably large either.

As a short-term work-around I'd be fine with disabling the override contributor if we're completing in a part file just to ensure that we can provide all of the other completions.

I can do that if you think it's worthwhile - though I don't know whether it's a good or bad trade-off (users with smaller libraries that aren't so slow might miss these). My message wasn't a prompt to do something, just an FYI in case you hear similar complaints :-)

It could be worth making the budget apply in more places (so we'd never take this long), although there's a part of me that dislikes stopping work early and giving incomplete results (because sometimes I might prefer to wait a little longer and get results than not).

Anyway, something to come back to :-)

@scheglov

This is resolved by @scheglov's change at dart-lang/sdk@92faa11. In my test case with 100 parts in a single library, the time dropped from 17s (first) and 9s (subsequent) to 760ms (first) and 175ms (subsequent). This was a deliberately extreme case though so the total times are likely much lower for real world code.

The change is part of the Dart SDK (not the VS Code extension), so it will show up after a future SDK update.

Navigation Menu

Repository navigation

analyzer takes many seconds for Snippets+Overrides completions in files that are `part of` "large" libraries #4364

33 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

analyzer takes many seconds for Snippets+Overrides completions in files that are part of "large" libraries #4364

Description

Activity

DanTup commented on Jan 26, 2023

josh-audio commented on Jan 26, 2023

josh-audio commented on Jan 26, 2023

DanTup commented on Jan 26, 2023

josh-audio commented on Jan 26, 2023

josh-audio commented on Jan 26, 2023

DanTup commented on Jan 26, 2023

josh-audio commented on Jan 26, 2023

DanTup commented on Jan 26, 2023

josh-audio commented on Feb 2, 2023

DanTup commented on Feb 2, 2023

josh-audio commented on Feb 2, 2023

josh-audio commented on Feb 2, 2023

33 remaining items

DanTup commented on Mar 22, 2023

DanTup commented on May 18, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

analyzer takes many seconds for Snippets+Overrides completions in files that are `part of` "large" libraries #4364