You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
analysis server consumes significant memory when VS Code restores a dirty file buffer for a project that no longer exists #4280
And then reproduce the issue, and see whether this log file is being written to as the memory usage is growing? You can attach the log or email it to me at [email protected] if it's not too big (but please reproduce this with a small sample app, because the log may contain parts of source code from the open files).
And then reproduce the issue, and see whether this log file is being written to as the memory usage is growing? You can attach the log or email it to me at [email protected] if it's not too big (but please reproduce this with a small sample app, because the log may contain parts of source code from the open files).
Thanks!
I have e-mailed the information you mentioned, along with about 2 minutes of screenshot files (reproducing the problem).
@zinface thank you for the detailed video and logs! I think I see what's going on. When VS Code is opened with that unsaved file, the server tries to create a watcher for it - however the file does not exist on disk and this causes an error:
1669305711347:Err:Temporary watcher error; restarting context build.
FileSystemException:: Failed to watch path, path = '/tmp/tmp.75uS4AnOM4/file_demo/bin/file_demo.dart' (OS Error:: No such file or directory, errno = 2)
1669305711437:Err:Watcher error; refreshing contexts.
FileSystemException:: Failed to watch path, path = '/tmp/tmp.75uS4AnOM4/file_demo/bin/file_demo.dart' (OS Error:: No such file or directory, errno = 2)
1669305711439:Err:Temporary watcher error; restarting context build.
FileSystemException:: Failed to watch path, path = '/tmp/tmp.75uS4AnOM4/file_demo/bin/file_demo.dart' (OS Error:: No such file or directory, errno = 2)
This server interprets this error as something it can recover from, so it tries again. And it keeps retrying repeatedly forever.
I'm not certain why the memory usage climbs so high, but now I understand the cause I might have better luck reproducing it so I can work on a fix. Thanks!
changed the title [-]Oh, My Computer is Die! Memory Done.[/-][+]analysis server consumes significant memory when VS Code restores a dirty file buffer for a project that no longer exists[/+]on Nov 30, 2022
@zinface sorry for the delay. I'm working in this now but struggling to reproduce (even on Linux). The attachment links in you email have now expired - do you still have a copy you can re-send to me to [email protected] so I can understand what's different between my setup and yours? Thanks!
@zinface sorry for the delay. I'm working in this now but struggling to reproduce (even on Linux). The attachment links in you email have now expired - do you still have a copy you can re-send to me to [email protected] so I can understand what's different between my setup and yours? Thanks!
I just saw the email now. Of course, it's already the weekend. I have re-sent the content I prepared last time.
Ok, I have a better idea of what's happening now. In the log there are two sets of errors alternating:
1676999680132:Err:Temporary watcher error; restarting context build.
PathNotFoundException:: Failed to watch path, path = '/home/danny/Desktop/deleteme/deleteme2/foo.dart' (OS Error:: No such file or directory, errno = 2)
1676999680153:Err:Watcher error; refreshing contexts.
PathNotFoundException:: Failed to watch path, path = '/home/danny/Desktop/deleteme/deleteme2/foo.dart' (OS Error:: No such file or directory, errno = 2)
The first error ("Temporary watcher error") is the one we're handling while trying to build the context roots. We will try up to 5 times before giving up.
The second error is code that tries to handle errors generally and triggers a context rebuild. It's this one that's causing the rebuilds to occur in a loop, not the original "temporary watcher" one (but because it triggers a rebuild, it causes that code to run again, and the error to be logged).
I think there may be a difference in how these errors occur on Linux versus other platforms which is why it's only seen here, although it seems like we should detect if we get into a situation where we're endlessly trying to rebuild contexts (they're likely failing for a reason that isn't going to change if we try again).
Two thoughts:
If we seem to be endlessly rebuilding contexts, shut down with an error messages
Detect when we're failing to set up a watcher because the folder doesn't exist, and handle it differently (in this case, not creating the watcher would be better - and perhaps what's happening on other platforms?).. although we won't handle the case where that folder is re-created - but I think we may already not handle this correctly in some cases.
(@bwilkerson@scheglov FYI.. I think I have a fix for this, but some background for when you get the review..)
I have a very easy way to reproduce what is essentially the same issue (it only occurs on Linux - there are some differences in how this kind of error is raised across platforms, I think it's been discussed before although I can't find the discussion):
run dart language-server --protocol=analyzer --instrumentation-log-file=/tmp/analyzer.txt
in another terminal, run tail -f /tmp/analyzer.txt
paste into the server process {"id": "1", "method": "analysis.setAnalysisRoots", "params": { "included": ["/home/not/existing"], "excluded": []}}
You'll see the log file file up as we infinitely try to rebuild the analysis roots and consume a lot of resources.
Although setting an analysis root to a non-existing directory is unlikely (though suspect deleting a valid root while the server is running will do the same), the original case here is where VS Code restores an unsaved file in a deleted directory at startup (for ex you had an open dirty file in /tmp and rebooted then re-opened VS Code) so we end up trying to create a root for it.
The reason this occurs is that creating a watcher on a folder that doesn't exist emits an error on the stream (on Linux) and triggers this code that wants to restart (because a watcher error means we can't trust our state of the world):
Rebuilding doesn't help because the same error will occur again, and we'll just sit in a loop doing this as fast as possible.
My proposed fix is to skip that rebuild if (error is PathNotFoundException). We're not going to get anything useful from the watcher after the error, and rebuilding isn't going to give us anything more. I believe we document that we handle directories created later, but as far as I know that isn't working (package:watcher doesn't support supporting watching non-existing directories so supporting this would be complicated, and potentially expensive if we have to end up watching grandparent folders, potentially even the root?).
Activity
zinface commentedon Nov 23, 2022
in /tmp/tmp.xxxx(
code \mktemp -d``) workspace, i edit a file . now restart my computer/vscode, Vscode has an unsaved file.. top.DanTup commentedon Nov 23, 2022
Can you run
code --statuswhile the memory usage is high and post the full output here?zinface commentedon Nov 23, 2022
I can explain the recurrence steps of this problem hand in hand:
my environment is Deepin Linux.
Let's reproduce the problem.
You use
mktemp -dor any other method to create a temporary folder, which we will delete later.This is a way to simulate the use of memory space (tmpfs) or to create temporary engineering projects under
/tmp.You go to the folder and use
dart create dart_projectto create a dart project and open it using VSCode.You find a dart file that exists in the project to edit it, but you do not save it, so we will directly close VSCode.
We delete the temporary directory in this step and simulate that the contents in the tmpfs are emptied after restart.
We open VSCode again, and VSCode will try to open the project before the last instance was closed.
Because we have deleted the folder, there is only one unsaved dart file in the window.
At this time, I found that the memory footprint on my machine was rising.
zinface commentedon Nov 23, 2022
Reply: information from
code --statusMemory 20%(It just started.)
Memory 40%
Memory 60%
Memory 80%
zinface commentedon Nov 23, 2022
I also repeated this question when I turned off all plug-ins and opened only one Dart plug-in.
DanTup commentedon Nov 23, 2022
@zinface thanks for the detailed steps. Unfortunately I haven't been able to reproduce it (although I'm on macOS, I don't know if that's a factor).
Is it possible you can set the analyzer instrumentation log in your VS Code User Settings:
https://dartcode.org/docs/logging/#analyzer-instrumentation
And then reproduce the issue, and see whether this log file is being written to as the memory usage is growing? You can attach the log or email it to me at [email protected] if it's not too big (but please reproduce this with a small sample app, because the log may contain parts of source code from the open files).
Thanks!
zinface commentedon Nov 24, 2022
I have e-mailed the information you mentioned, along with about 2 minutes of screenshot files (reproducing the problem).
DanTup commentedon Nov 28, 2022
@zinface thank you for the detailed video and logs! I think I see what's going on. When VS Code is opened with that unsaved file, the server tries to create a watcher for it - however the file does not exist on disk and this causes an error:
This server interprets this error as something it can recover from, so it tries again. And it keeps retrying repeatedly forever.
I'm not certain why the memory usage climbs so high, but now I understand the cause I might have better luck reproducing it so I can work on a fix. Thanks!
[-]Oh, My Computer is Die! Memory Done.[/-][+]analysis server consumes significant memory when VS Code restores a dirty file buffer for a project that no longer exists[/+]DanTup commentedon Dec 12, 2022
@zinface sorry for the delay. I'm working in this now but struggling to reproduce (even on Linux). The attachment links in you email have now expired - do you still have a copy you can re-send to me to [email protected] so I can understand what's different between my setup and yours? Thanks!
zinface commentedon Dec 17, 2022
I just saw the email now. Of course, it's already the weekend. I have re-sent the content I prepared last time.
zinface commentedon Dec 17, 2022
In addition, I have prepared a shorter video to provide a clearer review process.
DanTup commentedon Dec 19, 2022
@zinface thank you - I got the files and they are attached to the email this time, so there is no danger of them expiring :)
I'm out of the office for a little over Christmas, but I will try to get to the bottom of this in the new year. Thanks!
DanTup commentedon Feb 21, 2023
Sorry for the delay. I've been able to reproduce this in an Ubuntu VM. I'm not sure exactly what's going on yet, but will do some digging.
For my future reference, here's a log file that has some additional logging from running the server from source in the VM.
instrum.txt
DanTup commentedon Feb 21, 2023
Ok, I have a better idea of what's happening now. In the log there are two sets of errors alternating:
The first error ("Temporary watcher error") is the one we're handling while trying to build the context roots. We will try up to 5 times before giving up.
The second error is code that tries to handle errors generally and triggers a context rebuild. It's this one that's causing the rebuilds to occur in a loop, not the original "temporary watcher" one (but because it triggers a rebuild, it causes that code to run again, and the error to be logged).
I think there may be a difference in how these errors occur on Linux versus other platforms which is why it's only seen here, although it seems like we should detect if we get into a situation where we're endlessly trying to rebuild contexts (they're likely failing for a reason that isn't going to change if we try again).
Two thoughts:
DanTup commentedon Feb 22, 2023
(@bwilkerson @scheglov FYI.. I think I have a fix for this, but some background for when you get the review..)
I have a very easy way to reproduce what is essentially the same issue (it only occurs on Linux - there are some differences in how this kind of error is raised across platforms, I think it's been discussed before although I can't find the discussion):
dart language-server --protocol=analyzer --instrumentation-log-file=/tmp/analyzer.txttail -f /tmp/analyzer.txt{"id": "1", "method": "analysis.setAnalysisRoots", "params": { "included": ["/home/not/existing"], "excluded": []}}You'll see the log file file up as we infinitely try to rebuild the analysis roots and consume a lot of resources.
Although setting an analysis root to a non-existing directory is unlikely (though suspect deleting a valid root while the server is running will do the same), the original case here is where VS Code restores an unsaved file in a deleted directory at startup (for ex you had an open dirty file in
/tmpand rebooted then re-opened VS Code) so we end up trying to create a root for it.The reason this occurs is that creating a watcher on a folder that doesn't exist emits an error on the stream (on Linux) and triggers this code that wants to restart (because a watcher error means we can't trust our state of the world):
https://github.com/dart-lang/sdk/blob/8ef5104d05e2b43541a69640ac77b8619cb576f3/pkg/analysis_server/lib/src/context_manager.dart#L770-L777
Rebuilding doesn't help because the same error will occur again, and we'll just sit in a loop doing this as fast as possible.
My proposed fix is to skip that rebuild
if (error is PathNotFoundException). We're not going to get anything useful from the watcher after the error, and rebuilding isn't going to give us anything more. I believe we document that we handle directories created later, but as far as I know that isn't working (package:watcherdoesn't support supporting watching non-existing directories so supporting this would be complicated, and potentially expensive if we have to end up watching grandparent folders, potentially even the root?).DanTup commentedon Feb 22, 2023
This fix seems to work in my VM: https://dart-review.googlesource.com/c/sdk/+/284660
[analysis_server] Don't endlessly rebuild analysis contexts on PathNo…
DanTup commentedon Feb 28, 2023
This is fixed by dart-lang/sdk@e231f12. The change is in the Dart SDK, so will ship in a future Dart/Flutter release rather than a Dart-Code update.