Skip to content

Flutter or Dart has been leaking processes on Windows #29141

Closed
Closed
Flutter or Dart has been leaking processes on Windows#29141
Assignees
godofredoc
Labels
a: tests"flutter test", flutter_test, or one of our testsc: contributor-productivityTeam-specific productivity, code health, technical debt.c: flakeTests that sometimes, but not always, incorrectly passdependency: dartDart team may need to help usteam-infraOwned by Infrastructure team

Description

@dnfield
dnfield
opened on Mar 11, 2019

A routine cause of Windows devicelab failures at this point is that the flutter directory can't be deleted after tests because a flutter (dart.exe) process is still running that should have finished.

We could easily add a script to kill the remaining processes on Windows, but ideally we wouldn't be leaking the process to begin with.

We should:

  • update our tests on the devicelab to make sure we're not leaking any processes we control at the end. If we do, the test that leaked should fail rather than the next test that can't clean up.
  • Fix whatever bugs in the tool and/or dart that are causing this leak

Activity

  1. dnfield
    added
    a: tests"flutter test", flutter_test, or one of our tests
    c: contributor-productivityTeam-specific productivity, code health, technical debt.
    toolAffects the "flutter" command-line tool. See also t: labels.
    dependency: dartDart team may need to help us
    team: flakes
    on Mar 11, 2019
  2. dnfield
    added this to the milestone on Mar 11, 2019

  3. dnfield commented on Mar 12, 2019

    @dnfield
    dnfield
    on Mar 12, 2019
    ContributorAuthor

    We may be leaking processes on other platforms too, but Windows is particularly harmful because it causes the subsequent run to fail.

    On mac I think we've seen some leaks, which eventually makes the machine unreliable.

  4. dnfield commented on Mar 19, 2019

    @dnfield
    dnfield
    on Mar 19, 2019
    ContributorAuthor

    For now:

    • Kill off leaked processes at the end of a test and log warnings.
    • Add some test(s) to check for leaked process and mark them as flaky until we fix them.

    Eventually:

    • Fail any test that leaks a process.

  5. digiter commented on Jun 27, 2019

    @digiter
    digiter
    on Jun 27, 2019
    Contributor

    I found two leaked dart.exe processes on a Windows machine today that causes this issue, here are the command line of each one.

    The parent dart.exe has the command line of

    "C:\Users\Win1\.cocoon\flutter\bin\cache\dart-sdk\bin\dart.exe"  --packages="C:\Users\Win1\.cocoon\flutter\packages\flutter_tools\.packages"  "C:\Users\Win1\.cocoon\flutter\bin\cache\flutter_tools.snapshot" run --hot -d ZY2245DC4Z --benchmark --verbose --resident

    The child dart.exe has the command line of

    C:\Users\Win1\.cocoon\flutter\bin\cache\dart-sdk\bin\dart.EXE C:\Users\Win1\.cocoon\flutter\bin\cache\artifacts\engine\windows-x64\frontend_server.dart.snapshot --sdk-root C:\Users\Win1\.cocoon\flutter\bin\cache\artifacts\engine\common\flutter_patched_sdk/ --incremental --strong --target=flutter --output-dill build\app.dill --packages C:\Users\Win1\AppData\Local\Temp\edited_flutter_gallery\.packages --filesystem-scheme org-dartlang-root

    It looks like that the compiler doesn't exit properly in hot reload mode.

  6. jmagman commented on Jun 27, 2019

    @jmagman
    jmagman
    on Jun 27, 2019
    Member

    The child dart.exe has the command line of

    C:\Users\Win1\.cocoon\flutter\bin\cache\dart-sdk\bin\dart.EXE C:\Users\Win1\.cocoon\flutter\bin\cache\artifacts\engine\windows-x64\frontend_server.dart.snapshot --sdk-root C:\Users\Win1\.cocoon\flutter\bin\cache\artifacts\engine\common\flutter_patched_sdk/ --incremental --strong --target=flutter --output-dill build\app.dill --packages C:\Users\Win1\AppData\Local\Temp\edited_flutter_gallery\.packages --filesystem-scheme org-dartlang-root

    It looks like that the compiler doesn't exit properly in hot reload mode.

    That one looks similar to #34613.

  7. dnfield commented on Jun 27, 2019

    @dnfield
    dnfield
    on Jun 27, 2019
    ContributorAuthor

    Looks like that originates in hot_mode_tests.dart: https://github.com/flutter/flutter/blob/master/dev/devicelab/lib/tasks/hot_mode_tests.dart#L26

  8. tvolkert commented on Jun 27, 2019

    @tvolkert
    tvolkert
    on Jun 27, 2019
    Contributor

    @a-siva do you know who on the Dart team might be a good person to take a look?

  9. a-siva
    assigned
    zichangg
    on Jun 27, 2019

  10. aam commented on Jun 27, 2019

    @aam
    aam
    on Jun 27, 2019
    Member

    I found two leaked dart.exe processes on a Windows machine today that causes this issue, here are the command line of each one.

    The parent dart.exe has the command line of

    "C:\Users\Win1\.cocoon\flutter\bin\cache\dart-sdk\bin\dart.exe"  --packages="C:\Users\Win1\.cocoon\flutter\packages\flutter_tools\.packages"  "C:\Users\Win1\.cocoon\flutter\bin\cache\flutter_tools.snapshot" run --hot -d ZY2245DC4Z --benchmark --verbose --resident

    ...
    It looks like that the compiler doesn't exit properly in hot reload mode.

    This instance of dart.exe/flutter_tools is supposed to stay resident in memory driven by hot_mode_tests.dart. I would try to capture logs for hot_mode_test.dart and confirm that flutter_tools stdout that hot_mode_test sees matches it's expectations. In particular note how hot_mode_test only exists after what it believes it sees two successful reload requests. It might be that reload doesn't happen, it might be that output is structured slightly differently from hot_mode_tests expectations.

  11. digiter commented on Jun 28, 2019

    @digiter
    digiter
    on Jun 28, 2019
    Contributor

    I forgot to mention that this issue is flaky, it doesn't reproduce every time.

  12. 12 remaining items

  13. dnfield commented on Nov 15, 2019

    @dnfield
    dnfield
    on Nov 15, 2019
    ContributorAuthor

    Do we have any sense for how often it's happening now days, and how reproducible it is?

    If we're in a situation where we don't know what's causing this and it's extremely flaky, closing the tree over it seems excessive. If it's not normally happening and we think we've fixed the bugs, then we should close the tree if we reintroduce it.

  14. godofredoc commented on Nov 15, 2019

    @godofredoc
    godofredoc
    on Nov 15, 2019
    Contributor

    This is happening very frequently on Linux + Android tests. I've seen a few tests failing because the test finished and there were some processes still running. I'll add links when I find another one.

    I personally prefer the tests to fail rather than just hiding the error.

  15. keyonghan commented on Nov 21, 2019

    @keyonghan
    keyonghan
    on Nov 21, 2019
    Contributor

    Have been keeping an eye on leaking processes on Windows these days, but didn't notice any.

  16. godofredoc commented on Nov 21, 2019

    @godofredoc
    godofredoc
    on Nov 21, 2019
    Contributor

    Let's monitor windows for one more week, if there are not leaked processes we can close this one and open a new one for linux.

  17. zanderso commented on Jan 15, 2020

    @zanderso
    zanderso
    on Jan 15, 2020
    Member

    @godofredoc @keyonghan What's the status of this?

  18. zanderso
    removed
    toolAffects the "flutter" command-line tool. See also t: labels.
    on Jan 15, 2020
  19. zanderso
    assigned
    godofredoc
    and unassigned
    zichangg
    on Jan 15, 2020

  20. godofredoc commented on Jan 15, 2020

    @godofredoc
    godofredoc
    on Jan 15, 2020
    Contributor

    We can safely close this one start/kill the agent process differently and it has not been leaking any processes in the last few months.

  21. godofredoc
    closed this as completedon Jan 15, 2020

  22. kontinuity commented on Jun 27, 2020

    @kontinuity
    kontinuity
    on Jun 27, 2020
     ~ ps -ef | grep dart
  501  6697  6679   0 Sun10AM ??         0:00.73 /Users/redacted/apps/flutter/bin/cache/dart-sdk/bin/dart --disable-dart-dev --packages=/Users/redacted/apps/flutter/packages/flutter_tools/.packages /Users/redacted/apps/flutter/bin/cache/flutter_tools.snapshot --no-color pub global run devtools --machine --port=0
  501  6710  6697   0 Sun10AM ??         0:02.39 /Users/redacted/apps/flutter/bin/cache/dart-sdk/bin/dart /Users/redacted/apps/flutter/bin/cache/dart-sdk/bin/snapshots/pub.dart.snapshot global run devtools --machine --port=0
  501 11254 11236   0 Sun11AM ??         0:00.71 /Users/redacted/apps/flutter/bin/cache/dart-sdk/bin/dart --disable-dart-dev --packages=/Users/redacted/apps/flutter/packages/flutter_tools/.packages /Users/redacted/apps/flutter/bin/cache/flutter_tools.snapshot --no-color pub global run devtools --machine --port=0
  501 11264 11254   0 Sun11AM ??         0:02.29 /Users/redacted/apps/flutter/bin/cache/dart-sdk/bin/dart /Users/redacted/apps/flutter/bin/cache/dart-sdk/bin/snapshots/pub.dart.snapshot global run devtools --machine --port=0
  501 33484 33466   0 Sun05PM ??         0:00.75 /Users/redacted/apps/flutter/bin/cache/dart-sdk/bin/dart --disable-dart-dev --packages=/Users/redacted/apps/flutter/packages/flutter_tools/.packages /Users/redacted/apps/flutter/bin/cache/flutter_tools.snapshot --no-color pub global run devtools --machine --port=0
  501 33494 33484   0 Sun05PM ??         0:02.65 /Users/redacted/apps/flutter/bin/cache/dart-sdk/bin/dart /Users/redacted/apps/flutter/bin/cache/dart-sdk/bin/snapshots/pub.dart.snapshot global run devtools --machine --port=0
  501 44299 44281   0 Sun07PM ??         0:00.82 /Users/redacted/apps/flutter/bin/cache/dart-sdk/bin/dart --disable-dart-dev --packages=/Users/redacted/apps/flutter/packages/flutter_tools/.packages /Users/redacted/apps/flutter/bin/cache/flutter_tools.snapshot --no-color pub global run devtools --machine --port=0
  501 44311 44299   0 Sun07PM ??         0:02.27 /Users/redacted/apps/flutter/bin/cache/dart-sdk/bin/dart /Users/redacted/apps/flutter/bin/cache/dart-sdk/bin/snapshots/pub.dart.snapshot global run devtools --machine --port=0
  501 82834 82816   0 Mon12PM ??         0:00.76 /Users/redacted/apps/flutter/bin/cache/dart-sdk/bin/dart --disable-dart-dev --packages=/Users/redacted/apps/flutter/packages/flutter_tools/.packages /Users/redacted/apps/flutter/bin/cache/flutter_tools.snapshot --no-color pub global run devtools --machine --port=0
  501 82844 82834   0 Mon12PM ??         0:02.31 /Users/redacted/apps/flutter/bin/cache/dart-sdk/bin/dart /Users/redacted/apps/flutter/bin/cache/dart-sdk/bin/snapshots/pub.dart.snapshot global run devtools --machine --port=0

    Intellij Idea, AVD, everything else was shutdown at the time.

  23. jmagman commented on Jun 29, 2020

    @jmagman
    jmagman
    on Jun 29, 2020
    Member

    @kontinuity Can you file a new issue at https://github.com/flutter/devtools/issues? This issue wasn't tracking devtools but they can take a look at that.

  24. github-actions commented on Aug 14, 2021

    @github-actions
    github-actionsbot
    on Aug 14, 2021

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  25. github-actions
    locked as resolved and limited conversation to collaborators on Aug 14, 2021
  26. flutter-triage-bot
    added
    c: flakeTests that sometimes, but not always, incorrectly pass
    on Jul 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

a: tests"flutter test", flutter_test, or one of our testsc: contributor-productivityTeam-specific productivity, code health, technical debt.c: flakeTests that sometimes, but not always, incorrectly passdependency: dartDart team may need to help usteam-infraOwned by Infrastructure team

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions