Skip to content

Linux web_tool_tests is 2.44% flaky #93256

Closed
Closed
Linux web_tool_tests is 2.44% flaky#93256
Assignees
annagrin
Labels
P1High-priority issues at the top of the work listc: flakeTests that sometimes, but not always, incorrectly passplatform-webWeb applications specificallytoolAffects the "flutter" command-line tool. See also t: labels.

Description

@fluttergithubbot
fluttergithubbot
opened on Nov 8, 2021

The post-submit test builder Linux web_tool_tests had a flaky ratio 2.44% for the past 15 days, which is above our 2.00% threshold.

One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_tool_tests/4048,%204047
Commit: a4b5123
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_tool_tests/4048
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_tool_tests/4047
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_tool_tests/3984
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_tool_tests/3980

Succeeded builds (3 most recent):
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_tool_tests/4075
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_tool_tests/4074
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_tool_tests/4073

Please follow https://github.com/flutter/flutter/wiki/Reducing-Test-Flakiness#fixing-flaky-tests to fix the flakiness and enable the test back after validating the fix (internal dashboard to validate: go/flutter_test_flakiness).

Activity

  1. fluttergithubbot
    added
    P1
    c: flakeTests that sometimes, but not always, incorrectly pass
    team: flakes
    toolAffects the "flutter" command-line tool. See also t: labels.
    on Nov 8, 2021
  2. fluttergithubbot
    assigned
    zanderso
    on Nov 8, 2021

  3. zanderso commented on Nov 8, 2021

    @zanderso
    zanderso
    on Nov 8, 2021
    Member

    The flaky error is the following:

    03:01 +7 ~2 -1: test/web.shard/web_run_test.dart: flutter run works on web devices with a unary main function [E]                                                                                      
  Exception: Exception: Received app.stop event while waiting for app.started event
  
  [  +99 ms] SocketException: Failed to create server socket (OS Error: The shared flag to bind() needs to be `true` if binding multiple times on the same (address, port) combination.), address = ::1, port = 34463
             #0      _NativeSocket.bind (dart:io-patch/socket_patch.dart:988:7)
             <asynchronous suspension>
             #1      HttpMultiServer._loopback (package:http_multi_server/http_multi_server.dart:183:22)
             <asynchronous suspension>
             #2      ExtensionBackend.start (package:dwds/src/servers/extension_backend.dart:57:18)
             <asynchronous suspension>
             #3      Dwds.start (package:dwds/dwds.dart:138:26)
             <asynchronous suspension>
             #4      WebAssetServer.start (package:flutter_tools/src/isolated/devfs_web.dart:274:23)
             <asynchronous suspension>
             #5      WebDevFS.create (package:flutter_tools/src/isolated/devfs_web.dart:727:22)
             <asynchronous suspension>
             #6      ResidentWebRunner.run.<anonymous closure> (package:flutter_tools/src/isolated/resident_web_runner.dart:285:25)
             <asynchronous suspension>
             #7      asyncGuard.<anonymous closure> (package:flutter_tools/src/base/async_guard.dart:111:24)
             <asynchronous suspension>
  
  Failed to establish connection with the application instance in Chrome.
  This can happen if the websocket connection used by the web tooling is unable to correctly establish a connection, for example due to a firewall.
  
  #0      throwToolExit (package:flutter_tools/src/base/common.dart:10:3)
  #1      ResidentWebRunner.run (package:flutter_tools/src/isolated/resident_web_runner.dart:336:7)
  <asynchronous suspension>
  #2      AppDomain.launch.<anonymous closure> (package:flutter_tools/src/commands/daemon.dart:599:9)
  <asynchronous suspension>
  #3      AppDomain.launch (package:flutter_tools/src/commands/daemon.dart:597:5)
  <asynchronous suspension>
  #4      RunCommand.runCommand (package:flutter_tools/src/commands/run.dart:558:15)
  <asynchronous suspension>
  #5      FlutterCommand.run.<anonymous closure> (package:flutter_tools/src/runner/flutter_command.dart:1135:27)
  <asynchronous suspension>
  #6      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
  <asynchronous suspension>
  #7      CommandRunner.runCommand (package:args/command_runner.dart:209:13)
  <asynchronous suspension>
  #8      FlutterCommandRunner.runCommand.<anonymous closure> (package:flutter_tools/src/runner/flutter_command_runner.dart:281:9)
  <asynchronous suspension>
  #9      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
  <asynchronous suspension>
  #10     FlutterCommandRunner.runCommand (package:flutter_tools/src/runner/flutter_command_runner.dart:229:5)
  <asynchronous suspension>
  #11     run.<anonymous closure>.<anonymous closure> (package:flutter_tools/runner.dart:62:9)
  <asynchronous suspension>
  #12     AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
  <asynchronous suspension>
  #13     main (package:flutter_tools/executable.dart:93:3)
  <asynchronous suspension>
  4. zanderso
    assigned
    annagrin
    and unassigned
    zanderso
    on Nov 8, 2021
  5. christopherfujino
    added
    platform-webWeb applications specifically
    on Nov 9, 2021

  6. zanderso commented on Nov 9, 2021

    @zanderso
    zanderso
    on Nov 9, 2021
    Member

    @sigmundch Sorry if my information is stale, but is @annagrin still the right person to look into dwds issues that come up in the flutter tool?

  7. sigmundch commented on Nov 10, 2021

    @sigmundch
    sigmundch
    on Nov 10, 2021
    Contributor

    👍 @annagrin is the right contact. FYI she is out a few days so this will likely not be acted on immediately. Can you clarify on the P1 prioritization, just want to align on expectations. It's my understanding that flutter-web-tools flakiness issues have been an ongoing battle and not something I expect to see it fixed so quickly.

  8. zanderso commented on Nov 10, 2021

    @zanderso
    zanderso
    on Nov 10, 2021
    Member

    @sigmundch Flutter's flaky test process (go/reduce_flutter_flakiness) treats tests that flake at a rate > 2% as P1 issues. This is currently the only open issue related to flutter-web-tools.

  9. zanderso commented on Nov 10, 2021

    @zanderso
    zanderso
    on Nov 10, 2021
    Member

    @yjbanov I believe @annagrin won't be able to take a look at this until next week. Do you know of anyone else who might be familiar with this area?

  10. annagrin commented on Nov 18, 2021

    @annagrin
    annagrin
    on Nov 18, 2021
    Contributor

    I see two types of errors in the logs:

    1. failure to start dwds's extension backend

      stack trace 
      [  +92 ms] SocketException: Failed to create server socket (OS Error: The shared flag to bind() needs to be `true` if binding multiple times on the same (address, port) combination.), address = ::1, port = 42919
             #0      _NativeSocket.bind (dart:io-patch/socket_patch.dart:988:7)
             <asynchronous suspension>
             #1      HttpMultiServer._loopback (package:http_multi_server/http_multi_server.dart:183:22)
             <asynchronous suspension>
             #2      ExtensionBackend.start (package:dwds/src/servers/extension_backend.dart:57:18)
             <asynchronous suspension>
             #3      Dwds.start (package:dwds/dwds.dart:138:26)
             <asynchronous suspension>
             #4      WebAssetServer.start (package:flutter_tools/src/isolated/devfs_web.dart:274:23)
             <asynchronous suspension>
             #5      WebDevFS.create (package:flutter_tools/src/isolated/devfs_web.dart:727:22)
             <asynchronous suspension>
             #6      ResidentWebRunner.run.<anonymous closure> (package:flutter_tools/src/isolated/resident_web_runner.dart:285:25)
             <asynchronous suspension>
             #7      asyncGuard.<anonymous closure> (package:flutter_tools/src/base/async_guard.dart:111:24)
             <asynchronous suspension>

    Opened issue: dart-lang/webdev#1450

    1. failure to connect to chrome

      stack trace 
      [  +26 ms] SocketException: OS Error: Connection refused, errno = 111, address = localhost, port = 37232
             #0      _NativeSocket.startConnect (dart:io-patch/socket_patch.dart:681:35)
             #1      _RawSocket.startConnect (dart:io-patch/socket_patch.dart:1808:26)
             #2      RawSocket.startConnect (dart:io-patch/socket_patch.dart:27:23)
             #3      Socket._startConnect (dart:io-patch/socket_patch.dart:2027:22)
             #4      Socket.startConnect (dart:io/socket.dart:784:21)
             #5      _ConnectionTarget.connect (dart:_http/http_impl.dart:2439:18)
             #6      _HttpClient._getConnection.connect (dart:_http/http_impl.dart:2808:12)
             #7      _HttpClient._getConnection (dart:_http/http_impl.dart:2813:12)
             #8      _HttpClient._openUrl (dart:_http/http_impl.dart:2697:12)
             #9      _HttpClient.getUrl (dart:_http/http_impl.dart:2574:48)
             #10     ChromeConnection.getUrl (package:webkit_inspection_protocol/webkit_inspection_protocol.dart:75:33)
             #11     ChromeConnection.getTabs (package:webkit_inspection_protocol/webkit_inspection_protocol.dart:41:26)
             #12     ChromiumLauncher._connect (package:flutter_tools/src/web/chrome.dart:398:39)
             #13     ChromiumLauncher.launch (package:flutter_tools/src/web/chrome.dart:229:12)
             <asynchronous suspension>
             #14     ChromiumDevice.startApp (package:flutter_tools/src/web/web_device.dart:134:17)
             <asynchronous suspension>
             #15     ResidentWebRunner.run.<anonymous closure> (package:flutter_tools/src/isolated/resident_web_runner.dart:307:9)
             <asynchronous suspension>
             #16     asyncGuard.<anonymous closure> (package:flutter_tools/src/base/async_guard.dart:111:24)
             <asynchronous suspension>

    Opened issue: #93889

  11. danagbemava-nc
    added
    passed first triage
    on Nov 19, 2021

  12. 4 remaining items

  13. fluttergithubbot commented on Dec 13, 2021

    @fluttergithubbot
    fluttergithubbot
    on Dec 13, 2021
    ContributorAuthor

    Current flaky ratio for the past (up to) 100 commits is 1.00%.
    One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_tool_tests/4272
    Commit: 16ec56e
    Flaky builds:
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_tool_tests/4272

    Recent test runs:
    https://flutter-dashboard.appspot.com/#/build?taskFilter=Linux%20web_tool_tests

  14. annagrin
    added
    P1High-priority issues at the top of the work list
    and removed
    P1
    on Dec 14, 2021

  15. annagrin commented on Dec 14, 2021

    @annagrin
    annagrin
    on Dec 14, 2021
    Contributor

    Changed priority to P3 since the flakiness is now below 2%.

  16. fluttergithubbot commented on Dec 20, 2021

    @fluttergithubbot
    fluttergithubbot
    on Dec 20, 2021
    ContributorAuthor

    Current flaky ratio for the past (up to) 100 commits is 0.00%. Flaky number: 0; total number: 100.

  17. fluttergithubbot commented on Dec 27, 2021

    @fluttergithubbot
    fluttergithubbot
    on Dec 27, 2021
    ContributorAuthor

    Current flaky ratio for the past (up to) 100 commits is 1.00%. Flaky number: 1; total number: 100.
    One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_tool_tests/4463
    Commit: c584a5e
    Flaky builds:
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_tool_tests/4463

    Recent test runs:
    https://flutter-dashboard.appspot.com/#/build?taskFilter=Linux%20web_tool_tests

  18. zanderso commented on Dec 28, 2021

    @zanderso
    zanderso
    on Dec 28, 2021
    Member

    Build 4463 looks like the tool gets an unexpected response when trying to connect to the chrome debug port:

    Unable to connect to Chrome debug port: 37021
   FormatException: Unexpected character (at character 1)
  <html>

  19. fluttergithubbot commented on Jan 3, 2022

    @fluttergithubbot
    fluttergithubbot
    on Jan 3, 2022
    ContributorAuthor

    Current flaky ratio for the past (up to) 100 commits is 1.00%. Flaky number: 1; total number: 100.
    One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_tool_tests/4463
    Commit: c584a5e
    Flaky builds:
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_tool_tests/4463

    Recent test runs:
    https://flutter-dashboard.appspot.com/#/build?taskFilter=Linux%20web_tool_tests

  20. sigmundch commented on Jan 6, 2022

    @sigmundch
    sigmundch
    on Jan 6, 2022
    Contributor

    Indeed - looks like build 4463 is the same as #93256 (comment) (tracked also in #94886)

  21. fluttergithubbot commented on Jan 10, 2022

    @fluttergithubbot
    fluttergithubbot
    on Jan 10, 2022
    ContributorAuthor

    Current flaky ratio for the past (up to) 100 commits is 0.00%. Flaky number: 0; total number: 100.

  22. annagrin commented on Jan 10, 2022

    @annagrin
    annagrin
    on Jan 10, 2022
    Contributor

    The remaining issue is tracked in #94886. The flakiness does not appear to be critical at this point.

  23. zanderso commented on Jan 10, 2022

    @zanderso
    zanderso
    on Jan 10, 2022
    Member

    Closing based on comment in #93256 (comment)

  24. zanderso
    closed this as completedon Jan 10, 2022

  25. github-actions commented on Jan 24, 2022

    @github-actions
    github-actionsbot
    on Jan 24, 2022

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  26. github-actions
    locked as resolved and limited conversation to collaborators on Jan 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P1High-priority issues at the top of the work listc: flakeTests that sometimes, but not always, incorrectly passplatform-webWeb applications specificallytoolAffects the "flutter" command-line tool. See also t: labels.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions