Mac web_tool_tests is 5.05% flaky #121708

Activity

1. fluttergithubbot

   added

   P1

   c: flakeTests that sometimes, but not always, incorrectly passTests that sometimes, but not always, incorrectly pass

   team: flakes

   toolAffects the "flutter" command-line tool. See also t: labels.Affects the "flutter" command-line tool. See also t: labels.

   on Mar 1, 2023
2. fluttergithubbot

   assigned

   zanderso

   on Mar 1, 2023
3. zanderso commented on Mar 1, 2023

   

   zanderso

   on Mar 1, 2023

   Last edited by zanderso

   Member

   More actions

   All flakes were:

   02:21 +4 ~1 -1: /Volumes/Work/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/hot_reload_web_test.dart: Hot reload (index.html: flutter.js (promises)): newly added code executes during hot restart [E]
     TimeoutException after 0:00:15.000000: Future not completed
     
   
   To run this test again: /Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/dart test /Volumes/Work/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/hot_reload_web_test.dart -p vm --plain-name 'Hot reload (index.html: flutter.js (promises)): newly added code executes during hot restart'
   

   All the flakyness here happened while #121215 was in the tree. I don't know whether it is related, but it is suspicious. Routing to platform: web and dependency: dart. cc @bkonyi
4. zanderso

   removed their assignment

   on Mar 1, 2023
5. zanderso

   added

   dependency: dartDart team may need to help usDart team may need to help us

   platform-webWeb applications specificallyWeb applications specifically

   and removed

   toolAffects the "flutter" command-line tool. See also t: labels.Affects the "flutter" command-line tool. See also t: labels.

   on Mar 1, 2023
6. a-siva commented on Mar 6, 2023

   

   a-siva

   on Mar 6, 2023

   Contributor

   More actions

   //cc @annagrin @sigmundch
7. a-siva commented on Mar 6, 2023

   

   a-siva

   on Mar 6, 2023

   Contributor

   More actions

   We seem to have a test that flakily times out

   02:21 +4 ~1 -1: /Volumes/Work/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/hot_reload_web_test.dart: Hot reload (index.html: flutter.js (promises)): newly added code executes during hot restart [E]
     TimeoutException after 0:00:15.000000: Future not completed
     
   
   To run this test again: /Volumes/Work/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/dart test /Volumes/Work/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/hot_reload_web_test.dart -p vm --plain-name 'Hot reload (index.html: flutter.js (promises)): newly added code executes during hot restart
   ```'
   
8. yjbanov commented on Mar 6, 2023

   

   yjbanov

   on Mar 6, 2023

   Contributor

   More actions

   Deduplicating into #120014
9. yjbanov

   closed this as completedon Mar 6, 2023
10. zanderso commented on Mar 6, 2023

    

    zanderso

    on Mar 6, 2023

    Member

    More actions

    @yjbanov The flakes in #120014 were due to running slowly on arm64 machines, but the flakes reported in this issue were all on x64 machines, so I'm going to re-open this as a separate issue.
11. 5 remaining items

    Load more

12. ditman commented on Mar 8, 2023

    

    ditman

    on Mar 8, 2023

    Member

    More actions

    current flaky ratio for the past (up to) 100 commits is 1.01%. This was cut as it was > 5%.

    Was the offending test disabled, or did this get fixed?
13. annagrin commented on Mar 9, 2023

    

    annagrin

    on Mar 9, 2023

    Last edited by annagrin

    Contributor

    More actions

    @ditman the ratio was higher due to the same test failing more often on arm64 machines, I think the web tests were disabled on them altogether. amd64 runs still show the failure, just less often. Let's wait for now and see if another flake happens, now with error logging.

    👍React with 👍1Reacted by David Iglesias
14. annagrin

    added

    P1High-priority issues at the top of the work listHigh-priority issues at the top of the work list

    and removed

    P1

    on Mar 9, 2023
15. annagrin commented on Mar 9, 2023

    

    annagrin

    on Mar 9, 2023

    Contributor

    More actions

    Lowering the priority to p3 due to lower rate of flakes.
16. ditman commented on Mar 9, 2023

    

    ditman

    on Mar 9, 2023

    Last edited by ditman

    Member

    More actions

    @annagrin, @mdebbar is working on a fix for the issue you linked above, it's here: flutter-team-archive/engine#40154

    I think he's working to find a backwards-compatible way of landing it (without requiring to roll and deploy a new canvaskit version, if I understood correctly!)
17. annagrin commented on Mar 11, 2023

    

    annagrin

    on Mar 11, 2023

    Contributor

    More actions

    A new flake has happened, this time with a trace:

    [        ] ChromeProxyService: Cannot get evaluation result for isolate 1:  isolate exited.
    [  +46 ms] DwdsVmClient: Successful hot restart
    [   +1 ms] ChromeProxyService: Caching package:flutter/src/widgets/widget_inspector.dart in expression compiler worker
    [        ] DevHandler: VmService proxy responded with an error:
    {jsonrpc: 2.0, id: 15, error: {code: -32603, message: getIsolate: [Sentinel kind: Collected, valueAsString: Unrecognized isolateId: 1] from getIsolate(), data: {details: #0      ChromeProxyService._throwSentinel (package:dwds/src/services/chrome_proxy_service.dart:1154:5)
    #1      ChromeProxyService._checkIsolate (package:dwds/src/services/chrome_proxy_service.dart:1146:7)
    #2      ChromeProxyService.getIsolate.<anonymous closure> (package:dwds/src/services/chrome_proxy_service.dart:569:7)
    <asynchronous suspension>
    #3      StackZoneSpecification._registerUnaryCallback.<anonymous closure> (package:stack_trace/src/stack_zone_specification.dart:124:15)
    <asynchronous suspension>
    }}}
    

    It looks like a race condition between the hot restart and something that calls getIsolate with the old isolate ID - could be flutter test or flutter tools... I can try this on my machine with adding delays in some places to simulate the race.

    👍React with 👍1Reacted by David Iglesias
18. fluttergithubbot commented on Mar 15, 2023

    

    fluttergithubbot

    on Mar 15, 2023

    ContributorAuthor

    More actions

    [prod pool] current flaky ratio for the past (up to) 100 commits is 3.06%. Flaky number: 3; total number: 98.
    One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20web_tool_tests/11165
    Commit: 7717839
    Flaky builds:
    https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20web_tool_tests/11165
    https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20web_tool_tests/11152
    https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20web_tool_tests/11149

    Recent test runs:
    https://flutter-dashboard.appspot.com/#/build?taskFilter=Mac%20web_tool_tests
19. fluttergithubbot

    added

    P1

    and removed

    P1High-priority issues at the top of the work listHigh-priority issues at the top of the work list

    on Mar 15, 2023
20. christopherfujino

    closed this as completedin #122776on Mar 16, 2023
21. github-actions commented on Mar 30, 2023

    

    github-actionsbot

    on Mar 30, 2023

    More actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.
22. github-actions

    locked as resolved and limited conversation to collaborators on Mar 30, 2023
23. flutter-triage-bot

    added

    P0Critical issues such as a build break or regressionCritical issues such as a build break or regression

    and removed

    P1

    on Jun 28, 2023