Skip to content

Windows_android flavors_test_win is severe flaky #107681

Closed
Closed
Windows_android flavors_test_win is severe flaky#107681
Assignees
dkwingsmt
Labels
P1High-priority issues at the top of the work listc: flakeTests that sometimes, but not always, incorrectly passframeworkflutter/packages/flutter repository. See also f: labels.

Description

@keyonghan
keyonghan
opened on Jul 14, 2022

Windows_android flavors_test_win flakes 4 times today:

https://ci.chromium.org/ui/p/flutter/builders/prod/Windows_android%20flavors_test_win/4957/overview
https://ci.chromium.org/ui/p/flutter/builders/prod/Windows_android%20flavors_test_win/4953/overview
https://ci.chromium.org/ui/p/flutter/builders/prod/Windows_android%20flavors_test_win/4945/overview
https://ci.chromium.org/ui/p/flutter/builders/prod/Windows_android%20flavors_test_win/4944/overview

Error message:

_test_win] [STDOUT] stdout: [        ] executing: C:\opt\s\w\ir\cache\android\sdk\platform-tools\adb.exe -s ZY223C7M4J shell am force-stop com.yourcompany.flavors
[flavors_test_win] [STDOUT] stdout: [        ] test 0: Test process is no longer needed by test harness
[flavors_test_win] [STDOUT] stdout: [+1155 ms] executing: C:\opt\s\w\ir\cache\android\sdk\platform-tools\adb.exe -s ZY223C7M4J uninstall com.yourcompany.flavors
[flavors_test_win] [STDOUT] stdout: [ +554 ms] Success
[flavors_test_win] [STDOUT] stdout: [        ] test 0: deleting temporary directory
[flavors_test_win] [STDOUT] stdout: [        ] test 0: finished
[flavors_test_win] [STDOUT] stdout: 
[flavors_test_win] [STDOUT] stdout: 00:55 +0 -1: Flavor Test check flavor - did not complete [E]                                                                                                                                           
[flavors_test_win] [STDOUT] stdout: 
[flavors_test_win] [STDOUT] stdout: 00:55 +0 -1: (tearDownAll) - did not complete [E]                                                                                                                                                      
[flavors_test_win] [STDOUT] stdout: 
[flavors_test_win] [STDOUT] stdout: 00:55 +0 -1: Some tests failed.                                                                                                                                                                        
[flavors_test_win] [STDOUT] stdout: [   +7 ms] test package returned with exit code 1
[flavors_test_win] [STDOUT] stdout: [   +2 ms] "flutter test" took 56,478ms.
[flavors_test_win] [STDOUT] stderr: [   +3 ms] 
[flavors_test_win] [STDOUT] stderr:            #0      throwToolExit (package:flutter_tools/src/base/common.dart:10:3)
[flavors_test_win] [STDOUT] stderr:            #1      TestCommand.runCommand (package:flutter_tools/src/commands/test.dart:468:7)
[flavors_test_win] [STDOUT] stderr:            <asynchronous suspension>
[flavors_test_win] [STDOUT] stderr:            #2      FlutterCommand.run.<anonymous closure> (package:flutter_tools/src/runner/flutter_command.dart:1209:27)
[flavors_test_win] [STDOUT] stderr:            <asynchronous suspension>
[flavors_test_win] [STDOUT] stderr:            #3      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
[flavors_test_win] [STDOUT] stderr:            <asynchronous suspension>
[flavors_test_win] [STDOUT] stderr:            #4      CommandRunner.runCommand (package:args/command_runner.dart:209:13)
[flavors_test_win] [STDOUT] stderr:            <asynchronous suspension>
[flavors_test_win] [STDOUT] stderr:            #5      FlutterCommandRunner.runCommand.<anonymous closure> (package:flutter_tools/src/runner/flutter_command_runner.dart:281:9)
[flavors_test_win] [STDOUT] stderr:            <asynchronous suspension>
[flavors_test_win] [STDOUT] stderr:            #6      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
[flavors_test_win] [STDOUT] stderr:            <asynchronous suspension>
[flavors_test_win] [STDOUT] stderr:            #7      FlutterCommandRunner.runCommand (package:flutter_tools/src/runner/flutter_command_runner.dart:229:5)
[flavors_test_win] [STDOUT] stderr:            <asynchronous suspension>
[flavors_test_win] [STDOUT] stderr:            #8      run.<anonymous closure>.<anonymous closure> (package:flutter_tools/runner.dart:62:9)
[flavors_test_win] [STDOUT] stderr:            <asynchronous suspension>
[flavors_test_win] [STDOUT] stderr:            #9      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
[flavors_test_win] [STDOUT] stderr:            <asynchronous suspension>
[flavors_test_win] [STDOUT] stderr:            #10     main (package:flutter_tools/executable.dart:91:3)
[flavors_test_win] [STDOUT] stderr:            <asynchronous suspension>
[flavors_test_win] [STDOUT] stderr: 
[flavors_test_win] [STDOUT] stderr: 
[flavors_test_win] [STDOUT] stdout: [   +1 ms] ensureAnalyticsSent: 0ms
[flavors_test_win] [STDOUT] stdout: [        ] Running shutdown hooks
[flavors_test_win] [STDOUT] stdout: [        ] Shutdown hooks complete
[flavors_test_win] [STDOUT] stdout: [        ] exiting with code 1

Activity

  1. keyonghan commented on Jul 14, 2022

    @keyonghan
    keyonghan
    on Jul 14, 2022
    ContributorAuthor

    As all the flakes happened on only window4, @zanderso do you mind taking a look to see if this is related to test or testbed? At the same time, I manually quarantined the testbed: #107682.

  2. keyonghan
    assigned
    zanderso
    on Jul 14, 2022
  3. keyonghan
    added
    toolAffects the "flutter" command-line tool. See also t: labels.
    P1
    c: flakeTests that sometimes, but not always, incorrectly pass
    on Jul 14, 2022

  4. zanderso commented on Jul 15, 2022

    @zanderso
    zanderso
    on Jul 15, 2022
    Member

    All failures are:

    [flavors_test_win] [STDOUT] stdout:   Failed to load "C:\opt\s\w\ir\x\w\recipe_cleanup\tmp0l5a2s0v\flutter sdk\dev\integration_tests\flavors\integration_test\integration_test.dart": Null check operator used on a null value
[flavors_test_win] [STDOUT] stdout:   package:flutter_test/src/widget_tester.dart 828:57  WidgetTester.dispatchEvent
[flavors_test_win] [STDOUT] stdout:   package:flutter_test/src/binding.dart 1653:34       LiveTestWidgetsFlutterBinding.dispatchEvent
[flavors_test_win] [STDOUT] stdout:   package:flutter/src/gestures/binding.dart 395:7     GestureBinding._handlePointerEventImmediately
[flavors_test_win] [STDOUT] stdout:   package:flutter/src/gestures/binding.dart 357:5     GestureBinding.handlePointerEvent
[flavors_test_win] [STDOUT] stdout:   package:flutter_test/src/binding.dart 1636:25       LiveTestWidgetsFlutterBinding.handlePointerEvent.<fn>
[flavors_test_win] [STDOUT] stdout:   package:flutter_test/src/binding.dart 536:11        TestWidgetsFlutterBinding.withPointerEventSource
[flavors_test_win] [STDOUT] stdout:   package:flutter_test/src/binding.dart 1635:11       LiveTestWidgetsFlutterBinding.handlePointerEvent
[flavors_test_win] [STDOUT] stdout:   package:flutter/src/gestures/binding.dart 314:7     GestureBinding._flushPointerEventQueue
[flavors_test_win] [STDOUT] stdout:   package:flutter/src/gestures/binding.dart 295:7     GestureBinding._handlePointerDataPacket

    Routing to @dkwingsmt who is in the blame for the line with the null check error.

  5. zanderso
    added
    frameworkflutter/packages/flutter repository. See also f: labels.
    and removed
    toolAffects the "flutter" command-line tool. See also t: labels.
    on Jul 15, 2022
  6. zanderso
    assigned
    dkwingsmt
    and unassigned
    zanderso
    on Jul 15, 2022

  7. dkwingsmt commented on Jul 20, 2022

    @dkwingsmt
    dkwingsmt
    on Jul 20, 2022
    Contributor

    The code is assuming that TestWidgetsFlutterBinding.instance.renderViewElement is never null. It's easy for me to loosen the assertion (resulting in a null for innerTargetElement), but this is probably wrong, and will just move the flakiness to a deeper place. It seems that the pointer event sometimes occurs before a renderViewElement is established.

    Is it possibly caused by the first frame being too late?

    I'll need some help from whoever knows this test and the testing architecture better. Maybe @zanderso ?

  8. Hixie commented on Jul 26, 2022

    @Hixie
    Hixie
    on Jul 26, 2022
    Contributor

    cc @goderbauer

  9. goderbauer commented on Jul 26, 2022

    @goderbauer
    goderbauer
    on Jul 26, 2022
    Member

    Is it possibly caused by the first frame being too late?

    @dkwingsmt Could we add some logs to the test to verify this hypothesis?

  10. dkwingsmt commented on Aug 8, 2022

    @dkwingsmt
    dkwingsmt
    on Aug 8, 2022
    Contributor

    I tried to come up with #109148 (is it what you mean?). But it fails with the following error:

    [flavors_test_win] [STDOUT] stdout: 00:14 +0 -1: loading C:\Users\DKWings\src\flutter\dev\integration_tests\flavors\integration_test\integration_test.dart [E]

[flavors_test_win] [STDOUT] stdout:   Failed to load "C:\Users\DKWings\src\flutter\dev\integration_tests\flavors\integration_test\integration_test.dart": 'package:flutter/src/foundation/binding.dart': Failed assertion: line 146 pos 12: '_debugInitializedType == null': is not true.
[flavors_test_win] [STDOUT] stdout:   dart:core                                                                                                _AssertionError._throwNew
[flavors_test_win] [STDOUT] stdout:   package:flutter/src/foundation/binding.dart 146:12                                                       new BindingBase
[flavors_test_win] [STDOUT] stdout:   package:flutter_test/src/binding.dart                                                                    new _TestWidgetsFlutterBinding&BindingBase&SchedulerBinding
[flavors_test_win] [STDOUT] stdout:   package:flutter_test/src/binding.dart                                                                    new _TestWidgetsFlutterBinding&BindingBase&SchedulerBinding&ServicesBinding
[flavors_test_win] [STDOUT] stdout:   package:flutter_test/src/binding.dart                                                                    new _TestWidgetsFlutterBinding&BindingBase&SchedulerBinding&ServicesBinding&GestureBinding
[flavors_test_win] [STDOUT] stdout:   package:flutter_test/src/binding.dart                                                                    new _TestWidgetsFlutterBinding&BindingBase&SchedulerBinding&ServicesBinding&GestureBinding&SemanticsBinding
[flavors_test_win] [STDOUT] stdout:   package:flutter_test/src/binding.dart                                                                    new _TestWidgetsFlutterBinding&BindingBase&SchedulerBinding&ServicesBinding&GestureBinding&SemanticsBinding&RendererBinding
[flavors_test_win] [STDOUT] stdout:   package:flutter_test/src/binding.dart                                                                    new _TestWidgetsFlutterBinding&BindingBase&SchedulerBinding&ServicesBinding&GestureBinding&SemanticsBinding&RendererBinding&PaintingBinding
[flavors_test_win] [STDOUT] stdout:   package:flutter_test/src/binding.dart                                                                    new _TestWidgetsFlutterBinding&BindingBase&SchedulerBinding&ServicesBinding&GestureBinding&SemanticsBinding&RendererBinding&PaintingBinding&WidgetsBinding
[flavors_test_win] [STDOUT] stdout:   package:flutter_test/src/binding.dart                                                                    new _TestWidgetsFlutterBinding&BindingBase&SchedulerBinding&ServicesBinding&GestureBinding&SemanticsBinding&RendererBinding&PaintingBinding&WidgetsBinding&TestDefaultBinaryMessengerBinding
[flavors_test_win] [STDOUT] stdout:   package:flutter_test/src/binding.dart                                                                    new TestWidgetsFlutterBinding
[flavors_test_win] [STDOUT] stdout:   package:flutter_test/src/binding.dart                                                                    new LiveTestWidgetsFlutterBinding
[flavors_test_win] [STDOUT] stdout:   package:integration_test/integration_test.dart                                                           new IntegrationTestWidgetsFlutterBinding
[flavors_test_win] [STDOUT] stdout:   c:/Users/DKWings/src/flutter/dev/integration_tests/flavors/integration_test/integration_test.dart        new FlavorTestWidgetsFlutterBinding
[flavors_test_win] [STDOUT] stdout:   c:/Users/DKWings/src/flutter/dev/integration_tests/flavors/integration_test/integration_test.dart 22:7   FlavorTestWidgetsFlutterBinding.ensureInitialized
[flavors_test_win] [STDOUT] stdout:   c:/Users/DKWings/src/flutter/dev/integration_tests/flavors/integration_test/integration_test.dart 41:35  main

    This is really weird because I can't think of any place that would have assigned to _debugInitializedType. I even tried to acquire the stacktrace at

    flutter/packages/flutter/lib/src/foundation/binding.dart

    Line 264 in a494776

    _debugInitializedType = runtimeType;
    the only place that assigns to _debugInitializedType (this change not included in the PR), but the stacktrace turns out to be null. Is it caused by some different setup by integration tests?

  11. goderbauer commented on Aug 8, 2022

    @goderbauer
    goderbauer
    on Aug 8, 2022
    Member

    Can you print out what _debugInitializedType is initialized to? That may give us a clue.

  12. dkwingsmt commented on Aug 8, 2022

    @dkwingsmt
    dkwingsmt
    on Aug 8, 2022
    Contributor

    Yes, it is initialized as IntegrationTestWidgetsFlutterBinding.

  13. goderbauer commented on Aug 8, 2022

    @goderbauer
    goderbauer
    on Aug 8, 2022
    Member

    Actually, this test looks overwhelmingly green: https://flutter-dashboard.appspot.com/#/build?taskFilter=flavors_test_win&authorFilter&messageFilter&hashFilter&showMac=true&showWindows=true&showiOS=true&showLinux=true&showAndroid=true&showStaging=false&repo=flutter&branch=master

    There don't appear to be any flakes in the last few weeks. I think we should just remove the flakiness label from it and close this bug.

  14. goderbauer
    added
    P1High-priority issues at the top of the work list
    and removed
    P1
    on Aug 8, 2022

  15. jmagman commented on Dec 13, 2022

    @jmagman
    jmagman
    on Dec 13, 2022
    Member

    It seems to be failing 100% now on cert installation, filed #117001

  16. godofredoc
    closed this as completedon Apr 12, 2023

  17. github-actions commented on Apr 26, 2023

    @github-actions
    github-actionsbot
    on Apr 26, 2023

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  18. github-actions
    locked as resolved and limited conversation to collaborators on Apr 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P1High-priority issues at the top of the work listc: flakeTests that sometimes, but not always, incorrectly passframeworkflutter/packages/flutter repository. See also f: labels.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions