Skip to content

flutter_test_performance is newly flaky #34448

Closed
#34457
Closed
flutter_test_performance is newly flaky#34448
#34457
Labels
a: tests"flutter test", flutter_test, or one of our testsc: flakeTests that sometimes, but not always, incorrectly pass

Activity

  1. Hixie
    added
    a: tests"flutter test", flutter_test, or one of our tests
    ⚠ TODAY
    team: flakes
    on Jun 13, 2019
  2. Hixie
    added this to the milestone on Jun 13, 2019

  3. tvolkert commented on Jun 13, 2019

    @tvolkert
    tvolkert
    on Jun 13, 2019
    Contributor

    When the test fails, it's because the line "Downloading Web SDK..." is appearing in stdout, causing the following failure:

    Task failed: Exception: flutter test rendered unexpected output (1 bad lines)

    @jonahwilliams

  4. jonahwilliams commented on Jun 13, 2019

    @jonahwilliams
    jonahwilliams
    on Jun 13, 2019
    Contributor

    The first time this test is run, it dumps newly updated dependencies from pub get:

    [flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn): Building flutter tool...
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn): Resolving dependencies...
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   analyzer 0.36.3
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   archive 2.0.9 (2.0.10 available)
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   args 1.5.2
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   async 2.2.0
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   bazel_worker 0.1.21
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   boolean_selector 1.0.4
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   bsdiff 0.1.0
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   build 1.1.4
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   build_config 0.4.0
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   build_daemon 0.6.1 (1.0.0 available)
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   build_modules 2.2.0 (2.3.0 available)
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   build_resolvers 1.0.5
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   build_runner 1.4.0 (1.5.2 available)
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   build_runner_core 3.0.5 (3.0.6 available)
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   build_test 0.10.8
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   build_vm_compilers 1.0.0
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   build_web_compilers 2.1.0 (2.1.1 available)
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   built_collection 4.2.2
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   built_value 6.6.0
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   charcode 1.1.2
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   code_builder 3.2.0
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   collection 1.14.11
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   completion 0.2.1+1
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   convert 2.1.1
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   coverage 0.12.4
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   crypto 2.0.6
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   csslib 0.16.0
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   dart_style 1.2.7 (1.2.8 available)
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   file 5.0.8
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   file_testing 2.1.0
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   fixnum 0.10.9
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   flutter_goldens_client 0.0.0 from path ../flutter_goldens_client
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   front_end 0.1.18
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   glob 1.1.7
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   graphs 0.2.0
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   html 0.14.0+2
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   http 0.12.0+2
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   http_multi_server 2.0.6 (2.1.0 available)
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   http_parser 3.1.3
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   intl 0.15.8
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   io 0.3.3
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   js 0.6.1+1
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   json_annotation 2.4.0
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   json_rpc_2 2.1.0
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   kernel 0.3.18
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   linter 0.1.91
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   logging 0.11.3+2
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   matcher 0.12.5
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   meta 1.1.6 (1.1.7 available)
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   mime 0.9.6+3
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   mockito 4.1.0
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   multi_server_socket 1.0.2
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   multicast_dns 0.1.1
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   mustache 1.1.1
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   node_preamble 1.4.4
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   package_config 1.0.5
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   package_resolver 1.0.10
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   path 1.6.2
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   pedantic 1.7.0
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   petitparser 2.3.0
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   platform 2.2.0
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   plugin 0.2.0+3
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   pool 1.4.0
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   process 3.0.9
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   protobuf 0.13.11 (0.13.12 available)
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   pub_semver 1.4.2
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   pubspec_parse 0.1.4
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   quiver 2.0.3
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   scratch_space 0.0.3+2
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   shelf 0.7.5
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   shelf_packages_handler 1.0.4
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   shelf_static 0.2.8
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   shelf_web_socket 0.2.3
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   source_map_stack_trace 1.1.5
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   source_maps 0.10.8
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   source_span 1.5.5
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   stack_trace 1.9.3
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   stream_channel 2.0.0
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   stream_transform 0.0.19
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   string_scanner 1.0.4
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   term_glyph 1.1.0
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   test 1.6.3 (1.6.4 available)
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   test_api 0.2.5 (0.2.6 available)
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   test_core 0.2.5 (0.2.6 available)
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   timing 0.1.1+1
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   typed_data 1.1.6
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   usage 3.4.1
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   vm_service_client 0.2.6+2
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   vm_service_lib 3.17.0+1
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   watcher 0.9.7+10
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   web_socket_channel 1.0.13
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   webkit_inspection_protocol 0.4.0
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   xml 3.5.0
[flutter_test_performance] [STDOUT] test stdout (TestStep.testWritesFirstCarriageReturn):   yaml 2.1.15

    The second time those are gone so it seems like it might pass

  5. jonahwilliams commented on Jun 13, 2019

    @jonahwilliams
    jonahwilliams
    on Jun 13, 2019
    Contributor

    Also potentially package getting can throw it off:

    [flutter_test_performance] [STDOUT] test stdout (TestStep.starting): Running "flutter pub get" in automated_tests...                     0.4s
[flutter_test_performance] [STDOUT] test stdout (TestStep.starting):
...
[flutter_test_performance] [STDOUT] "/Users/jonahwilliams/Documents/flutter/dev/devicelab/../../bin/flutter" exit code: 0
[flutter_test_performance] [STDERR] Task failed: Exception: flutter test rendered unexpected output (1 bad lines)

    Seems like we really shouldn't be using regular expressions here

  6. jonahwilliams
    closed this as completedin #34457on Jun 14, 2019

  7. github-actions commented on Aug 13, 2021

    @github-actions
    github-actionsbot
    on Aug 13, 2021

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  8. github-actions
    locked as resolved and limited conversation to collaborators on Aug 13, 2021
  9. flutter-triage-bot
    added
    c: flakeTests that sometimes, but not always, incorrectly pass
    on Jul 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    a: tests"flutter test", flutter_test, or one of our testsc: flakeTests that sometimes, but not always, incorrectly pass

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions