Skip to content

web_long_running_tests is flaky #86375

Closed
Closed
web_long_running_tests is flaky#86375
Assignees
yjbanov
Labels
P0Critical issues such as a build break or regressiona: tests"flutter test", flutter_test, or one of our testsc: flakeTests that sometimes, but not always, incorrectly passplatform-webWeb applications specifically

Description

@yjbanov
yjbanov
opened on Jul 13, 2021

According to infra metrics it's flaky 2.08% of the time (eyeballing the LUCI dashboard though, the rate is 3/200 or 1.5%; not sure how the dashboard computes it). There's no error messages or stack traces in the logs, unfortunately. We should probably fix that first.

Sample
Sample
Sample

Activity

  1. yjbanov
    added
    a: tests"flutter test", flutter_test, or one of our tests
    platform-webWeb applications specifically
    P1High-priority issues at the top of the work list
    c: flakeTests that sometimes, but not always, incorrectly pass
    on Jul 13, 2021
  2. chunhtai
    added
    team: flakes
    on Jul 14, 2021

  3. fluttergithubbot commented on Jul 26, 2021

    @fluttergithubbot
    fluttergithubbot
    on Jul 26, 2021
    Contributor

    Current flaky ratio for the past 15 days is 1.86%.
    One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/934
    Commit: b0e51db
    Flaky builds:
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/934
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/908
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/835

  4. darshankawar
    added
    passed first triage
    on Jul 28, 2021

  5. fluttergithubbot commented on Aug 2, 2021

    @fluttergithubbot
    fluttergithubbot
    on Aug 2, 2021
    Contributor

    Current flaky ratio for the past 15 days is 1.12%.
    One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/934
    Commit: b0e51db
    Flaky builds:
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/934
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/908

  6. fluttergithubbot commented on Aug 9, 2021

    @fluttergithubbot
    fluttergithubbot
    on Aug 9, 2021
    Contributor

    Current flaky ratio for the past 15 days is 0.00%.

  7. fluttergithubbot commented on Aug 16, 2021

    @fluttergithubbot
    fluttergithubbot
    on Aug 16, 2021
    Contributor

    Current flaky ratio for the past 15 days is 0.77%.
    One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1208
    Commit: 91e0759
    Flaky builds:
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1208

  8. fluttergithubbot commented on Aug 23, 2021

    @fluttergithubbot
    fluttergithubbot
    on Aug 23, 2021
    Contributor

    Current flaky ratio for the past 15 days is 2.99%.
    One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1254
    Commit: d83fba1
    Flaky builds:
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1266
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1254
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1233
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1208

  9. fluttergithubbot
    added
    P1
    and removed
    P1High-priority issues at the top of the work list
    on Aug 23, 2021
  10. fluttergithubbot
    assigned
    yjbanov
    on Aug 23, 2021

  11. fluttergithubbot commented on Aug 30, 2021

    @fluttergithubbot
    fluttergithubbot
    on Aug 30, 2021
    Contributor

    Current flaky ratio for the past 15 days is 3.37%.
    One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1397
    Commit: c5c4dcb
    Flaky builds:
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1397
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1377
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1328
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1266
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1254
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1233

  12. 5 remaining items

  13. fluttergithubbot commented on Sep 20, 2021

    @fluttergithubbot
    fluttergithubbot
    on Sep 20, 2021
    Contributor

    Current flaky ratio for the past 15 days is 1.71%.
    One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1624
    Commit: 83dfb22
    Flaky builds:
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1624
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1558

  14. annagrin commented on Sep 20, 2021

    @annagrin
    annagrin
    on Sep 20, 2021
    Contributor

    @yjbanov I've been removing some flakes in web_tool_tests. They look stable except for canvaskit tests (skipped or replaced by html renderer use for now). Might be related to this issue, or the hot restart problem. See:

    #85043

    PR that disabled the tests:

    #86885

    PR that switched to html:

    #85325

  15. yjbanov commented on Sep 21, 2021

    @yjbanov
    yjbanov
    on Sep 21, 2021
    ContributorAuthor

    Update: we now have a version lock for CanvasKit (https://github.com/flutter/flutter/blob/master/bin/internal/canvaskit.version) and a roll job (https://skia-review.googlesource.com/c/buildbot/+/450796). Once the roll job takes effect we can update the tool to fetch CanvasKit to local disk and use it when running tests instead of unpkg.com.

  16. yjbanov commented on Oct 5, 2021

    @yjbanov
    yjbanov
    on Oct 5, 2021
    ContributorAuthor

    Update: #91328 is the first roll that rolls CanvasKit to the framework. Once that lands (not anticipating any issues) we can start using local CanvasKit instead of fetching it from the network. I'm working on PRs that will do that as we speak. There will be one engine PR and one framework PR.

  17. fluttergithubbot commented on Oct 18, 2021

    @fluttergithubbot
    fluttergithubbot
    on Oct 18, 2021
    Contributor

    Current flaky ratio for the past 15 days is 2.03%.
    One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1874
    Commit: 158939a
    Flaky builds:
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1874
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1836
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1763
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1761
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1760
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1759

  18. yjbanov commented on Oct 19, 2021

    @yjbanov
    yjbanov
    on Oct 19, 2021
    ContributorAuthor

    Update: here's a draft for the PR that's pulling CanvasKit from CIPD. Once that lands I can move the integration tests over to the local build.

    Having said that, most of the breakages listed in #86375 (comment) do not look related to CDN flakiness. A few are about the missing WebDriver server (1, 2, 3, 4), and one is a Flutter Tool crash. These won't be solved by bundling CanvasKit.

  19. yjbanov commented on Oct 22, 2021

    @yjbanov
    yjbanov
    on Oct 22, 2021
    ContributorAuthor

    The flakiness reported as https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1558/overview seems to have been a race in the test. Fix: #92303

  20. fluttergithubbot commented on Oct 25, 2021

    @fluttergithubbot
    fluttergithubbot
    on Oct 25, 2021
    Contributor

    Current flaky ratio for the past 15 days is 1.46%.
    One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1874
    Commit: 158939a
    Flaky builds:
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1874
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/1836

  21. fluttergithubbot commented on Nov 1, 2021

    @fluttergithubbot
    fluttergithubbot
    on Nov 1, 2021
    Contributor

    Current flaky ratio for the past 15 days is 0.83%.
    One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/2018
    Commit: dcb3037
    Flaky builds:
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_1_5/2018

  22. yjbanov commented on Nov 3, 2021

    @yjbanov
    yjbanov
    on Nov 3, 2021
    ContributorAuthor

    Closing this issue. Everything related to the web functionality should be fixed. I don't see the old flakiness happening any more. The one remaining tool issue is already filed as #92042.

  23. yjbanov
    closed this as completedon Nov 3, 2021

  24. github-actions commented on Nov 17, 2021

    @github-actions
    github-actionsbot
    on Nov 17, 2021

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  25. github-actions
    locked as resolved and limited conversation to collaborators on Nov 17, 2021
  26. flutter-triage-bot
    added
    P0Critical issues such as a build break or regression
    and removed
    P1
    on Jun 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P0Critical issues such as a build break or regressiona: tests"flutter test", flutter_test, or one of our testsc: flakeTests that sometimes, but not always, incorrectly passplatform-webWeb applications specifically

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions