Skip to content

Linux web_long_running_tests_5_5 is 2.02% flaky #146189

Closed
#147736
Closed
Linux web_long_running_tests_5_5 is 2.02% flaky#146189
#147736
Assignees
yjbanov
Labels
P0Critical issues such as a build break or regressionc: flakeTests that sometimes, but not always, incorrectly passplatform-webWeb applications specificallyteam-webOwned by Web platform teamtriaged-webTriaged by Web platform team

Description

@fluttergithubbot
fluttergithubbot
opened on Apr 3, 2024

The post-submit test builder Linux web_long_running_tests_5_5 had a flaky ratio 2.02% for the past (up to) 100 commits, which is above our 2.00% threshold.

One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/15903,%2015902
Commit: 0ee2592

Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/15903
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/15902
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/15852
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/15851
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/15850

Recent test runs:
https://flutter-dashboard.appspot.com/#/build?taskFilter=Linux%20web_long_running_tests_5_5

Please follow https://github.com/flutter/flutter/wiki/Reducing-Test-Flakiness#fixing-flaky-tests to fix the flakiness and enable the test back after validating the fix (internal dashboard to validate: go/flutter_test_flakiness).

Activity

  1. fluttergithubbot
    added
    c: flakeTests that sometimes, but not always, incorrectly pass
    P0Critical issues such as a build break or regression
    team-webOwned by Web platform team
    on Apr 3, 2024
  2. fluttergithubbot
    assigned
    yjbanov
    on Apr 3, 2024
  3. yjbanov
    added
    platform-webWeb applications specifically
    triaged-webTriaged by Web platform team
    on Apr 3, 2024

  4. yjbanov commented on Apr 3, 2024

    @yjbanov
    yjbanov
    on Apr 3, 2024
    Contributor

    Not sure what's going on yet, and whether this is related to #143834.

  5. yjbanov commented on Apr 3, 2024

    @yjbanov
    yjbanov
    on Apr 3, 2024
    Contributor

    Actually, most of the failures are related to #128382 (see this log file). The tests complete OK, but they complain about "Command did not write the response file but expected response file written". Reassigning to @linchen2chris for initial investigation.

  6. yjbanov
    removed their assignment
    on Apr 3, 2024

  7. yjbanov commented on Apr 3, 2024

    @yjbanov
    yjbanov
    on Apr 3, 2024
    Contributor

    Looks like I cannot assign to @linchen2chris. So assigning to the first reviewer listed in the PR. @cbracken you drew the short straw :)

  8. yjbanov
    assigned
    cbracken
    on Apr 3, 2024
  9. cbracken
    removed their assignment
    on Apr 9, 2024

  10. cbracken commented on Apr 9, 2024

    @cbracken
    cbracken
    on Apr 9, 2024
    Member

    Unassigning since this is one of the web-tests and the issue seems to be related to flutter driver, which which I'm not familiar. Looking at the PR I suspect I was the second required approver. Do we just need someone to revert that patch?

  11. cbracken commented on Apr 9, 2024

    @cbracken
    cbracken
    on Apr 9, 2024
    Member

    Looking quickly at the output and code, it looks like:

    1. We synchronously delete build/integration_response_data.json
    2. We run whatever web test this is
    3. We synchronously check for the presence of build/integration_response_data.json and fail to find it causing a failure.

    That logic seems correct to me. @yjbanov what makes you think the file is definitely being written (and written on time)?

  12. fluttergithubbot commented on Apr 10, 2024

    @fluttergithubbot
    fluttergithubbot
    on Apr 10, 2024
    ContributorAuthor

    [prod pool] flaky ratio for the past (up to) 100 commits between 2024-04-03 and 2024-04-09 is 3.19%. Flaky number: 3; total number: 94.
    One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16026
    Commit: 7971b35
    Flaky builds:
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16026
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/15991
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/15987

    Recent test runs:
    https://flutter-dashboard.appspot.com/#/build?taskFilter=Linux%20web_long_running_tests_5_5

  13. fluttergithubbot
    assigned
    yjbanov
    on Apr 10, 2024

  14. yjbanov commented on Apr 17, 2024

    @yjbanov
    yjbanov
    on Apr 17, 2024
    Contributor

    I actually don't know if it's being written. I also don't know what this file is and why it needs to be written. For now, I'm inclined to revert the change that expects it.

  15. fluttergithubbot commented on Apr 17, 2024

    @fluttergithubbot
    fluttergithubbot
    on Apr 17, 2024
    ContributorAuthor

    [prod pool] flaky ratio for the past (up to) 100 commits between 2024-04-08 and 2024-04-16 is 2.04%. Flaky number: 2; total number: 98.
    One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16066
    Commit: 74b65d4
    Flaky builds:
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16066
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16058

    Recent test runs:
    https://flutter-dashboard.appspot.com/#/build?taskFilter=Linux%20web_long_running_tests_5_5

  16. keyonghan commented on Apr 22, 2024

    @keyonghan
    keyonghan
    on Apr 22, 2024
    Contributor

    Just came across the dashboard and noticed the flaky percentage (11 out recent 50 runs are flaky) continues high: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16274/infra

  17. yjbanov commented on Apr 24, 2024

    @yjbanov
    yjbanov
    on Apr 24, 2024
    Contributor

    @keyonghan Yes, all the "web_long_running_tests" failures seem to have the same cause.

  18. fluttergithubbot commented on Apr 24, 2024

    @fluttergithubbot
    fluttergithubbot
    on Apr 24, 2024
    ContributorAuthor

    [prod pool] flaky ratio for the past (up to) 100 commits between 2024-04-18 and 2024-04-23 is 8.25%. Flaky number: 8; total number: 97.
    One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16281,%2016282
    Commit: 89a4ffa
    Flaky builds:
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16282
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16281
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16274
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16273
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16272
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16263
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16262
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16257
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16237
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16223
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16206
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16204

    Recent test runs:
    https://flutter-dashboard.appspot.com/#/build?taskFilter=Linux%20web_long_running_tests_5_5

  19. fluttergithubbot commented on May 1, 2024

    @fluttergithubbot
    fluttergithubbot
    on May 1, 2024
    ContributorAuthor

    [prod pool] flaky ratio for the past (up to) 100 commits between 2024-04-23 and 2024-04-30 is 3.03%. Flaky number: 3; total number: 99.
    One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16336
    Commit: 9751d4d
    Flaky builds:
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16336
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16330
    https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_long_running_tests_5_5/16298

    Recent test runs:
    https://flutter-dashboard.appspot.com/#/build?taskFilter=Linux%20web_long_running_tests_5_5

  20. auto-submit
    added a commit that references this issue on May 1, 2024

    [web] remove platform_messages_integration test (#147654)

    f256f68

  21. yjbanov commented on May 2, 2024

    @yjbanov
    yjbanov
    on May 2, 2024
    Contributor

    This may be the culprit, which I think is what @eyebrowsoffire already suspected:

    [0501/183306.597435:INFO:CONSOLE(5)] "Injecting <script> tag. Using callback.", source: http://localhost:35733/flutter_bootstrap.js (5)
[0501/183306.726152:INFO:CONSOLE(168)] "Uncaught Error: Mismatched anonymous define() module: () => CanvasKitInit
http://requirejs.org/docs/errors.html#mismatch", source: http://localhost:35733/require.js (168)
  22. yjbanov
    added a commit that references this issue on May 2, 2024

    [web] increase chromedriver logging level (#147687)

    1d0e798
  23. auto-submit
    added a commit that references this issue on May 2, 2024

    [web] skip debug mode CanvasKit e2e tests due to flakiness; unskip al…

    d5836ee
  24. auto-submit
    closed this as completedin #147736on May 2, 2024

  25. github-actions commented on May 16, 2024

    @github-actions
    github-actionsbot
    on May 16, 2024 – with GitHub Actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  26. github-actions
    locked as resolved and limited conversation to collaborators on May 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P0Critical issues such as a build break or regressionc: flakeTests that sometimes, but not always, incorrectly passplatform-webWeb applications specificallyteam-webOwned by Web platform teamtriaged-webTriaged by Web platform team

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions