Mac_ios microbenchmarks_ios fails with random seed 225627159 #156506

Activity

1. christopherfujino

   added

   P0Critical issues such as a build break or regressionCritical issues such as a build break or regression

   on Oct 9, 2024
2. danagbemava-nc

   added

   in triagePresently being triaged by the triage teamPresently being triaged by the triage team

   team-infraOwned by Infrastructure teamOwned by Infrastructure team

   and removed

   in triagePresently being triaged by the triage teamPresently being triaged by the triage team

   on Oct 10, 2024
3. github-project-automation

   added this to [DEPRECATED] Infra Ticket Queueon Oct 10, 2024
4. github-project-automation

   moved this to New in [DEPRECATED] Infra Ticket Queueon Oct 10, 2024
5. christopherfujino

   changed the title [-]`Mac_ios microbenchmarks_ios` failing very frequently[/-] [+]`Mac_ios microbenchmarks_ios` fails with random seed 225627159[/+] on Oct 10, 2024
6. christopherfujino commented on Oct 10, 2024

   

   christopherfujino

   on Oct 10, 2024

   Last edited by christopherfujino

   ContributorAuthor

   More actions

   It appears this was failing due to the daily random seed used for ordering the tests: 225627159. I'm reducing the priority on this since today it is no longer failing, and assigning to @jtmcdole to debug.

   👍React with 👍1Reacted by John "codefu" McDole
7. christopherfujino

   added

   P2Important issues not at the top of the work listImportant issues not at the top of the work list

   triaged-infraTriaged by Infrastructure teamTriaged by Infrastructure team

   and removed

   P0Critical issues such as a build break or regressionCritical issues such as a build break or regression

   on Oct 10, 2024
8. christopherfujino

   assigned

   jtmcdole

   on Oct 10, 2024
9. jtmcdole commented on Oct 10, 2024

   

   jtmcdole

   on Oct 10, 2024

   Member

   More actions

   Yep - this failed locally for me on the first microbenchmark run in the seed (layout/text_intrinsic_bench.dart)
10. jason-simmons commented on Oct 10, 2024

    

    jason-simmons

    on Oct 10, 2024

    Member

    More actions

    I was able to reproduce this by:

    • patching https://github.com/flutter/flutter/blob/master/dev/benchmarks/microbenchmarks/lib/benchmark_collection.dart to only run the layout/text_intrinsic_bench.dart benchmark
    • running lib/benchmark_collection.dart in release mode on an iPhone 11

    The error occurs when two calls to LiveTestWidgetsFlutterBinding.handleBeginFrame happen with no handleDrawFrame call in between.

    Bad state: handleBeginFrame() called before previous handleDrawFrame()
     #0      LiveTestWidgetsFlutterBinding.handleBeginFrame (package:flutter_test/src/binding.dart:1788)
     #1      BenchmarkingBinding.handleBeginFrame (package:microbenchmarks/benchmark_binding.dart:15)
     #2      SchedulerBinding._handleBeginFrame (package:flutter/src/scheduler/binding.dart:1155)
     #6      _invoke1 (dart:ui/hooks.dart:330)
     #7      PlatformDispatcher._beginFrame (dart:ui/platform_dispatcher.dart:405)
     #8      _beginFrame (dart:ui/hooks.dart:272)
    
    

    On my setup that happened due to a race between the handleBeginFrame calls made by WidgetTester.pumpBenchmark and handleBeginFrame calls initiated by the engine in response to system events.

    Shortly after launching the benchmark the iOS engine received an event that triggered a call to updateViewportMetrics. This causes the engine to schedule a frame and call handleBeginFrame at the next vsync. If that handleBeginFrame happens during a frame interval started by pumpBenchmark, then you get the exception above.

    @goderbauer Would it make sense for LiveTestWidgetsFlutterBinding to be less strict about the interleaving of handleBeginFrame and handleDrawFrame calls? The test framework can not assume that it has complete control over frame scheduling.

    Instead of throwing,LiveTestWidgetsFlutterBinding.handleBeginFrame could warn about an unexpected handleBeginFrame and skip the call to the superclass.
11. jtmcdole commented on Oct 11, 2024

    

    jtmcdole

    on Oct 11, 2024

    Member

    More actions

    I'm currently moving to cycle the microbenchmarks from native, with new a FlutterEngine created for each to keep the test time low and try to stabilize the test numbers.
12. flutter-triage-bot

    added

    ⏳Bot is counting down the days until it unassigns the issueBot is counting down the days until it unassigns the issue

    on Feb 14, 2025
13. flutter-triage-bot commented on Feb 14, 2025

    

    flutter-triage-botbot

    on Feb 14, 2025

    More actions

    This issue is assigned to @jtmcdole but has had no recent status updates. Please consider unassigning this issue if it is not going to be addressed in the near future. This allows people to have a clearer picture of what work is actually planned. Thanks!
14. github-merge-queue

    added a commit that references this issue on Mar 12, 2025

    Update Mac_mokey microbenchmarks with issue to the bringup: true in c…

    ...

    eef31a1
15. flutter-triage-bot

    removed

    triaged-infraTriaged by Infrastructure teamTriaged by Infrastructure team

    on May 3, 2025
16. flutter-triage-bot

    unassigned

    jtmcdole

    on May 3, 2025
17. flutter-triage-bot commented on May 3, 2025

    

    flutter-triage-botbot

    on May 3, 2025

    More actions

    This issue was assigned to @jtmcdole but has had no status updates in a long time. To remove any ambiguity about whether the issue is being worked on, the assignee was removed.
18. flutter-triage-bot

    removed

    ⏳Bot is counting down the days until it unassigns the issueBot is counting down the days until it unassigns the issue

    on May 3, 2025
19. matanlurey

    closed this as not plannedon May 28, 2025
20. github-actions commented on Jun 11, 2025

    

    github-actionsbot

    on Jun 11, 2025 – with GitHub Actions

    More actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.
21. github-actions

    locked as resolved and limited conversation to collaborators on Jun 11, 2025