Skip to content

Benchmark tests are failing #9674

Description

@elliette

Our dart2wasm benchmark tests are failing with the following error (example run):

Can run web benchmarks with WASM (failed)
  Building Flutter web app (compiler: dart2wasm)...
  Build took 146s to complete.
  Launching Chrome.
  Launching Google Chrome 145.0.7632.76 
  
  Waiting for the benchmark to report benchmark profile.
  [CHROME]: 
  [CHROME]: DevTools listening on ws://127.0.0.1:10000/devtools/browser/3409c080-2f4f-4ae9-947d-b3de97f85223
  Connecting to DevTools: ws://localhost:10000/devtools/page/6A2B907AC5919DAE0A4AC907955D80F8
  Connected to Chrome tab:  (http://localhost:9999/?compiler=wasm)
  Launching benchmark "devtools_navigateThroughOfflineScreens"
  [APP] 2026-02-21 00:37:04.316: TEST STATUS: Warming up.
  [APP] 2026-02-21 00:37:04.428: TEST STATUS: Warm-up finished.
  [APP] 2026-02-21 00:37:04.429: TEST STATUS: Navigate through offline DevTools tabs
  [APP] 2026-02-21 00:37:04.431: TEST STATUS: switching to home screen (icon null, iconAsset: icons/app_bar/devtools.png)
  [APP] 2026-02-21 00:37:07.450: TEST STATUS: switching to performance screen (icon null, iconAsset: icons/app_bar/performance.png)
  [APP] 2026-02-21 00:37:10.485: TEST STATUS: switching to cpu-profiler screen (icon null, iconAsset: icons/app_bar/cpu_profiler.png)
  [APP] 2026-02-21 00:37:13.518: TEST STATUS: switching to memory screen (icon null, iconAsset: icons/app_bar/memory.png)
  [APP] 2026-02-21 00:37:16.537: TEST STATUS: switching to network screen (icon null, iconAsset: icons/app_bar/network.png)
  [APP] 2026-02-21 00:37:19.556: TEST STATUS: switching to app-size screen (icon null, iconAsset: icons/app_bar/app_size.png)
  [APP] 2026-02-21 00:37:22.589: TEST STATUS: switching to deep-links screen (icon null, iconAsset: icons/app_bar/deep_links.png)
  [APP] 2026-02-21 00:37:25.625: TEST STATUS: End navigate through offline DevTools tabs
  Extracted 1255 measured frames.
  Skipped 123 non-measured frames.
  [APP] Client preparing to reload the window to: "http://localhost:9999?compiler=wasm"
  Launching benchmark "devtools_offlineCpuProfilerScreen"
  [APP] 2026-02-21 00:37:31.227: TEST STATUS: Warming up.
  [APP] 2026-02-21 00:37:31.344: TEST STATUS: Warm-up finished.
  [APP] 2026-02-21 00:37:31.344: TEST STATUS: Loading offline CPU profiler data and interacting
  [APP] 2026-02-21 00:38:04.528: TEST STATUS: On Bottom Up tab by default. Scrolling through table.
  [APP] 2026-02-21 00:38:06.061: TEST STATUS: Switching to Call Tree tab.
  [APP] 2026-02-21 00:38:12.098: TEST STATUS: Scrolling through Call Tree table.
  [APP] 2026-02-21 00:38:13.648: TEST STATUS: Switching to Method Table tab.
  [APP] 2026-02-21 00:38:19.664: TEST STATUS: Scrolling through Method Table.
  [APP] 2026-02-21 00:38:21.198: TEST STATUS: Switching to CPU Flame Chart tab.
  Caught browser-side error: Exception: `startMeasureFrame` called twice in a row.
      at $.DomWindow|requestAnimationFrame (http://localhost:9999/main.dart.wasm:wasm-function[5969]:0x21c941)
      at $.EnginePlatformDispatcher.scheduleFrame (http://localhost:9999/main.dart.wasm:wasm-function[5967]:0x21c6a1)
      at $._RecordingWidgetsBinding.handleDrawFrame (http://localhost:9999/main.dart.wasm:wasm-function[29143]:0x50cdcc)
      at $._MixinApplication2&BindingBase&GestureBinding&SchedulerBinding.handleBeginFrame (http://localhost:9999/main.dart.wasm:wasm-function[11963]:0x2d431f)
      at $._RecordingWidgetsBinding.handleBeginFrame (http://localhost:9999/main.dart.wasm:wasm-function[29144]:0x50ceea)
      at $._MixinApplication2&BindingBase&GestureBinding&SchedulerBinding._handleBeginFrame tear-off trampoline (http://localhost:9999/main.dart.wasm:wasm-function[5974]:0x21ca75)
      at $.invoke1 (http://localhost:9999/main.dart.wasm:wasm-function[2066]:0x1aee6c)
      at $.closure wrapper at org-dartlang-sdk:///lib/_engine/engine/frame_service.dart:91:37 trampoline (http://localhost:9999/main.dart.wasm:wasm-function[5968]:0x21c7db)
  dart:async                                     _Completer.completeError
  package:web_benchmarks/src/runner.dart 254:25  BenchmarkServer.run.<fn>
  ===== asynchronous gap ===========================
  dart:async                                     _CustomZone.registerUnaryCallback
  package:web_benchmarks/src/runner.dart 248:27  BenchmarkServer.run.<fn>
  package:shelf/src/cascade.dart 77:53           Cascade.handler.<fn>.<fn>
  ===== asynchronous gap ===========================
  dart:async                                     Future.then
  package:shelf/src/cascade.dart 76:58           Cascade.handler.<fn>
  package:shelf/src/cascade.dart 76:47           Cascade.handler.<fn>.<fn>
  dart:async                                     new Future.sync
  package:shelf/src/cascade.dart 76:21           Cascade.handler.<fn>
  package:shelf/shelf_io.dart 140:29             handleRequest
  package:shelf/shelf_io.dart 93:9               serveRequests.<fn>.<fn>
  ===== asynchronous gap ===========================
  dart:_http                                     _HttpServer.listen
  package:shelf/shelf_io.dart 92:14              serveRequests.<fn>
  package:shelf/src/util.dart 21:20              catchTopLevelErrors
  package:shelf/shelf_io.dart 91:3               serveRequests
  package:web_benchmarks/src/runner.dart 316:16  BenchmarkServer.run
  ===== asynchronous gap ===========================
  dart:async                                     _CustomZone.registerUnaryCallback
  package:web_benchmarks/src/runner.dart 136:42  BenchmarkServer.run
  package:web_benchmarks/server.dart 75:5        serveWebBenchmark
  benchmark/devtools_benchmarks_test.dart 57:28  _runBenchmarks
  ===== asynchronous gap ===========================
  dart:async                                     _CustomZone.registerUnaryCallback
  benchmark/devtools_benchmarks_test.dart 54:25  _runBenchmarks
  benchmark/devtools_benchmarks_test.dart 43:15  main.<fn>
Shell: Starting web benchmark tests ...
Shell: Web benchmark tests finished.
Shell: Verifying devtools_navigateThroughOfflineScreens.js scores against expected thresholds.
Shell: Verifying devtools_offlineCpuProfilerScreen.js scores against expected thresholds.
Shell: Verifying devtools_offlinePerformanceScreen.js scores against expected thresholds.

Activity

  1. added
    P1high priority issues at the top of the work list, actively being worked on.
    on Feb 23, 2026
  2. elliette commented on Feb 26, 2026

    @elliette
    MemberAuthor
  3. elliette commented on Feb 26, 2026

    @elliette
    MemberAuthor

    Still seeing initial failure as well: https://github.com/flutter/devtools/actions/runs/22461618910/job/65057339304

    I'm going to disable the tests for now as I investigate these flakes.

  4. elliette commented on Mar 6, 2026

    @elliette
    MemberAuthor

    Investigation notes:

    failure for job/65057339304

    failure for job/65029549692

    The following error is due to this tabs.single getter in package:web_benchmarks - it expects there to be a single open http://localhost tab but for whatever reason there is none. Filed flutter/flutter#183335 on the web_benchmarks package.

      Connecting to DevTools: ws://localhost:10001/devtools/page/84039657253FC755ED7D526B2FC40AA8
      Bad state: No element
      dart:core                                       Iterable.single
      package:web_benchmarks/src/browser.dart 292:30  _connectToChromeDebugPort
      ===== asynchronous gap ===========================
      dart:async                                      _CustomZone.registerBinaryCallback
      package:web_benchmarks/src/browser.dart 113:38  Chrome.launch
      package:web_benchmarks/src/runner.dart 332:34   BenchmarkServer.run
      ===== asynchronous gap ===========================
      dart:async                                      _CustomZone.registerUnaryCallback
      package:web_benchmarks/src/runner.dart 136:42   BenchmarkServer.run
      package:web_benchmarks/server.dart 75:5         serveWebBenchmark
      benchmark/devtools_benchmarks_test.dart 57:28   _runBenchmarks
    

    failure for job/64720745754

    Legitimate failure of CPU profiler not passing benchmarks, this might require adding some retry logic but due to #9397 we will get false passes if we add back in retries on the test case itself.

        Actual: '[devtools_offlineCpuProfilerScreen.wasm] flutter_frame.total_time.p90 was 32860.0 μs, which exceeded the expected threshold, 16666.6 μs.\n'
                  ''
      [WASM Benchmarks] The following benchmark scores exceeded their expected thresholds:
      
      [devtools_offlineCpuProfilerScreen.wasm] flutter_frame.total_time.p90 was 32860.0 μs, which exceeded the expected threshold, 16666.6 μs.
      
      
      package:matcher                                expect
      benchmark/devtools_benchmarks_test.dart 120:3  _runBenchmarks
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1high priority issues at the top of the work list, actively being worked on.type-test

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions