Restore DDC Expression Evaluation tests (#168017): Causing suite timeouts

#168017 significantly added time to {PLATFORM} web_tool_tests, causing timeouts:

https://ci.chromium.org/ui/p/flutter/builders/luci.flutter.prod/Mac%20web_tool_tests
https://ci.chromium.org/ui/p/flutter/builders/luci.flutter.prod/Linux%20web_tool_tests
Also #169277

Looking at a single execution:

24:45 +36 ~10: /b/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/expression_evaluation_web_amd_test.dart: Flutter test for web can evaluate complex expressions in a test
25:25 +37 ~10: /b/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/expression_evaluation_web_amd_test.dart: Flutter test for web evaluated expression includes correctly mapped stack trace
26:07 +38 ~10: /b/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/expression_evaluation_web_amd_test.dart: Flutter test for web evaluated expression includes web library environment defines
26:45 +39 ~10: /b/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/expression_evaluation_web_amd_test.dart: Flutter test for web can evaluate trivial expressions in a test
27:23 +40 ~10: /b/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/expression_evaluation_web_amd_test.dart: Flutter test for web cannot evaluate expressions if feature is disabled
28:01 +41 ~10: /b/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/expression_evaluation_web_amd_test.dart: Flutter test for web can evaluate complex expressions in library without pause
28:38 +42 ~10: /b/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/expression_evaluation_web_amd_test.dart: Flutter test for web can evaluate trivial expressions in library without pause
29:17 +43 ~10: /b/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/expression_evaluation_web_amd_test.dart: Flutter run for web can evaluate complex expressions in build method
29:52 +44 ~10: /b/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/expression_evaluation_web_amd_test.dart: Flutter run for web cannot evaluate expression if feature is disabled
30:26 +45 ~10: /b/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/expression_evaluation_web_amd_test.dart: Flutter run for web can evaluate trivial expressions in library without pause
31:01 +46 ~10: /b/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/expression_evaluation_web_amd_test.dart: Flutter run for web can evaluate complex expressions in top level function
31:35 +47 ~10: /b/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/expression_evaluation_web_amd_test.dart: Flutter run for web evaluated expression includes web library environment defines
32:10 +48 ~10: /b/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/expression_evaluation_web_amd_test.dart: Flutter run for web can evaluate trivial expressions in top level function
32:45 +49 ~10: /b/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/expression_evaluation_web_amd_test.dart: Flutter run for web can handle compilation errors
33:20 +50 ~10: /b/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/expression_evaluation_web_amd_test.dart: Flutter run for web evaluated expression includes correctly mapped stack trace
33:58 +51 ~10: /b/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/expression_evaluation_web_amd_test.dart: Flutter run for web can evaluate trivial expressions in build method
34:32 +52 ~10: /b/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/expression_evaluation_web_amd_test.dart: Flutter run for web can evaluate complex expressions in library without pause
35:06 +53 ~10: /b/s/w/ir/x/w/flutter/packages/flutter_tools/test/web.shard/expression_evaluation_web_amd_test.dart: Flutter run for web shows no native javascript objects in static scope

One test suite (there are two of the above PR) takes over 10 minutes.

Skip tests from #169304.

Skip `{PLAT}_web_tools_test#expression_evaluation_*_test.dart`, turni…

@srujzs

We hit similar limits before and increased the timeouts from 30m to 60m. But that was only for Linux.

As of earlier today, the timeout for Windows has been increased. Mac timeout increase is also coming.

With the timeout increases and skipping some recent slow tests, the tree should be good. But we need to work on a more scalable solution for adding (and re-enabling) these tests.

cc @srujzs

@nshahan

:( I was worried about the timeouts when adding these tests.

The long-term solution for many of these tests is to only run the DDC library bundle version. @nshahan is working on enabling the DDC library bundle version by default, but that's still a flag that users can opt out of.

We're in a weird state where we want to test both formats until we're ready to treat the new format as the only one supported on Flutter and drop that flag entirely. That will take a while (probably at least one stable release after this next one). If we do think we're resource-constrained and don't want to run web tests for both (which is entirely fair considering how long these tests are), I propose then that we just run it with the new format as we enable it by default.

@nshahan thoughts?

At this time we temporarily have two code paths in package:flutter_tools to support --web-experimental-hot-reload and --no-web-experimental-hot-reload. We have forked the tests to cover both code paths. I'm currently working through a change to enable hot reload by default for the next release. If that goes well we will remove the ability to opt out in the release after and we can delete that half of these tests.

Would it be reasonable to shard the two sets of tests so their runs get their own timeouts, especially knowing that it is not intended to be a permanent increase of testing infra?

@matanlurey

Thanks for the clarification!

I'll let @matanlurey and/or @jtmcdole comment on the temporary timeout increase. I'm personally okay with increasing the timeout temporarily while we phase out the old code path, but I don't know enough about our resource limitations to make an educated decision here.

The only thing I would caution is that we have a tendency to forget to reduce timeouts later (example: #120901, that builder is consistently under 25m yet the timeout is still set at 90m).

A few thoughts from discussing this with Matan and John McD yesterday:

Is there an overhead caused by running these tests from different files vs something like:

void main() async {
  await testAll(useDDCLibraryBundleFormat: false);
  await testAll(useDDCLibraryBundleFormat: true);
}

Looking at the code that invokes these tests, it's calling dart run test once and passing all the files at once. Not sure what happens on the Dart side, is there a significant overhead for launching each test file?

We could limit the number of platforms that run the legacy version. Say we run the legacy version only on Linux to maintain some coverage while we phase that version out. Is this a reasonable possibility? Is there enough platform-specific functionality that requires running on multiple platforms?

I'm not a fan of extending the timeouts because a) they often times never get reduced and b) increases the amount of time for the tree to go green (e.g. if we had one test that took 120 minutes to run, the tree would be red for 2 hours).

I am putting up the following guide to help inform this (and other) questions in terms of CI usage:

https://github.com/flutter/flutter/blob/ed25bc8cca445f852896d6e1f08c22e64d09416d/docs/infra/Ci-Best-Practices.md

For these expression evaluation tests in particular, I'm comfortable with them only running on one platform. Since there seems to be the most linux bots, that seems fine to me.

I agree there is a high likely hood of forgetting to reduce the timeout again when we delete half of the tests (especially for me since I rarely work in this repo and infrastructure). I do have an extremely high incentive to delete them though because they are directly tied to forked code in the Dart SDK that I am responsible for maintaining. If I create an issue to delete half of the tests and and TODO to reduce it the timeout does that give enough accountability that we think it will eventually get reduced?

If I create an issue to delete half of the tests and and TODO to reduce it the timeout does that give enough accountability that we think it will eventually get reduced?

(if I may add:)

Clean up an existing timeout that we had forgotten.
Schedule a recurring calendar entry as a reminder.

3. Clean up an existing timeout that we had forgotten.

Here: #169512

@mdebbar

IIUC the current situation the tests that were timing out are now skipped so the urgency there is only in the current lack of coverage.

I'm currently working on a change that will enable --web-experimental-hot-reload by default for the next stable release and allow for manually disabling disabling it if needed.

To support this and make sure we have continued test coverage I'm proposing:

Create a copy of the packages/flutter_tools/test/web.shard directory.
- Configure one to run with --web-experimental-hot-reload and the other with --no-web-experimental-hot-reload
Refactor the tests that were timing out so that they only run with a single reload mode in each shard.
- This should allow us to remove the skips without timing out.
Short term (through the next stable flutter release):
- The --no-web-experimental-hot-reload shard runs only on linux.
- The --web-experimental-hot-reload shard runs on all platforms.
Long term (ideally for the next-next stable release):
- Delete the --no-web-experimental-hot-reload shard.

If that sounds good I'll get started but I'm not familiar with how to setup a new testbot configuration. @mdebbar could you assist in getting a configuration that would only run a new web.shard directory in Linux?

[tests] Increase linux timeout and skip on others

[tests] Increase linux timeout and skip on others (#169627)

[tests] Increase linux timeout and skip on others (flutter#169627)

Combine expression evaluation tests to reduce testing time (#169860)

Squashed commit of the following:

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

[tests] Increase linux timeout and skip on others (#169627)

Navigation Menu

Repository navigation

Restore DDC Expression Evaluation tests (#168017): Causing suite timeouts #169304

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Restore DDC Expression Evaluation tests (#168017): Causing suite timeouts #169304

Description

Activity

mdebbar commented on May 22, 2025

srujzs commented on May 22, 2025

nshahan commented on May 22, 2025

mdebbar commented on May 23, 2025

jtmcdole commented on May 23, 2025

matanlurey commented on May 23, 2025

nshahan commented on May 23, 2025

mdebbar commented on May 27, 2025

mdebbar commented on May 27, 2025

nshahan commented on May 27, 2025

github-actions commented on Jun 12, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions