Repository navigation
Remove no-shuffle tags from all framework tests #85160
Description
Activity
In #86793, I marked a lot of tests with
no-shuffle. Here are the tests from that PR that need fixing, with a random seed that caused them to fail at the time:Fixed File Seed ✅ packages/flutter/test/animation/animation_controller_test.dart 123 ✅ packages/flutter/test/cupertino/action_sheet_test.dart 123 ✅ packages/flutter/test/cupertino/date_picker_test.dart 123 ✅ packages/flutter/test/cupertino/nav_bar_transition_test.dart 456 ✅ packages/flutter/test/cupertino/text_field_test.dart 123 ✅ packages/flutter/test/foundation/service_extensions_test.dart 123 ✅ packages/flutter/test/gestures/debug_test.dart 123 ✅ packages/flutter/test/gestures/long_press_test.dart 123 ✅ packages/flutter/test/gestures/tap_test.dart 1000 ✅ packages/flutter/test/material/app_test.dart 123 ✅ packages/flutter/test/material/dropdown_test.dart 1408669812 ✅ packages/flutter/test/material/paginated_data_table_test.dart 1000 ✅ packages/flutter/test/material/progress_indicator_test.dart 456 ✅ packages/flutter/test/material/scaffold_test.dart 123 ✅ packages/flutter/test/material/scrollbar_test.dart 382757700 no packages/flutter/test/material/text_field_test.dart 20230318 ✅ packages/flutter/test/painting/image_stream_test.dart 456 ✅ packages/flutter/test/physics/newton_test.dart 123 ✅ packages/flutter/test/rendering/editable_test.dart 20210704 ✅ packages/flutter/test/rendering/viewport_test.dart 123 ✅ packages/flutter/test/scheduler/scheduler_test.dart 123 ✅ packages/flutter/test/scheduler/ticker_test.dart 4281596210 ✅ packages/flutter/test/services/platform_channel_test.dart 20210826 ✅ packages/flutter/test/widgets/app_overrides_test.dart 123 ✅ packages/flutter/test/widgets/binding_test.dart 382757700 ✅ packages/flutter/test/widgets/clip_test.dart 456 ✅ packages/flutter/test/widgets/debug_test.dart 123 ✅ packages/flutter/test/widgets/dismissible_test.dart 456 ✅ packages/flutter/test/widgets/draggable_test.dart 123 ✅ packages/flutter/test/widgets/platform_view_test.dart 4281596210 ✅ packages/flutter/test/widgets/routes_test.dart 123 ✅ packages/flutter/test/widgets/scroll_aware_image_provider_test.dart 123 ✅ packages/flutter/test/widgets/widget_inspector_test.dart 456 ✅ packages/flutter_driver/test/src/real_tests/extension_test.dart 20210721 ✅ packages/flutter_driver/test/src/real_tests/flutter_driver_test.dart 20210721 ✅ packages/flutter_goldens/test/flutter_goldens_test.dart 123 ✅ packages/flutter_test/test/bindings_test.dart 20210721 ✅ packages/flutter_test/test/test_text_input_test.dart 20210721 ✅ packages/flutter_test/test/window_test.dart 123 ✅ packages/flutter_tools/test/commands.shard/hermetic/build_ios_framework_test.dart 1000 ✅ packages/flutter_tools/test/commands.shard/hermetic/daemon_test.dart 1000 ✅ packages/flutter_tools/test/commands.shard/hermetic/doctor_test.dart 20210723 ✅ packages/flutter_tools/test/commands.shard/hermetic/run_test.dart 1000 ✅ packages/flutter_tools/test/commands.shard/permeable/build_bundle_test.dart 20210722 ✅ packages/flutter_tools/test/commands.shard/permeable/packages_test.dart 1000 ✅ packages/flutter_tools/test/general.shard/build_system/targets/dart_plugin_registrant_test.dart 20210722 ✅ packages/flutter_tools/test/general.shard/channel_test.dart 20210723 ✅ packages/flutter_tools/test/general.shard/dart/language_version_test.dart 456 ✅ packages/flutter_tools/test/integration.shard/break_on_framework_exceptions_test.dart 1000 ✅ packages/flutter_tools/test/integration.shard/overall_experience_test.dart 1000 ✅ packages/flutter_tools/test/integration.shard/test_test.dart 1000 @gspencergoog I have started working on this, and fixed a few failing tests. I cannot reproduce some of the failing tests though. For example, all the the tests in the cupertino folder passes successfully for me even when shuffling is turned on.
I remove the 'no-shuffle' tag at the top of the test file and run the chosen test file with --test-randomize-ordering-seed=123 flag. Are you sure the seed you supplied is correct, or am I doing something wrong? 🙂
Thanks for looking at these @Swiftaxe! All I know is that some cupertino tests fail for me at that seed (I just tried it again). I'd try running it 100 times with the seed set to "random" and see if it ever fails. It can also be that things fail because of other reasons (flaky tests, timing issues), so perhaps the failure I saw wasn't because of the seed.
My method for for running the tests is:
- comment out the
allow_test_randomization: falseline in thedart_test.yamlinpackages/flutter. - run
flutter test --test-randomize-ordering-seed=123in thepackages/flutterdirectory. I run all the tests, since that's how they'll be run on the server (and we're trying to fix interdependencies here).
The failures I currently get in cupertino are:
00:13 +971 ~1 -2: /home/gspencer/code/flutter/packages/flutter/test/cupertino/action_sheet_test.dart: CupertinoActionSheet scrollbars controllers should be different 00:13 +995 ~1 -5: /home/gspencer/code/flutter/packages/flutter/test/cupertino/dialog_test.dart: CupertinoDialogRoute is state restorable- comment out the
Thanks for replying @gspencergoog.
I am an experienced developer but new to flutter and open source in general. Is it possible for me to create pull requests for some of the tests at a time?
Dividing the whole lot into several pull requests will avoid one large pull request, and will get things moving faster.I didn't find any information on this so I don't know how the procedure goes. I'm guessing I need to create new issues related to this one for each pull request, is that right?
Reacted by Tong MuSure, in fact that is preferred. That's why I set it up with the tags, so that we could tackle them one at a time instead of all at once.
You don't need to create a new issue for each one, you can just reference this issue in each PR.
You might also read our Contributor's Guide if you want to learn more about how to contribute. Specifically, the Tree Hygiene page.
Thanks! I have finished a bunch and will create a PR.
- added a commit that references this issue
on Aug 16, 2021 - added a commit that references this issue
on Aug 19, 2021 - added 4 commits that reference this issue
on Aug 23, 2021 32 remaining items
- added a commit that references this issue
on Aug 28, 2026 - added a commit that references this issue
on Sep 2, 2026 - added a commit that references this issue
on Sep 3, 2026 - added a commit that references this issue
on Sep 8, 2026 - added a commit that references this issue
on Sep 8, 2026 - added 4 commits that reference this issue
on Sep 9, 2026 - added a commit that references this issue
on Sep 30, 2026 - added a commit that references this issue
on Oct 6, 2026
Description
In #86793 we turned on shuffling of test order based on a seed in order to root out test state leakage. In the process, we added a testing tag called 'no-shuffle' which was applied to the majority of tests which currently fail with the shuffling turned on.
This issue is to clean up all of those instances of broken tests, and finally to remove the "no-shuffle" tag entirely to turn on shuffling for all tests.
Related issues