Repository navigation
Start running tests with --test-randomize-ordering-seed=random #47233
Description
We should shuffle our tests. I've encountered a number of test files that depend on state set by previous tests. This ends up hiding bugs and/or creating unnecessary breaking changes later on when we have to fix it.
Package test takes an argument --test-randomize-ordering-seed=random. For regular test runs this looks like pub run test --test-randomize-ordering-seed=random. For flutter test we'll have to create a way to pipe in the argument. In general, we'll want to update dev/bots/test.dart to use this argument by default when running tests for CI.
/cc @jonahwilliams @goderbauer @tvolkert @Hixie @yjbanov @zanderso - I would not be surprised if this surfaces various breaking changes in tool/flutter/etc.
Activity
goderbauer commentedon Dec 17, 2019
Have you tried running it with that a couple of times just for fun to see what would be breaking?
zanderso commentedon Dec 17, 2019
Could you clarify in the description what that flag gets passed to?
dnfield commentedon Dec 17, 2019
The flag is passed to
package:test. We'll need to pipe it in forflutter testas well, but we could probably do it pretty easily with the tool tests.I haven't tried running it yet. I know for sure it would mess some things up on for Flutter until #47177 is merged.
dnfield commentedon Mar 25, 2020
Bumping the milestone as this has turned into a large effort.
Hixie commentedon Mar 25, 2020
I recommend using the git hash as the seed, by the way, so that reproducing the failures is easy. Otherwise we're intentionally introducing randomness which will just manifest as flakiness.
dnfield commentedon Mar 25, 2020
The pr number might be safer since this has to be a 32 bit integer.
Hixie commentedon Mar 25, 2020
there might not be a PR (e.g. consider a local branch). Turning a git hash into a 32bit number is easy though, just chop off all but the first 32 bits. :-)
dnfield commentedon Dec 8, 2020
Removing assignment and milestone for this. I came pretty close to landing this at one point, but it's a pretty tough moving target to hit, since new tess kept getting added that would break randomizability constraints.
There are some ways that we should be able to do this - e.g. automatically detecting a flake related to randomization by rerunning without randomization if failure occurs, but I have some higher priority work at the moment and don't expect to be able to get to it soon.
github-actions commentedon Aug 10, 2021
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.