Repository navigation
Make tests compile and run faster #69429
Description
Use case
According to flutter.dev one of the things flutter aims at is fast development. I think being able to develop in a TDD workflow contributes to this goal.
In our case, we have suite about 500 tests, consisting of unit and widget tests. In order to facilitate a real TDD workflow, all the tests should run very fast. There are some sources that suggest a maximum executing time of less than 10s: https://blog.ploeh.dk/2012/05/24/TDDtestsuitesshouldrunin10secondsorless/#:~:text=The%20test%20suite%20you%20work,be%20faster%20than%2010%20milliseconds.
Currently, our test suite is executing in about 2min, which makes the TDD approach very hard, because it disrupts your workflow (better described in the article).
Proposal
The aim of this ticket is more to collect all the tickets, that would contribute to a better TDD experience. So please add additional things that prevent you from a good TDD experience.
Live Unit Testing
#4719
Dart-Code/Dart-Code#2878
Improve Performance of Tests
Hot Reload for Unit Tests
#25973
https://github.com/szotp/hottie/
Please consider 👍 this and linked tickets if you thing this is important.
Activity
pedromassangocode commentedon Nov 2, 2020
Hi @HerrNiklasRaab
I looked at the link you posted an I got this piece:
I'm wondering if you are running all the tests of the project or just a subset of tests related to the [new] test you are adding?
I personally think that the duration of tests will always depend on the numbers of tests you are running and we can't just make 10.000+ tests to run in 10 seconds. As your tests increase the time will increase as well.
HerrNiklasRaab commentedon Nov 2, 2020
Hey @pedromassangocode,
thanks for your response. Sure if the project grows very much, at some point you will need to split it into multiple test suits. But currently, we have 500 tests, who take about 2min. This I think is definitely too long.
Currently, we are running the tests of the whole project.
Maybe we can keep this ticket open to track the experience from the point of view of a TDD developer.
Best regards,
Niklas
pedromassangocode commentedon Nov 2, 2020
I'm still not sure what can be done here so will leave this opened for the right people to handle it.
Cc @Hixie @HansMuller
Hixie commentedon Nov 2, 2020
@HerrNiklasRaab We will need more information to help here, for example, some profiles and traces of running the tests in question to see where the time is going.
It would also be good to have an estimate of how long each test should take. Is it reasonable for a test to take 250ms on average? Or should it be faster? Also, it would be useful to know whether you get different performance by changing the
-joption to change how many cores are being used.HerrNiklasRaab commentedon Nov 12, 2020
Hi @Hixie,
Thanks for your answer.
I think the most important part of unit tests is that they run fast, with fast I mean very fast. As soon as they are so slow that you don't run them every few minutes they turn into a burden. With fast tests your workflow looks like:
With fast tests:
With slow tests
My argument is, that the faster the tests run, the shorter this loop is, the more efficient you are developing, because your mind only needs to concentrate on one thing. You never need to debug.
Having very fast tests is crucial. That's why I think 250ms is way too long (no offense). I worked on projects with 5ms per test. This means you can have 2000 tests in your suite and they are still running in 10s.
With 250ms you can only execute 40 tests in 10s. This means you either run them not as often, (consequences described above) or you need to spend a lot of time thinking which test to run. This is very difficult (because you don't know which tests are affected by your change), time-consuming and there is no support in the IDE for running only a subset of tests.
I tried it with "-j", and it yielded a slight improvement. I have also the logs when running with "-v" but I am not sure if this is what you meant with "some profiles and traces of running the tests".
Verbose logs: text-verbose.txt.zip
Thanks very much and very interested in your response,
Niklas
PS: If improving the testspeed by this ammount is not possible due to technical reasons. Maybe it might be worth having a look at this project, which yields the hot reload: https://github.com/szotp/hottie/
16 remaining items
svarlet commentedon Aug 31, 2021
Agreed, until a real change is visible in the stable channel, It feels odd to "close" the issue.
jakemac53 commentedon Aug 31, 2021
Ya sorry I hadn't read the full context of the issue :). The change in the test package is very specifically geared towards regular Dart tests also, and has no effect on flutter tests. It should provide a dramatic improvement for regular Dart tests though 👍 .
I believe that flutter tests are already optimized using similar means (it is what inspired the VM test optimization). So if you are experiencing slow flutter tests I would look at the flutter test platform code and/or flutter test utilities.
Hixie commentedon Sep 1, 2021
We're nowhere near tests running fast even on master (on my 20 core machine, using all 20 cores, it takes SEVEN SECONDS for
flutter testto even begin to run the first test in theflutterpackage, for example, and the whole test suite takes 3m20s, which is about 0.55ms per test line, which seems really high!), so I agree that this bug shouldn't be closed.That said, it is our policy to close issues once they are fixed on master, we don't leave bugs open until they are fixed on stable. This is because otherwise we would be flooded with literally thousands of open bugs that are not actionable, whose close date would be months later. The GitHub issues are our development bug database and they therefore track master, not stable.
orestesgaolin commentedon Jan 17, 2022
Just wanted to add that the most problematic for us is the performance of coverage reporting tests on CI. For example a suite of 1400 tests takes over 20 minutes to finish. In other words it's about 1 second per test. The same process but without
--coveragetakes around 6 minutes. It doesn't help much to cache build artifacts or dependencies. This really slows down the loop when doing the code reviews. We're looking into sharding the tests between multiple parallel runs, but in practice we wish it was just faster to run tests with coverage.Hixie commentedon Jan 19, 2022
Coverage being slow is a separate issue (it requires changes to different parts of the codebase). We've made significant improvements over the past year to the coverage (mostly around accuracy, but also performance), but if you are still running into performance issues please file a bug specifically on that. If you have specific tests we can run that demonstrate the problem that is ideal (the Flutter framework tests are probably a good place to start if your project isn't open source itself).
cyberalt1 commentedon Mar 3, 2025
szotp commentedon Nov 27, 2025
Coming back to this topic as I have found something pretty cool. You can run any test file via regular flutter command using
The best part is: it's normal flutter execution with hot restart & roload, but test execute as usual.
Making tests run on every reload was a bit tricky but I got a demo working: https://github.com/szotp/hottie_next.
Cache plugin registrant in test compiler (flutter#185598)
Skip platform-specific plugin registration if no platforms enabled
Skip platform-specific plugin registration if no platforms enabled (f…
Skip platform-specific plugin registration if no platforms enabled (f…
Skip platform-specific plugin registration if no platforms enabled (f…