Repository navigation
Speed up "pythoneval" tests #1668
Description
The one slowest test task in our test suite is testpythoneval.py aka run eval-test -- in fact, on my desktop with 8-way parallelism, the whole suite takes 1m28s, and if that one task is omitted it takes 51s. This is because for much of that time it's the single straggler task running, even though since commit 56bb4ba we start it at the very beginning. (There are also some quick wins we can get to make the suite much faster than 51s, once we make this task much faster and/or break it into smaller pieces.)
Looking inside the test code, it looks like it does two things for each of a number of test cases:
- Run the type-checker on the test program
- Run the test program
and then compares the output to an expected output.
Is step 2 still useful? It made a lot of sense when mypy was its own implementation of a Python-like language, but now it seems like it just serves as a regression check on CPython. In principle it could be useful to check that our tests run the way we think they do, but that's basically a test of those test cases themselves. I'm not sure how often that's useful; I think most of the test cases themselves are very straightforward. So I suggest we cut it out.
Activity
gvanrossum commentedon Jun 7, 2016
I don't think step 2 is useful. We should however keep step 1, since these are the only tests that don't use the fake builtins from mypy/test/data/fixtures/. Unfortunately this means somewhat painful surgery on the test case data files, since the checker output and the CPython output are just combined in the [out] sections. Probably you can run myunit with the -u or -i flag to make it generate new versions of the test case data files.
JukkaL commentedon Jun 7, 2016
Step 2 isn't essential any more. (I'd still consider it at least slightly useful as long as PEP 484 is provisional and the
typingmodule is still evolving. However, it might be better to just move anything that seems valuable to the thetypingmodule test suite.)The biggest bottleneck is processing the stubs for builtins and typing for every test case. If we'd use incremental type checking to cache these stubs the test cases would be much faster. We could have a setup method that populates the cache. (This has been discussed elsewhere.)
Beyond that and removing step 2, we have at least these options for speeding up things:
buildfunction directly.ddfisher commentedon Jun 7, 2016
If we're going to be investing significantly in our test suite, I think we should consider moving to a standard test runner like py.test. This would make it easier for new contributors to start on the project, and should open the door to things like getting test coverage, which I think would be pretty useful (because I suspect we have some important uncovered sections).
gnprice commentedon Jun 7, 2016
Yeah, I agree that something like py.test would be a good choice and I wouldn't want to spend a lot of effort reinventing the kind of thing it would do for us.
Clearing out unnecessary test cases and steps is effort that I think will commute pretty well across such a change -- whatever tests we have will get translated across systems, and clearing them out shouldn't be much harder in the existing setup than it will be with py.test. It might even make a migration somewhat easier, if there's less to translate.
Apart from that, the other kind of effort that I think can make sense to invest in our test system now is where there's a big win for a small amount of work, like the reordering and other fixes I made in March. I don't think there's an opportunity to do that right now, though there might be after cutting out step 2 of these tests.
I'll cut more specific tasks for removing step 2 and so on.
ddfisher commentedon Jun 7, 2016
Agreed.
ilevkivskyi commentedon May 18, 2018
We first try #5083 then if it will not give enough speed-up we can re-consider other options discussed here.