Repository navigation
Test the test runner options #5806
Description
Activity
I'm putting this into milestone 2.0. If no one else picks it up before the due date, ping me about it.
I want to go ahead with this. Although since it's tagged
Package-IntermediateI need some help from where to start.@mohanagr - I suggest to target the
package-noviceand not theEffort-lowissues first, this one for example needs knowledge about how our testing infrastructure works (e.g. see the related issues for which this is a follow-up issue).Reacted by Mohan AgrawalRemoving milestone, but this can go in until release as it only affects testing.
I'm thinking about picking this one up if no one else is working on it. Just to clarify, are you looking for just simple validation of arguments (e.g. making sure using the
-toption actually results in a valid command string), or do you actually want to execute the resulting tests? The latter seems a little more complicated since you now have a test that relies on the execution of other tests.@drdavella - Thanks for picking this up. I think we need a bit of the mixture of the two you mentioned, e.g. we don't need to execute the tests, but need to run them until the test collection finishes to make sure e.g. the
-toption only picks up the single file rather than all of them.I wonder if it's possible to make a dummy test for this. That is, the test only has "pass". It doesn't make sense to run a real test multiple times.
So I have been looking at this for mumble hours and I'm sorry to say that it's just not obvious to me how this should be done. In order to test options given to the top-level test runner (e.g.
-t), we need to be able to callsetup.py. However, this isn't available from within an individual pytest since the tests get installed to a separate temporary build tree.One approach I considered is to modify the
AstropyTestobject to provide an option to test itself. For example, I might be able to call./setup.py test --meta-test, which would run itself with a few different options that we care about. I made some progress towards this but am encountering some frustrating issues. And this approach has the problem of not actually being plugged into pytest at all, so it's not clear how we would interpret/report results.@pllim, you marked this as
Effort-low. Did you have any particular approach in mind when you opened this issue? Maybe I'm just missing something obvious.Okay, maybe I had underestimated the effort and skill level needed... 😱
Haha, well it's entirely possible that I've been down in the weeds for too long and I'm just missing a really obvious approach.
Just to clarify, "effort-low" means different things for different "package-xxx" skill levels. I thought naively it would be easy to add for someone who is somewhat familiar with pytest but that assumption is obviously wrong based on your report, so I updated it to "effort-high" for a "package-expert", just in case. Sorry for any confusion.
This issue was labeled as Close?. Remove Close? label or this will be closed after 7 days. (This is currently a dry-run.)
Astropy testing has changed a lot since this was opened, and we don't use
setup.py testanymore. So this can be closed as OBE.Reacted by Brigitta Sipőczp.s. Agree we can close this.
As mentioned in #5798 (comment) , we need to test that our test runner options (e.g.,
-t) do not break.