Split pytest plugins out into individual modules - #5770

@pllim I have just read your GSOC idea on splitting astropy test helper out into a seperate package. This is the first step in that I think.

I like the concept of making things more modular. I think there was a similar request from @josePhoenix a while back. However, CIs are failing.

yeah this needs some more work (it's tied to sunpy/sunpy#1983) but it's not very high on my todo list unfortunately.

Okay, I'll make a note of this PR number in the GSoC idea, just in case. Thanks!

@pllim We moved the ideas to the open astronomy website, I added the PR and put myself up as a mentor while I was moving it ;)

@Cadair @pllim This is specifically the idea I was interested in. Can you please elaborate on how I can get involved? (As of now I'm working on [ #5799 #5858 ] ). Thanks.

Well this currently breaks all the CI builds as the split clearly hasn't been done cleanly enough.

The basic idea here is to move the "plugin" code out of the conftest.py file and into files that can be imported separately by packages that depend on astropy (or copied out of astropy altogether) and then configure astropy to use those plugins by specifying them in the conftest.py file.

Also @pllim can you please tell me any small to-do's that I can begin with to start sorting this one out?

@Cadair I was a little confused as to why this project was given Intermediate/Advanced tag on OpenAstro project list tag if plugin's code was already implemented? (I do understand that some of astropy.utils and astropy.test.helpers needs to be re-implemented for separating out the modules.)

@mohanagr , the hard part is disentangling dependency and making it a viable independent package. If you find it easy, then great. Or if you find it too boring, then you are free to pick a different GSoC project to apply for.

@pllim As a matter of fact I'm planning to submit for the splitting project! I have been looking into how astropy is structured and the test suites. I think there will be a lot to learn about packaging projects and making them work across different versions.

@Cadair In the new standalone, are we to provide tests.helper functionality like custom warnings, handling deprecations (as exceptions), handling unicode guidelines etc. too? I mean features which are Astropy specific?
I mean do we single out functions/classes from astropy.tests.helpers and astropy.utils that the plugin code explicitly depends on or implement the whole thing?

@mohanagr , that is exactly the problem that I hope the GSoC project can solve. 😅

@pllim @Cadair Currently there is also this problem of running all plugin code from conftest.py itself. This implies I can't use --doctest-rst option if I run from let's say astropy but only from astropy/astropy and child directories. Separating out things will resolve this too.

This behavior was an issue with pytest.
Reference :
pytest-dev/pytest#906
In response a warning was added regarding the use of pytest.addoption -
pytest-dev/pytest@ccf7584

@mohanagr you mean registering the plugins in conftest rather than installing them and having them register as hooks?

@Cadair I meant that hook calls i.e. call to pytest_addoption from a plugin via pytest_plugins = ['plugin1'] might resolve the issue instead of all hook calls being in conftest.py itself.

Or perhaps declaring them in setup.py via entry_points.

This is what the comment by the maintainer reads:

If you change your conftest file into a plugin, it should work consistently no matter where you call py.test from, or what you pass on the command-line or perhaps via entry_points in setup.py.

It is unfortunate (but understandable) that there's this gotcha when working with pytest_addoption and conftest files, because conftest initialization being lazy as it is now can lead to surprising errors. Perhaps a note to pytest_addoption discouraging its use from conftest files would at least warn users of these pitfall?

Also, can you tell me what did you mean by installing plugins? I mean if they are there in the code itself.

Edit : Haven't tried it for this P.R. I was talking about the original code. You have already registered the plugins in pytest_plugins. I don't understand his statement about changing conftest file to a plugin.

I was wrong above. The current way of specifying plugins in a file via pytest_plugins= produces the same thing (Tried changes in this PR). Declaring entry_point for plugins might be an option. @Cadair If you can tell me what the maintainer meant by changing conftest file to a plugin I might make some headway.

Also, can you tell me what did you mean by installing plugins? I mean if they are there in the code itself.

@mohanagr - you may want to check out other pytest plugins for the infrastructure and how-to (e.g. pytest-mpl, pytest-cov, etc). As it was said above one of the idea for the gsoc project was to factor out all the additions we have (compared to pure pytest) into an independent plugin. That means that it needs to be installed, and we don't have it any more in astropy core (pending a deprecation period of course).

@bsipocz Oh all right. Until now I was assuming there'd be separate multiple plugins. Hence didn't understand why would one do pip install pytest-plugin1,2,3 multiple times instead of just cloning and using them.

@mohanagr - Users don't clone anything, but install released versions.

What if users are developers? 😃

You don't need to worry about that subset. Anyway cloning and using is not the preferred way for developers either when the aim is to build a stable package rather than hacking.

I think we are getting a bit too far off topic here. Questions related to the GSoC project should be discussed outside of this PR thread. As far as this PR is concerned, if you would like to take over, it is just simply splitting the current stuff into different modules, not a separate installable package.

@pllim I understand. I couldn't find a way to reach you apart from GH comments!

There is a developer mailing list https://groups.google.com/forum/#!forum/astropy-dev

@Cadair - Any chance you have time to make this work for v2.0 (ff in 4 weeks time). I only ask as it would be a good one for deprecating, and 3.0 is to remove things.

@Cadair - pinging you again, whether we need to remilestone this or you can make some progress with it to be merged this week?

I will not be able to get to this before the freeze :(

Superseded by #6384.