Skip to content

Astropy test customizations break pytest for affiliated packages #5402

Closed
Closed
Astropy test customizations break pytest for affiliated packages#5402
Labels
testing

Description

@josePhoenix
josePhoenix
opened on Oct 11, 2016

Moved from astropy/astropy-helpers#258

Every so often I make the mistake of running pytest (or py.test for old versions) instead of python setup.py test. It seems like this should work, but instead I get errors collecting tests that seem to be related to the astropy helpers. Even though the file doesn't appear in the traceback below, it seems to come from imports in the conftest.py from the package template.

This also appears to bite unwary users of packages that use the template: http://stackoverflow.com/questions/31410723/error-using-pytest-tutorial

$ pytest
=============================================== test session starts ================================================
platform darwin -- Python 2.7.12, pytest-3.0.3, py-1.4.31, pluggy-0.4.0
rootdir: /Users/jlong/software/webbpsf, inifile: setup.cfg
collected 0 items / 1 errors

====================================================== ERRORS ======================================================
________________________________________________ ERROR collecting  _________________________________________________
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/py-1.4.31-py2.7.egg/py/_path/common.py:332: in visit
    for x in Visitor(fil, rec, ignore, bf, sort).gen(self):
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/py-1.4.31-py2.7.egg/py/_path/common.py:368: in gen
    if p.check(dir=1) and (rec is None or rec(p))])
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/main.py:672: in _recurse
    ihook = self.gethookproxy(path)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/main.py:575: in gethookproxy
    my_conftestmodules = pm._getconftestmodules(fspath)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/config.py:314: in _getconftestmodules
    mod = self._importconftest(conftestpath)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/config.py:350: in _importconftest
    self.consider_conftest(mod)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/config.py:373: in consider_conftest
    if self.register(conftestmodule, name=conftestmodule.__file__):
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/config.py:225: in register
    ret = super(PytestPluginManager, self).register(plugin, name)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:371: in register
    hook._maybe_apply_history(hookimpl)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:768: in _maybe_apply_history
    res = self._hookexec(self, [method], kwargs)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:339: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:334: in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:614: in execute
    res = hook_impl.function(*args)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy/tests/pytest_plugins.py:137: in pytest_configure
    if not config.getoption('remote_data'):
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/config.py:1148: in getoption
    raise ValueError("no option named %r" % (name,))
E   ValueError: no option named u'remote_data'
============================================== pytest-warning summary ==============================================
WC1 None [pytest] section in setup.cfg files is deprecated, use [tool:pytest] instead.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
==================================== 1 pytest-warnings, 1 error in 1.62 seconds ====================================

At @pllim's request, I tried the last 2.x release of pytest as well:

$ py.test
=============================================== test session starts ================================================
platform darwin -- Python 2.7.12, pytest-2.9.2, py-1.4.31, pluggy-0.3.1
rootdir: /Users/jlong/software/webbpsf, inifile: setup.cfg
collected 0 items / 1 errors

====================================================== ERRORS ======================================================
________________________________________________ ERROR collecting  _________________________________________________
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/py-1.4.31-py2.7.egg/py/_path/common.py:332: in visit
    for x in Visitor(fil, rec, ignore, bf, sort).gen(self):
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/py-1.4.31-py2.7.egg/py/_path/common.py:368: in gen
    if p.check(dir=1) and (rec is None or rec(p))])
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/main.py:647: in _recurse
    ihook = self.gethookproxy(path)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/main.py:550: in gethookproxy
    my_conftestmodules = pm._getconftestmodules(fspath)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/config.py:297: in _getconftestmodules
    mod = self._importconftest(conftestpath)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/config.py:333: in _importconftest
    self.consider_conftest(mod)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/config.py:356: in consider_conftest
    if self.register(conftestmodule, name=conftestmodule.__file__):
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/config.py:208: in register
    ret = super(PytestPluginManager, self).register(plugin, name)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:370: in register
    hook._maybe_apply_history(hookimpl)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:747: in _maybe_apply_history
    res = self._hookexec(self, [method], kwargs)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:338: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:333: in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py:596: in execute
    res = hook_impl.function(*args)
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy/tests/pytest_plugins.py:137: in pytest_configure
    if not config.getoption('remote_data'):
../../homebrew/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_pytest/config.py:1061: in getoption
    raise ValueError("no option named %r" % (name,))
E   ValueError: no option named u'remote_data'
============================================= 1 error in 1.47 seconds ==============================================

I also confirmed that removing the conftest.py I mentioned above resolves the issue, which implicates the from astropy.tests.pytest_plugins import * line. Hence the moved issue.

Activity

  1. pllim
    added
    testing
    on Oct 11, 2016

  2. eteq commented on Oct 12, 2016

    @eteq
    eteq
    on Oct 12, 2016
    Member

    @josePhoenix - can you clarify why you need py.test to work? That is, is there some specific reason why python setup.py test is insufficient, or is it just that using py.test feel more natural?

  3. josePhoenix commented on Oct 12, 2016

    @josePhoenix
    josePhoenix
    on Oct 12, 2016
    ContributorAuthor

    For projects I'm part of that use more-or-less "vanilla" PyTest, the instructions are "cd projdir/; pytest". If questions come up, I can refer to http://docs.pytest.org/.

    For affiliated packages, the instructions are "python setup.py test". That much is equivalently simple. But as soon as you need to go beyond that, all bets are off. The command line arguments are not documented anywhere beyond the level in the python setup.py test --help output. I opened an issue with astropy-helpers because I'm using the package template and the setup.py customizations live in the helpers, and I was told it's actually in astropy. I guess there's documentation in the astropy docs (partially reproducing the pytest documentation), but I have yet to meet a casual contributor that reads all the documentation of the project they're contributing to and all the documentation of its dependencies.

    But really, at the end of the day, I'm quite lazy and would simply rather type

    pytest ./webbpsf/tests/test_errorhandling.py

    than

    ASTROPY_USE_SYSTEM_PYTEST=1 python setup.py test -t ./webbpsf/tests/test_errorhandling.py

    to accomplish the same thing.

  4. eteq commented on Oct 26, 2016

    @eteq
    eteq
    on Oct 26, 2016
    Member

    @josePhoenix - I'm 100% with you that it should work - I was asking to see whether there's a particular use case missing from the existing machinery that you need it for. So I'm not arguing that you shouldn't be able to use py.test here, but just letting you know that two of these are supported by the python setup.py test case to let you work around it for now:

    • I can control the version of the pytest executable

    You can use ASTROPY_USE_SYSTEM_PYTEST=1 for this if you want - then it will use whatever the system's default pytest is, which you can set to whichever one you want. Definitely not as easy as giving the executable directly, but it will work in a pinch.

    And note that if you want to force this for your package so that it never uses the bundled py.test, you can set that to be the default in the setup.cfg. That's the intended way to configure all of the test stuff (which pytest invocations should respect... but clearly the bug here is preventing that)

    • Not every pytest option is available through Astropy's setup.py test

    They should be, using the -a/--args python setup.py test options. Anything provided there gets passed directly to py.test.

    would simply rather type

    pytest ./webbpsf/tests/test_errorhandling.py

    Just to be clear, this is subtly different from the python setup.py test case. The former runs the test in the source directory and the latter in the build directory. These are subtly different, and usually the latter is slightly safer. So it's not quite identical...

    All of that said, there is clearly a bug here that should be fixed! I just want to make it clear that the python setup.py test machinery is there for a reason, not just to make for longer command-line typing.

  5. eteq commented on Oct 26, 2016

    @eteq
    eteq
    on Oct 26, 2016
    Member

    @pllim @astrofrog @embray - do any of you understand why this is happening? It seems like some of the plugin machinery is not getting recognized and its hence failing due to a lack of the remote_data markings.

    I'll try to dig into it when I get a chance if no one else has any ideas...

  6. pllim commented on Oct 26, 2016

    @pllim
    pllim
    on Oct 26, 2016
    Member

    I can't comment much... Sorry.

  7. josePhoenix commented on Oct 26, 2016

    @josePhoenix
    josePhoenix
    on Oct 26, 2016
    ContributorAuthor

    Well, it's from this commit: https://github.com/astropy/astropy/pull/2044/files#diff-e602ffb35fbd67477e5543c9cc3846d4R132

    And most likely because no default was specified (see http://docs.pytest.org/en/latest/writing_plugins.html?highlight=getoption#_pytest.config.Config.getoption ).

  8. pllim commented on Oct 26, 2016

    @pllim
    pllim
    on Oct 26, 2016
    Member

    👍 @josePhoenix , you are welcome to submit a fix to Astropy.

  9. josePhoenix
    added a commit that references this issue on Oct 26, 2016

    Fix missing default in during pytest_configure

    32e3d37

  10. pllim commented on Feb 3, 2017

    @pllim
    pllim
    on Feb 3, 2017
    Member

    Is this still an issue? There are PRs to fix compatibility with pytest 3 and also to update the doc for testing.

  11. pllim commented on Apr 18, 2017

    @pllim
    pllim
    on Apr 18, 2017
    Member

    Doc was updated and changes to deal with pytest 3.x were merged. Closing for now. Please re-open if you run into this again.

  12. pllim
    closed this as completedon Apr 18, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testing

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions