Repository navigation
Test suite needs to be updated for py.test 3.x #5277
Description
At the moment, our test suite fails if using pytest 3.x. Of course, we're not seeing that in the CI because we use the (old) bundled version of pytest 3.x, so this isn't too urgent. But to see the failures, you can do:
python setup.py build_ext --inplace
py.test astropy
which gives:
========================================================== ERRORS ===========================================================
_____________________________________________________ ERROR collecting _____________________________________________________
/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/_pytest/main.py:570: in gethookproxy
return self._fs2hookproxy[fspath]
E KeyError: local('/Users/tom/Dropbox/Code/Astropy/astropy/astropy/table/tests')
During handling of the above exception, another exception occurred:
/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/_pytest/config.py:301: in _getconftestmodules
return self._path2confmods[path]
E KeyError: local('/Users/tom/Dropbox/Code/Astropy/astropy/astropy/table/tests')
During handling of the above exception, another exception occurred:
/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/_pytest/config.py:332: in _importconftest
return self._conftestpath2mod[conftestpath]
E KeyError: local('/Users/tom/Dropbox/Code/Astropy/astropy/astropy/table/tests/conftest.py')
During handling of the above exception, another exception occurred:
/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/_pytest/config.py:338: in _importconftest
mod = conftestpath.pyimport()
/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/py/_path/local.py:650: in pyimport
__import__(modname)
astropy/table/__init__.py:28: in <module>
from .column import Column, MaskedColumn
astropy/table/column.py:15: in <module>
from ..utils.console import color_print
astropy/utils/console.py:29: in <module>
from IPython import get_ipython
/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/IPython/__init__.py:48: in <module>
from .core.application import Application
/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/IPython/core/application.py:25: in <module>
from IPython.core.profiledir import ProfileDir, ProfileDirError
/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/IPython/core/profiledir.py:29: in <module>
class ProfileDir(LoggingConfigurable):
/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/IPython/core/profiledir.py:52: in ProfileDir
`profile` option.""",
/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/traitlets/traitlets.py:421: in __init__
DeprecationWarning, stacklevel=stacklevel)
E DeprecationWarning: metadata {'config': True} was set from the constructor. Metadata should be set using the .tag() method, e.g., Int().tag(key1='value1', key2='value2')
During handling of the above exception, another exception occurred:
/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/py/_path/common.py:332: in visit
for x in Visitor(fil, rec, ignore, bf, sort).gen(self):
/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/py/_path/common.py:378: in gen
for p in self.gen(subdir):
/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/py/_path/common.py:367: in gen
dirs = self.optsort([p for p in entries
/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/py/_path/common.py:368: in <listcomp>
if p.check(dir=1) and (rec is None or rec(p))])
/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/_pytest/main.py:672: in _recurse
ihook = self.gethookproxy(path)
/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/_pytest/main.py:575: in gethookproxy
my_conftestmodules = pm._getconftestmodules(fspath)
/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/_pytest/config.py:315: in _getconftestmodules
mod = self._importconftest(conftestpath)
/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/_pytest/config.py:340: in _importconftest
raise ConftestImportFailure(conftestpath, sys.exc_info())
E _pytest.config.ConftestImportFailure: DeprecationWarning("metadata {'config': True} was set from the constructor. Metadata should be set using the .tag() method, e.g., Int().tag(key1='value1', key2='value2')",)
E File "/Users/tom/Dropbox/Code/Astropy/astropy/astropy/table/__init__.py", line 28, in <module>
E from .column import Column, MaskedColumn
E File "/Users/tom/Dropbox/Code/Astropy/astropy/astropy/table/column.py", line 15, in <module>
E from ..utils.console import color_print
E File "/Users/tom/Dropbox/Code/Astropy/astropy/astropy/utils/console.py", line 29, in <module>
E from IPython import get_ipython
E File "/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/IPython/__init__.py", line 48, in <module>
E from .core.application import Application
E File "/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/IPython/core/application.py", line 25, in <module>
E from IPython.core.profiledir import ProfileDir, ProfileDirError
E File "/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/IPython/core/profiledir.py", line 29, in <module>
E class ProfileDir(LoggingConfigurable):
E File "/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/IPython/core/profiledir.py", line 52, in ProfileDir
E `profile` option.""",
E File "/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/traitlets/traitlets.py", line 421, in __init__
E DeprecationWarning, stacklevel=stacklevel)
================================================== pytest-warning summary ===================================================
WC1 None [pytest] section in setup.cfg files is deprecated, use [tool:pytest] instead.
WC1 None pytest_funcarg__cov: declaring fixtures using "pytest_funcarg__" prefix is deprecated and scheduled to be removed in pytest 4.0. Please remove the prefix and use the @pytest.fixture decorator instead.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================== 2 pytest-warnings, 1 error in 3.93 seconds =========================================
Activity
[-]Test suite needs to be updated for pytest 3.x[/-][+]Test suite needs to be updated for py.test 3.x[/+]pllim commentedon Aug 30, 2016
EDIT: The results below are generated after I changed
[pytest]to[tool:pytest]insetup.cfg, as suggested above.Wow...
Log attached (no
--remote-data):ztestlog.txt
pllim commentedon Nov 2, 2016
Note: For another project using Astropy package template,
.. doctest-skip-alldirective no longer works under pytest 3.x as well.eteq commentedon Nov 2, 2016
Note that there's a suspicion from #5435 that a change in pytest 3.x altered how defaults are interpreted by
getoptionand similar pytest functions. This does not appear to be documented in the pytest 3.x docs (😡 ), but it may address some of this...bsipocz commentedon Nov 23, 2016
Appveyor breakage of astropy-helpers is also due to pytest 3.x (astropy/astropy-helpers#265 ). While the workaround there is super simple, I'm thinking about to pin the version number of pytest to the one that gets bundled into astropy to be a bit more self consistent.
mhvk commentedon Nov 23, 2016
Once we fix things, we should perhaps use latest pytest for the allowed-to-fail run (with numpy-dev and remote-data).
bsipocz commentedon Nov 23, 2016
OK, the helpers failure is a red herring, but the pytest behaviour is indeed changed. So the failure was within ci-helpers and that got fixed today.
The new puzzling issue is that astropy-helpers should have never picked up the ci-helpers test file and indeed it doesn't do it in travis only on appveyor.
mhvk commentedon Jan 8, 2017
While trying to investigate, I wanted to just use regular commands for testing so I could focus on specific packages. This, it turns out, was foreseen: one can just do
mhvk commentedon Jan 8, 2017
There seem to be some different failures:
yieldstructure are deprecated; these are easy to fix, and @kelle started some fixes forwcs-- I have an additional one forunits... doctest-skipseems to get ignored (both individual and whole-file).EDIT: Fixed in PRs mentioned here.
mhvk commentedon Jan 9, 2017
OK, so with #5678 and #5682 merged, the first item on the above little list is checked off. Now need to understand why the output changes (different line length assumed?) and why
doctest-skipno longer works...pllim commentedon Jan 10, 2017
@mhvk , I spent a good few hours trying to understand the doctest failure but got nowhere. I tried inserting print statements but they don't work with pytest, as it suppresses print outputs, I think. There is no traceback, just some obscure deprecation warnings. And no clear documented translation guide for those deprecations. 😰
astrofrog commentedon Jan 10, 2017
@pllim - you can run pytest with the
-soption to see the output of print statements.pllim commentedon Jan 11, 2017
Some warnings are due to a bug in pytest 3.0.5, see pytest-dev/pytest#2118, although that still does not explain why doctest is not skipping some files/snippets as it should. FYI.
Update: To avoid those warnings, pin pytest to 3.0.4 until the bug is fixed. Mind you, doctest still does not work properly but we will have less warnings to worry about.
mhvk commentedon Jan 12, 2017
I've been trying to narrow down a bit. A first conclusion is that the problem only occurs for text files, not for source files (modules; e.g.,
__doctest_skip__in *.py files does work). A second is that the options we pass on, in particularNORMALIZE_WHITESPACEinpytest_plugins.pytest_configuredoes not seem to get used for*.rstfiles....
...
...
Looking in more detail, the main problem seems to be that there is no
collectmethod defined for DocTestTextfilePlus. Indeed, by adding a collect method following the pytestDoctestTextfileclass, I can get the tests to pass! (well, minus some annoyingvotests, but what's new...) See #5688.mhvk commentedon Jan 19, 2017
Closing this, since with #5688 astropy now works well with pytest 3.x.
cdeil commentedon Aug 3, 2017
This just now appeared, I think because our CI switched to the recently released pytest 3.2:
https://travis-ci.org/hipspy/hips/jobs/260162492#L1211
Should I file a new issue about this? In which tracker?
bsipocz commentedon Aug 3, 2017
We're on it. Long story short it's not yet clear whether it's a bug in our plugin or in pytest so we need to limit the version number of pytest for now, I'm waiting for the builds to pass on ci-helpers (astropy/ci-helpers#222).
pytest-dev/pytest#2651