Repository navigation
Test suite needs to be updated for py.test 3.x #5277
Description
Activity
- changed the title
[-]Test suite needs to be updated for pytest 3.x[/-][+]Test suite needs to be updated for py.test 3.x[/+]on Aug 29, 2016 EDIT: The results below are generated after I changed
[pytest]to[tool:pytest]insetup.cfg, as suggested above.Wow...
8242 passed, 118 skipped, 37 xfailed, 721 pytest-warnings, 223 error in 308.78 secondsLog attached (no
--remote-data):
ztestlog.txtNote: For another project using Astropy package template,
.. doctest-skip-alldirective no longer works under pytest 3.x as well.Appveyor breakage of astropy-helpers is also due to pytest 3.x (astropy/astropy-helpers#265 ). While the workaround there is super simple, I'm thinking about to pin the version number of pytest to the one that gets bundled into astropy to be a bit more self consistent.
Once we fix things, we should perhaps use latest pytest for the allowed-to-fail run (with numpy-dev and remote-data).
OK, the helpers failure is a red herring, but the pytest behaviour is indeed changed. So the failure was within ci-helpers and that got fixed today.
The new puzzling issue is that astropy-helpers should have never picked up the ci-helpers test file and indeed it doesn't do it in travis only on appveyor.While trying to investigate, I wanted to just use regular commands for testing so I could focus on specific packages. This, it turns out, was foreseen: one can just do
export ASTROPY_USE_SYSTEM_PYTEST=1 python3 setup.py test -P ...There seem to be some different failures:
- tests using the
yieldstructure are deprecated; these are easy to fix, and @kelle started some fixes forwcs-- I have an additional one forunits. - for doctests, the output sometimes no longer has line breaks that used to be present.
-
.. doctest-skipseems to get ignored (both individual and whole-file).
EDIT: Fixed in PRs mentioned here.
- tests using the
@mhvk , I spent a good few hours trying to understand the doctest failure but got nowhere. I tried inserting print statements but they don't work with pytest, as it suppresses print outputs, I think. There is no traceback, just some obscure deprecation warnings. And no clear documented translation guide for those deprecations. 😰
Some warnings are due to a bug in pytest 3.0.5, see pytest-dev/pytest#2118, although that still does not explain why doctest is not skipping some files/snippets as it should. FYI.
Update: To avoid those warnings, pin pytest to 3.0.4 until the bug is fixed. Mind you, doctest still does not work properly but we will have less warnings to worry about.
I've been trying to narrow down a bit. A first conclusion is that the problem only occurs for text files, not for source files (modules; e.g.,
__doctest_skip__in *.py files does work). A second is that the options we pass on, in particularNORMALIZE_WHITESPACEinpytest_plugins.pytest_configuredoes not seem to get used for*.rstfiles.
...
...
...
Looking in more detail, the main problem seems to be that there is nocollectmethod defined for DocTestTextfilePlus. Indeed, by adding a collect method following the pytestDoctestTextfileclass, I can get the tests to pass! (well, minus some annoyingvotests, but what's new...) See #5688.This just now appeared, I think because our CI switched to the recently released pytest 3.2:
https://travis-ci.org/hipspy/hips/jobs/260162492#L1211INTERNALERROR> AttributeError: 'DocTestTextfilePlus' object has no attribute '_location'Should I file a new issue about this? In which tracker?
We're on it. Long story short it's not yet clear whether it's a bug in our plugin or in pytest so we need to limit the version number of pytest for now, I'm waiting for the builds to pass on ci-helpers (astropy/ci-helpers#222).
At the moment, our test suite fails if using pytest 3.x. Of course, we're not seeing that in the CI because we use the (old) bundled version of pytest 3.x, so this isn't too urgent. But to see the failures, you can do:
which gives: