Running tests using py.test command-line script

I've tried running the tests using the command-line py.test script with version 4ddb3d7 of the core astropy repository:

git clone git://github.com/astropy/astropy.git astropy_testing
cd astropy_testing
cd astropy
py.test

If I do this, then 809 tests fail. I just wanted to check whether this is ok, or whether this should skip tests that can not be run without compiling (i.e. should it be mostly skipped tests rather than failures).

I didn't even know there was a py.test command line script. I'd lean toward not using it at all: nosetests has the same problem, which is why nose includes the setup.py nosetests command: That's the only way it can find out about extension modules and make sure they're importable.

It's conceivable that there's a way to write a plugin that would make this work. But I'd lean toward "don't do that".

This will also fail with code that requires 2to3. Between code that requires C extensions and code requiring 2to3, I wonder if there will be much that can be usefully tested this way, and whether it's worth the effort to declare tests as skipped. Personally, I'd rather encourage a single way to test that relies on a built and working tree, rather than supporting two different approaches, one of which can't really be expected to work. (i.e. saying we support this increases our testing burden).

Another reason this approach won't work later is that it chops off the front of the namespace. When I was testing this on the vo branch, there is now "astropy.io", so "import io" actually imports "astropy.io" rather than the stdlib "io", which obviously doesn't work too well.

Personally, I think python setup.py test is great and should be the encouraged usage -- it performs all of the necessary steps to build and then test. (Though I just noticed it doesn't do 2to3 -- we probably will want to fix that at some point, too. It builds C extensions inplace and then runs that tree. A better approach might be to build normally and then run tests in the build tree).

That's just my 2c.

I definitely plan on fixing the test command to do 2to3. It's a little tricky because it requires messing with the path, but I know it can be done. I just wish tools like nose and py.test had built in support for 2to3. You'd think it's a common enough use case that there'd be a fix for it by now. Hmmm...

FWIW, here's what I do:

python3.2 setup.py build
py.test-3.2 build/lib.linux-x86_64-3.2

It should be easy enough to modify the test command to do the equivalent. The nice thing about this is it tests exactly what will be installed -- if it works when installed, it should work here without special cases to handle C extensions, 2to3 etc.

That shouldn't be too bad then, especially since the 'test' command, as written, runs the tests in a separate process anyways.

One important thing I just discovered: currently build is not running 2to3 as far as I can tell - we need to add in the use_2to3 = True keyword in the call to setup(). If I do that, I 2to3 seems to run fine, but it doesn't use the 2to3ed version when I run python setup.py test. The docs for distribute (http://packages.python.org/distribute/python3.html) seem to indicate that's supposed to work, though... so maybe there's some way to adjust our test command to inherit from distribute in a way that takes care of this?

Oh, but as I said in some pull requests, I think the command line script is a very good thing for pacakges that don't have any C code (or when you're not running in py 3). It makes it easy to just run the test for a given package by cding into the directory and doing py.test tests. I think we want to leave this option available, even if we don't generally recommend its use.

There's nothing gained from subclassing the built-in test command since it is not at all compatible with py.test. Adding in support for 2to3 can be done in more or less the same way it does though.

I'm surprised we didn't already have use_2to3 in setup.py--I could've sworn I'd seen it there before, but maybe I just added it on one of my own branches.

On second look, I suppose I could subclass distribute's test command just so I don't have to rewrite its with_project_on_sys_path() method. Though everything else would have to be overridden.

Well, if it's easier to leave things alone and just figure out a way to convince the test command to use the version built for 3.x, that's fine too. I don't think there's much point in trying to be too compatible with distutils given how confusing it already is. As long as 2to3 is run before the tests are run (and they get run on the 2to3ed code), the route there doesn't seem too important.

Oh, and I added use_2to3 is master in commit cece59d

so if you do setup.py test and then py.test or setup.py develop and then py.test, do the failures go away? If so, I think we can close this as "expected behavior". Or alternatively, we could add a note in the testing guidelines about this (e.g. something like "if you want to use the command line py.test be sure to use developer mode)

@astrofrog

@astrofrog - sorry, I was unclear in the last comment - it was directed at you. Assuming that works, we can probably close this?

@eteq

@eteq: if I do python setup.py test, all tests pass. If I then do py.test in the root directory all tests pass but I get 13 errors that are related to collecting tests in build:

 ERROR collecting build/lib.macosx-10.6-x86_64-2.7/astropy/constants/tests/test_constant.py 
import file mismatch:
imported module 'astropy.constants.tests.test_constant' has this __file__ attribute:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/astropy/constants/tests/test_constant.py
which is not the same as the test file we want to collect:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/build/lib.macosx-10.6-x86_64-2.7/astropy/constants/tests/test_constant.py
HINT: use a unique basename for your test file modules
 ERROR collecting build/lib.macosx-10.6-x86_64-2.7/astropy/io/vo/tests/converter_test.py 
import file mismatch:
imported module 'astropy.io.vo.tests.converter_test' has this __file__ attribute:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/astropy/io/vo/tests/converter_test.py
which is not the same as the test file we want to collect:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/build/lib.macosx-10.6-x86_64-2.7/astropy/io/vo/tests/converter_test.py
HINT: use a unique basename for your test file modules
 ERROR collecting build/lib.macosx-10.6-x86_64-2.7/astropy/io/vo/tests/exception_test.py 
import file mismatch:
imported module 'astropy.io.vo.tests.exception_test' has this __file__ attribute:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/astropy/io/vo/tests/exception_test.py
which is not the same as the test file we want to collect:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/build/lib.macosx-10.6-x86_64-2.7/astropy/io/vo/tests/exception_test.py
HINT: use a unique basename for your test file modules
 ERROR collecting build/lib.macosx-10.6-x86_64-2.7/astropy/io/vo/tests/tree_test.py 
import file mismatch:
imported module 'astropy.io.vo.tests.tree_test' has this __file__ attribute:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/astropy/io/vo/tests/tree_test.py
which is not the same as the test file we want to collect:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/build/lib.macosx-10.6-x86_64-2.7/astropy/io/vo/tests/tree_test.py
HINT: use a unique basename for your test file modules
 ERROR collecting build/lib.macosx-10.6-x86_64-2.7/astropy/io/vo/tests/ucd_test.py 
import file mismatch:
imported module 'astropy.io.vo.tests.ucd_test' has this __file__ attribute:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/astropy/io/vo/tests/ucd_test.py
which is not the same as the test file we want to collect:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/build/lib.macosx-10.6-x86_64-2.7/astropy/io/vo/tests/ucd_test.py
HINT: use a unique basename for your test file modules
 ERROR collecting build/lib.macosx-10.6-x86_64-2.7/astropy/io/vo/tests/util_test.py 
import file mismatch:
imported module 'astropy.io.vo.tests.util_test' has this __file__ attribute:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/astropy/io/vo/tests/util_test.py
which is not the same as the test file we want to collect:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/build/lib.macosx-10.6-x86_64-2.7/astropy/io/vo/tests/util_test.py
HINT: use a unique basename for your test file modules
 ERROR collecting build/lib.macosx-10.6-x86_64-2.7/astropy/io/vo/tests/vo_test.py 
import file mismatch:
imported module 'astropy.io.vo.tests.vo_test' has this __file__ attribute:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/astropy/io/vo/tests/vo_test.py
which is not the same as the test file we want to collect:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/build/lib.macosx-10.6-x86_64-2.7/astropy/io/vo/tests/vo_test.py
HINT: use a unique basename for your test file modules
 ERROR collecting build/lib.macosx-10.6-x86_64-2.7/astropy/tests/tests/test_run_tests.py 
import file mismatch:
imported module 'astropy.tests.tests.test_run_tests' has this __file__ attribute:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/astropy/tests/tests/test_run_tests.py
which is not the same as the test file we want to collect:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/build/lib.macosx-10.6-x86_64-2.7/astropy/tests/tests/test_run_tests.py
HINT: use a unique basename for your test file modules
 ERROR collecting build/lib.macosx-10.6-x86_64-2.7/astropy/tests/tests/test_skip_remote_data.py 
import file mismatch:
imported module 'astropy.tests.tests.test_skip_remote_data' has this __file__ attribute:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/astropy/tests/tests/test_skip_remote_data.py
which is not the same as the test file we want to collect:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/build/lib.macosx-10.6-x86_64-2.7/astropy/tests/tests/test_skip_remote_data.py
HINT: use a unique basename for your test file modules
 ERROR collecting build/lib.macosx-10.6-x86_64-2.7/astropy/utils/tests/test_odict.py 
import file mismatch:
imported module 'astropy.utils.tests.test_odict' has this __file__ attribute:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/astropy/utils/tests/test_odict.py
which is not the same as the test file we want to collect:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/build/lib.macosx-10.6-x86_64-2.7/astropy/utils/tests/test_odict.py
HINT: use a unique basename for your test file modules
 ERROR collecting build/lib.macosx-10.6-x86_64-2.7/astropy/wcs/tests/test_profiling.py 
import file mismatch:
imported module 'astropy.wcs.tests.test_profiling' has this __file__ attribute:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/astropy/wcs/tests/test_profiling.py
which is not the same as the test file we want to collect:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/build/lib.macosx-10.6-x86_64-2.7/astropy/wcs/tests/test_profiling.py
HINT: use a unique basename for your test file modules
 ERROR collecting build/lib.macosx-10.6-x86_64-2.7/astropy/wcs/tests/test_wcs.py 
import file mismatch:
imported module 'astropy.wcs.tests.test_wcs' has this __file__ attribute:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/astropy/wcs/tests/test_wcs.py
which is not the same as the test file we want to collect:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/build/lib.macosx-10.6-x86_64-2.7/astropy/wcs/tests/test_wcs.py
HINT: use a unique basename for your test file modules
 ERROR collecting build/lib.macosx-10.6-x86_64-2.7/astropy/wcs/tests/test_wcsprm.py 
import file mismatch:
imported module 'astropy.wcs.tests.test_wcsprm' has this __file__ attribute:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/astropy/wcs/tests/test_wcsprm.py
which is not the same as the test file we want to collect:
  /Users/tom/tmp/astropy_testing/astropy/astropy_testing_new/build/lib.macosx-10.6-x86_64-2.7/astropy/wcs/tests/test_wcsprm.py
HINT: use a unique basename for your test file modules

If I go into astropy and run py.test, all tests pass.

If I re-clone and do python setup.py develop then py.test in the root then all tests pass.

There are two solutions to this:

Tell anyone who uses py.test at the command line to call it as py.test astropy
Add a line in setup.cfg that tells py.test not to search for tests in build. I test this just now and it prevents the erroring tests you report here.

My preference is for option 2 - if that's ok with you, I'll make this change in master and close this issue.

I like option 2 as well. I think we should also make sure that we explicitly say somewhere in the docs that one cannot simply run py.test in the root directory before setup.

added setction in setup.cfg to not test build directory by default. C…

@jiffyclub

I added some documentation about this in 92c4a25 - @jiffyclub, you may want to take note of these changes and include them in the changes you're going to make to the testing docs with the pep8 addition.

added setction in setup.cfg to not test build directory by default. C…

Merge pull request astropy#53 from kali-maa/dependabot/add-v2-config-…

Merge pull request astropy#53 from sudheesh001/Patch_46

Navigation Menu

Repository navigation

Running tests using py.test command-line script #53

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Running tests using py.test command-line script #53

Description

Activity

embray commented on Nov 7, 2011

mdboom commented on Nov 7, 2011

embray commented on Nov 7, 2011

mdboom commented on Nov 7, 2011

embray commented on Nov 7, 2011

eteq commented on Nov 7, 2011

eteq commented on Nov 7, 2011

embray commented on Nov 7, 2011

embray commented on Nov 7, 2011

eteq commented on Nov 7, 2011

eteq commented on Nov 7, 2011

eteq commented on Nov 9, 2011

eteq commented on Nov 11, 2011

astrofrog commented on Nov 13, 2011

eteq commented on Nov 14, 2011

astrofrog commented on Nov 14, 2011

eteq commented on Nov 14, 2011

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions