Use unittest for tests - #699

Coverage decreased (-0.48%) when pulling 5142743 on hugovk:unittest3 into 6a79d80 on python-pillow:master.

Coverage decreased because more code is included in the report.

Before: 6584 OF 9036 RELEVANT LINES COVERED
After: 6587 OF 9100 RELEVANT LINES COVERED

These two files weren't included before and they brought the average down a bit:

COVERAGE        FILE    LINES   RELEVANT    COVERED MISSED  HITS/LINE
25.0%       PIL/ImageGrab.py    49  20  5   15  1.0
13.64%      PIL/ImageQt.py  89  44  6   38  1.0

Before: https://coveralls.io/builds/852429
After: https://coveralls.io/builds/852673

Yeah, I'm just happy we are tracking coverage 👍

Re the other tests:

• make_hash.py is used in the code (storage.c?) somewhere to determine some hard coded hash constants. We'd need to run that if there are changes that cause a hash collision.
• large_* are tests that are potential DOS against the test machine. Worth keeping, and worth running explicitly before releases, but not all the time.
• bench_cffi_access,py is a benchmark. Again, it's a DOS against the test machine.

These at least are useful.

Ok, this is still somewhat troublesome OMM, or at least for how I'm working.

<parenthetical>

For clarity, my setup is:

A bunch of virtualenvs:

~/vpy26
~/vpy27
~/vpy27-dbg
~/vpy32
~/vpy33
~/vpy33-dbg
~/vpy34
~/vpypy

Plus one Pillow directory. From within the pillow directory, I hack, and then:

source ~vpy27/bin/activate
python setup.py install && python Tests/run.py --installed
[[ repeat ]]

And in another terminal window, I'll have a 3.x virtualenv going with the same set of commands. If necessary, I'll use the dbg versions for gdb, or whatever.

</parenthetical>

The root problem, I think, is that these tests are working against an in place build, not an installed version. We worked around the import issue in the old test suite with the --installed flag.

I don't like in place builds for a few reasons:

• It's harder to do a clean build. The output files are intermingled with the source files. With non-in place builds, it's just rm -f build and I get a clean build across all of my python versions. (or python setup.py clean for just that arch. )
• You can't have Pillow for two different python versions installed at once. This is very useful when I'm tracking down weird crashing bugs.
• It's not immediately obvious which version the .so files are compiled for. You just get strange import errors.
• python setup.py develop was a useful hack pre-virtualenv, It's less useful now that we can have full virtualenvs that don't require sudo for install.

I also think that it's extremely valuable to be able to test the current installed Pillow build without building whatever is currently in the source directory.

@wiredfool Can we support both in-place and installed test running? For whatever it's worth, I think I find installed-test-running less attractive.

I'm not arguing for removing in place -- Clearly others find it useful. I'm arguing for whatever we do retaining the ability to test installed code.

👍

@hugovk Can that be added or do we need to revert again? 😄

OK, so the old runner calls os.environ["PYTHONPATH"] = "." early on when --installed is set.

@wiredfool Would an environment variable be ok rather than a command-line argument? That would be easier to implement for nose, otherwise we'll probably need different handling to run the full set with nose (perhaps via a plugin like this) and individually. I'd like to keep it so the tests can be run individually without nose (python Tests/test_file.py), and altogether in a batch with nose.

(Likewise there should be no problem running individual tests with nose (nosetests Tests/test_file.py) or the whole set without (something like python -m unittest discover -p 'Tests/test_*.py).)

It looks like the old runner calls os.environ["PYTHONPATH"] = "." early on when --installed is not set.

An environment variable is ok, not what I'd prefer, but it's going to be easier than monkeypatching. (the difference between FOO=bar nosetests and nosetests--foo-bar is minimal, just different typing)

There are a couple of possible command line flags for nosetests, --first-package-wins and --no-path-adjustment that might do something useful.

I could swear that there was some monkeying around with sys.path to make the imports work in the old testing setup, but it's just not there. (Well, it's in selftest.py, maybe that's what I was thinking.)

nosetests -s --no-path-adjustment gets the imports of PIL right, but it falls down on importing helper. Relative imports don't help, since we aren't importing the Tests/ module.

e.g., this test file:

import sys
print (sys.path)
import PIL
print (PIL.__file__)
from helper import unittest, PillowTestCase, tearDownModule, lena, py3

yields this:

$ nosetests --no-path-adjustment -s Tests/test_nose.py
['/home/erics/vpy33/bin', '/home/erics/vpy33/lib/python3.3/site-packages/distribute-0.6.34-py3.3.egg', '/home/erics/vpy33/lib/python3.3/site-packages/pip-1.3.1-py3.3.egg', '/home/erics/vpy33/lib/python3.3/site-packages/Pillow-2.4.0-py3.3-linux-x86_64.egg', '/home/erics/vpy33/lib/python3.3', '/home/erics/vpy33/lib/python3.3/plat-linux', '/home/erics/vpy33/lib/python3.3/lib-dynload', '/usr/lib/python3.3', '/usr/lib/python3.3/plat-linux', '/home/erics/vpy33/lib/python3.3/site-packages']
/home/erics/vpy33/lib/python3.3/site-packages/Pillow-2.4.0-py3.3-linux-x86_64.egg/PIL/__init__.py
E
======================================================================
ERROR: Failure: ImportError (No module named 'helper')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/erics/vpy33/lib/python3.3/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/home/erics/vpy33/lib/python3.3/site-packages/nose/loader.py", line 414, in loadTestsFromName
    addr.filename, addr.module)
  File "/home/erics/vpy33/lib/python3.3/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/home/erics/vpy33/lib/python3.3/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/home/erics/vpy33/lib/python3.3/imp.py", line 180, in load_module
    return load_source(name, filename, file)
  File "/home/erics/vpy33/lib/python3.3/imp.py", line 119, in load_source
    _LoadSourceCompatibility(name, pathname, file).load_module(name)
  File "<frozen importlib._bootstrap>", line 584, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1022, in load_module
  File "<frozen importlib._bootstrap>", line 1003, in load_module
  File "<frozen importlib._bootstrap>", line 560, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 868, in _load_module
  File "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed
  File "/home/erics/Pillow/Tests/test_nose.py", line 6, in <module>
    from helper import unittest, PillowTestCase, tearDownModule, lena, py3
ImportError: No module named 'helper'

And also this:

(vpy33)erics@builder-1204-x64:~/Pillow$ nosetests -s Tests/test_nose.py
['/home/erics/Pillow/Tests', '/home/erics/Pillow', '/home/erics/vpy33/bin', '/home/erics/vpy33/lib/python3.3/site-packages/distribute-0.6.34-py3.3.egg', '/home/erics/vpy33/lib/python3.3/site-packages/pip-1.3.1-py3.3.egg', '/home/erics/vpy33/lib/python3.3/site-packages/Pillow-2.4.0-py3.3-linux-x86_64.egg', '/home/erics/vpy33/lib/python3.3', '/home/erics/vpy33/lib/python3.3/plat-linux', '/home/erics/vpy33/lib/python3.3/lib-dynload', '/usr/lib/python3.3', '/usr/lib/python3.3/plat-linux', '/home/erics/vpy33/lib/python3.3/site-packages']
/home/erics/Pillow/PIL/__init__.py

----------------------------------------------------------------------
Ran 0 tests in 0.001s

OK

So, nose, by default adds the absolute path for . and the test location to the python path. I/We need the test location, but not the initial directory.

So far, I haven't been able to get it to work on command lines or from various options to nose.run()

No luck so far.

I found one suggestion saying to create a blank Tests/__init__.py file. This worked locally for me from a 2.x virtualenv and 2.x on Travis but not for 3.x on Travis.

I'm on Windows right now but I had problems on Mac with python setup.py install from a 3.4 virtualenv. It falls over with TypeError("Can't mix strings and bytes in path components.") from posixpath.py.

#706 is (among other things) a test runner helper that does the necessary path twiddling to get nosetests to run on the installed version of Pillow (at least in virtualenvs on ubuntu).

It gets me back to the point that I can test proposed patches against HEAD again.

Great!