Skip to content

Parallel testing on Python >= 3.3 breaks due to pseudo-nondeterministic test collection #2871

Closed
#2872
Closed
Parallel testing on Python >= 3.3 breaks due to pseudo-nondeterministic test collection#2871
#2872
Labels
Upstream Action RequiredWas: Upstream Fix Requiredtabletesting

Description

@embray
embray
opened on Aug 19, 2014

I will post the full output below, but the relevant portion is this:

INTERNALERROR> AssertionError: Different tests were collected between gw5 and gw3. The difference is:
INTERNALERROR> --- gw5
INTERNALERROR> 
INTERNALERROR> +++ gw3
INTERNALERROR> 
INTERNALERROR> @@ -2693,7 +2693,7 @@
INTERNALERROR> 
INTERNALERROR>  astropy/table/np_utils.py::astropy.table.np_utils.vstack
INTERNALERROR>  astropy/table/operations.py::astropy.table.operations.hstack
INTERNALERROR>  astropy/table/operations.py::astropy.table.operations.vstack
INTERNALERROR> -astropy/table/table.py::astropy.table.table.Row
INTERNALERROR> +astropy/table/table.py::astropy.table.table.Table.Row
INTERNALERROR>  astropy/table/table.py::astropy.table.table.Table.add_column
INTERNALERROR>  astropy/table/table.py::astropy.table.table.Table.add_columns
INTERNALERROR>  astropy/table/table.py::astropy.table.table.Table.add_row

I think that what's happening here is that astropy/table/table.py::astropy.table.table.Row and astropy/table/table.py::astropy.table.table.Table.Row both refer to the same object. But for some reason one test runner is finding the latter before the former. I'm going ahead and marking this "upstream fix required" because the test collector should be able to recognize those as the same object, though I'm not sure about the details.

This might also not be just on Python 3.4, but that was the only Python version where I could reproduce the issue. It's probably something pseudo-nondeterminisitic.

====================================================== test session starts =======================================================
platform linux -- Python 3.4.0 -- pytest-2.5.1

Running tests with Astropy version 1.0.dev9659.
Running tests in /tmp/astropy-test-moymwonq/lib.linux-x86_64-3.4/astropy /internal/1/root/src/astropy/astropy/docs.

Platform: Linux-2.6.32-431.20.3.el6.x86_64-x86_64-with-redhat-6.5-Santiago

Executable: /internal/1/root/bin/overrides/python3.4

Full Python Version: 
3.4.0 (default, Apr 29 2014, 16:13:55) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)]

encodings: sys: utf-8, locale: UTF-8, filesystem: utf-8
byteorder: little
float info: dig: 15, mant_dig: 15

Numpy: 1.9.0.dev-7f46687
Scipy: 0.13.3
Matplotlib: not available
h5py: not available
Running doctests in .rst files is not supported on Python 3.x

plugins: xdist, cov
gw0 [5386] / gw1 [5386] / gw2 [5386] / gw3 [5386] / gw4 [5386] / gw5 [5386]
scheduling tests via LoadScheduling
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "_pytest.main", line 81, in wrap_session
INTERNALERROR>     doit(config, session)
INTERNALERROR>   File "_pytest.main", line 117, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "_pytest.core", line 376, in __call__
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "_pytest.core", line 387, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "_pytest.core", line 288, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/xdist/dsession.py", line 244, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/xdist/dsession.py", line 262, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/xdist/dsession.py", line 315, in slave_collectionfinish
INTERNALERROR>     self.sched.init_distribute()
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/xdist/dsession.py", line 138, in init_distribute
INTERNALERROR>     node.gateway.id,
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/xdist/dsession.py", line 188, in report_collection_diff
INTERNALERROR>     raise AssertionError(msg)
INTERNALERROR> AssertionError: Different tests were collected between gw5 and gw3. The difference is:
INTERNALERROR> --- gw5
INTERNALERROR> 
INTERNALERROR> +++ gw3
INTERNALERROR> 
INTERNALERROR> @@ -2693,7 +2693,7 @@
INTERNALERROR> 
INTERNALERROR>  astropy/table/np_utils.py::astropy.table.np_utils.vstack
INTERNALERROR>  astropy/table/operations.py::astropy.table.operations.hstack
INTERNALERROR>  astropy/table/operations.py::astropy.table.operations.vstack
INTERNALERROR> -astropy/table/table.py::astropy.table.table.Row
INTERNALERROR> +astropy/table/table.py::astropy.table.table.Table.Row
INTERNALERROR>  astropy/table/table.py::astropy.table.table.Table.add_column
INTERNALERROR>  astropy/table/table.py::astropy.table.table.Table.add_columns
INTERNALERROR>  astropy/table/table.py::astropy.table.table.Table.add_row
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/main.py", line 81, in wrap_session
INTERNALERROR>     doit(config, session)
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/main.py", line 118, in _main
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/main.py", line 81, in wrap_session
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/core.py", line 377, in __call__
INTERNALERROR>     doit(config, session)
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/main.py", line 118, in _main
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/core.py", line 388, in _docall
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/core.py", line 377, in __call__
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/core.py", line 289, in execute
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/core.py", line 388, in _docall
INTERNALERROR>   File "<remote exec>", line 47, in pytest_runtestloop
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/execnet/gateway_base.py", line 701, in receive
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/core.py", line 289, in execute
INTERNALERROR>     raise self._getremoteerror() or EOFError()
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/main.py", line 81, in wrap_session
INTERNALERROR> EOFError
INTERNALERROR>   File "<remote exec>", line 47, in pytest_runtestloop
INTERNALERROR>     doit(config, session)
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/execnet/gateway_base.py", line 701, in receive
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/main.py", line 118, in _main
INTERNALERROR>     raise self._getremoteerror() or EOFError()
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR> EOFError
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/core.py", line 377, in __call__
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/core.py", line 388, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/core.py", line 289, in execute
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "<remote exec>", line 47, in pytest_runtestloop
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/main.py", line 81, in wrap_session
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/execnet/gateway_base.py", line 701, in receive
INTERNALERROR>     doit(config, session)
INTERNALERROR>     raise self._getremoteerror() or EOFError()
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/main.py", line 118, in _main
INTERNALERROR> EOFError
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/core.py", line 377, in __call__
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/core.py", line 388, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/core.py", line 289, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "<remote exec>", line 47, in pytest_runtestloop
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/execnet/gateway_base.py", line 701, in receive
INTERNALERROR>     raise self._getremoteerror() or EOFError()
INTERNALERROR> EOFError
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/main.py", line 81, in wrap_session
INTERNALERROR>     doit(config, session)
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/main.py", line 118, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/core.py", line 377, in __call__
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/core.py", line 388, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/core.py", line 289, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "<remote exec>", line 47, in pytest_runtestloop
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/execnet/gateway_base.py", line 701, in receive
INTERNALERROR>     raise self._getremoteerror() or EOFError()
INTERNALERROR> EOFError
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/main.py", line 81, in wrap_session
INTERNALERROR>     doit(config, session)
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/main.py", line 118, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/core.py", line 377, in __call__
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/core.py", line 388, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/_pytest/core.py", line 289, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "<remote exec>", line 47, in pytest_runtestloop
INTERNALERROR>   File "/internal/1/root/usr/local/lib/python3.4/site-packages/execnet/gateway_base.py", line 701, in receive
INTERNALERROR>     raise self._getremoteerror() or EOFError()
INTERNALERROR> EOFError
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/astropy-test-moymwonq/lib.linux-x86_64-3.4/astropy/extern/bundled/six.py' mode='r' encoding='utf-8'>
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/astropy-test-moymwonq/lib.linux-x86_64-3.4/astropy/extern/bundled/six.py' mode='r' encoding='utf-8'>
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/astropy-test-moymwonq/lib.linux-x86_64-3.4/astropy/extern/bundled/six.py' mode='r' encoding='utf-8'>

Activity

  1. embray
    added
    Upstream Fix Required
    on Aug 19, 2014

  2. embray commented on Aug 19, 2014

    @embray
    embray
    on Aug 19, 2014
    MemberAuthor

    Probably related: https://bitbucket.org/hpk42/pytest/issue/545/_pytest-markpy

  3. embray commented on Aug 19, 2014

    @embray
    embray
    on Aug 19, 2014
    MemberAuthor

    Also related: https://bitbucket.org/hpk42/pytest/issue/437/different-tests-collected-on-two-nodes

  4. embray commented on Aug 19, 2014

    @embray
    embray
    on Aug 19, 2014
    MemberAuthor

    A possible workaround might be to move the Row class to its own submodule. Looking at the code it seems like it shouldn't be a hard move to make, and it can't hurt to break up long modules a little where possible. At the same time, I don't know if it's worth it so I'm not going to make that call right now.

  5. taldcroft commented on Aug 19, 2014

    @taldcroft
    taldcroft
    on Aug 19, 2014
    Member

    Row is moved in #2872.

  6. embray commented on Aug 19, 2014

    @embray
    embray
    on Aug 19, 2014
    MemberAuthor

    #2872 seems to make progress, but then the same problem in pytest rears its ugly head here:

    INTERNALERROR> AssertionError: Different tests were collected between gw2 and gw4. The difference is:
INTERNALERROR> --- gw2
INTERNALERROR> 
INTERNALERROR> +++ gw4
INTERNALERROR> 
INTERNALERROR> @@ -3447,7 +3447,7 @@
INTERNALERROR> 
INTERNALERROR>  astropy/tests/tests/test_socketblocker.py::test_localconnect_succeeds[localhost]
INTERNALERROR>  astropy/tests/tests/test_socketblocker.py::test_localconnect_succeeds[127.0.0.1]
INTERNALERROR>  astropy/time/core.py::astropy.time.core.Time.get_delta_ut1_utc
INTERNALERROR> -astropy/time/core.py::astropy.time.core.val
INTERNALERROR> +astropy/time/core.py::astropy.time.core.TimeDatetime
INTERNALERROR>  astropy/time/tests/test_basic.py::TestBasic::()::test_simple
INTERNALERROR>  astropy/time/tests/test_basic.py::TestBasic::()::test_copy_time
INTERNALERROR>  astropy/time/tests/test_basic.py::TestBasic::()::test_getitem
INTERNALERROR> Traceback (most recent call last):

    Seems to be the same sort of problem :( I'm not sure there's much can be done there except wait for a fix in pytest. In the meantime #2872 is not a bad idea in its own right.

  7. taldcroft
    closed this as completedin #2872on Aug 20, 2014
  8. embray
    reopened this on Aug 20, 2014

  9. embray commented on Aug 20, 2014

    @embray
    embray
    on Aug 20, 2014
    MemberAuthor

    Reopening but changing the name to not specifically implicate Table.Row

  10. embray
    changed the title [-]Table.Row breaks parallel testing on Python 3.4[/-] [+]Parallel testing on Python >= 3.3 breaks due to pseudo-nondeterministic test collection[/+] on Aug 20, 2014
  11. bsipocz
    removed
    Affects-devPRs and issues that do not impact an existing Astropy release
    on Feb 9, 2017

  12. astrofrog commented on Aug 2, 2017

    @astrofrog
    astrofrog
    on Aug 2, 2017
    Member

    The latest issues appear to be due to a bug in pytest-xdist: pytest-dev/pytest-xdist#198

  13. astrofrog commented on Aug 2, 2017

    @astrofrog
    astrofrog
    on Aug 2, 2017
    Member

    @drdavella - changing the astropy coordinates fixtures to function level fixes things by the way, so I'll open a quick PR for that

  14. astrofrog
    closed this as completedin #6415on Aug 3, 2017

  15. astrofrog commented on Aug 3, 2017

    @astrofrog
    astrofrog
    on Aug 3, 2017
    Member

    This is now all fixed and we are running the tests with --parallel on CircleCI to prevent regressions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Upstream Action RequiredWas: Upstream Fix Requiredtabletesting

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions