Skip to content

Non-deterministic test collection order for deterministic fixtures makes xdist fail #920

Closed
Closed
Non-deterministic test collection order for deterministic fixtures makes xdist fail#920
Labels
plugin: xdistrelated to the xdist external plugintype: bugproblem that needs to be addressed

Description

@aldanor
aldanor
opened on Aug 6, 2015

Issues that may be related: #437, #594, #596, #669.

pytest-xdist seems to fail sometimes because it orders parametrized fixtures differently -- e.g., when a fixture depends on two or more parametrized fixtures. Using the latest pytest and pytest-xdist on Python 3.4 with this test script:

import pytest

@pytest.fixture(params=['foo', 'bar'])
def fixture1(request):
    return request.param

@pytest.fixture(params=['baz', 'foo'])
def fixture2(request):
    return request.param

def test_1(fixture1, fixture2):
    pass

Sometimes it works:

$ py.test -v -n 2
============================ test session starts ============================
platform linux -- Python 3.4.3 -- py-1.4.30 -- pytest-2.7.2 -- /home/test/env/bin/python3
plugins: xdist
[gw0] linux Python 3.4.3 cwd: /home/test/foo
[gw1] linux Python 3.4.3 cwd: /home/test/foo
[gw1] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
[gw0] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
gw0 [4] / gw1 [4]
scheduling tests via LoadScheduling

foo/test_1.py::test_1[2-3]
foo/test_1.py::test_1[1-3]
[gw0] PASSED foo/test_1.py::test_1[2-3]
[gw1] PASSED foo/test_1.py::test_1[1-3]
foo/test_1.py::test_1[1-4]
foo/test_1.py::test_1[2-4]
[gw0] PASSED foo/test_1.py::test_1[2-4]
[gw1] PASSED foo/test_1.py::test_1[1-4]

========================= 4 passed in 0.60 seconds ==========================

And sometimes it doesn't (because it chose to sort the fixtures by the second parameter first for some reason):

$ py.test -v -n 2
============================ test session starts ============================
platform linux -- Python 3.4.3 -- py-1.4.30 -- pytest-2.7.2 -- /home/test/env/bin/python3
plugins: xdist
[gw0] linux Python 3.4.3 cwd: /home/test/foo
[gw1] linux Python 3.4.3 cwd: /home/test/foo
[gw1] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
[gw0] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
gw0 [4] / gw1 [4]
scheduling tests via LoadScheduling
collecting 0 items / 1 errors
================================== ERRORS ===================================
___________________________ ERROR collecting gw0 ____________________________
Different tests were collected between gw1 and gw0. The difference is:
--- gw1

+++ gw0

@@ -1,4 +1,4 @@

 foo/test_1.py::test_1[foo-foo]
+foo/test_1.py::test_1[foo-bar]
+foo/test_1.py::test_1[bar-bar]
 foo/test_1.py::test_1[bar-foo]
-foo/test_1.py::test_1[bar-bar]
-foo/test_1.py::test_1[foo-bar]
========================== 1 error in 0.65 seconds ==========================

Activity

  1. aldanor
    changed the title [-]Non-deterministic test collection order for determinstic fixtures makes xdist fail[/-] [+]Non-deterministic test collection order for deterministic fixtures makes xdist fail[/+] on Aug 6, 2015

  2. RonnyPfannschmidt commented on Aug 6, 2015

    @RonnyPfannschmidt
    RonnyPfannschmidt
    on Aug 6, 2015
    Member

    interesting find, my initial guess is, this is a artifact of the python hash-seed and the fixture management is just sorted different in some mappings

    it'll take a while to figure the details, and it'll happen after the next release of xdist (which i am working on)

  3. aldanor commented on Aug 6, 2015

    @aldanor
    aldanor
    on Aug 6, 2015
    Author

    Yea, the test collection mechanism is a bit flaky – in a much larger example which I won't be able to reproduce here I was able to work around it by changing some fixtures from session-scoped to function-scoped, I have no clue why that helped. Also, this doesn't seem to happen on Python 2 (not that I've seen it fail, at least).

    By the way, I wonder why should it care about the order at all if the sets are the same?

  4. nicoddemus
    added
    type: bugproblem that needs to be addressed
    plugin: xdistrelated to the xdist external plugin
    on Aug 7, 2015

  5. RonnyPfannschmidt commented on Aug 7, 2015

    @RonnyPfannschmidt
    RonnyPfannschmidt
    on Aug 7, 2015
    Member

    That goes a few years back, I recall its a workaround for duplicate test ids or something like that

  6. jstasiak commented on May 5, 2016

    @jstasiak
    jstasiak
    on May 5, 2016

    @aldanor FYI I fail to be able to reproduce the failure using the example code you provided. My software: Python 2.7.9 and 3.5.1, pytest 2.9.1, xdist-1.14.

  7. nicoddemus commented on Jun 18, 2016

    @nicoddemus
    nicoddemus
    on Jun 18, 2016
    Member

    Hmm perhaps we can pass master's PYTHON_SEED value to workers? This way dict hashing should be the same across master and workers. This does not actually fix the fact that xdist uses indices to distribute tests, but at least seems sensible and would solve this issue (and others related).

  8. hectorv commented on Jul 1, 2016

    @hectorv
    hectorv
    on Jul 1, 2016

    I'm seeing this while trying to move away from nosetests on a legacy test suite:

    $ py.test -sxv -rw -n4 common/tests

    this is my environment:

    Python 2.7.11
pytest==2.9.2
pytest-xdist==1.14

    Edit: to be clear, this works fine:

    $ py.test -sxv -rw common/tests
  9. wence-
    added 4 commits that reference this issue on Jul 26, 2017

    jenkins: Temporary workaround for pytest-dev/pytest#920

    968b807

    jenkins: Temporary workaround for pytest-dev/pytest#920

    dd4b3c5

    jenkins: Temporary workaround for pytest-dev/pytest#920

    eec5060

    jenkins: Temporary workaround for pytest-dev/pytest#920

    0aa80bc
  10. nicoddemus
    closed this as completedin a546a61on Aug 1, 2017

  11. renefritze commented on Aug 2, 2017

    @renefritze
    renefritze
    on Aug 2, 2017

    I think we still see this in our project with pytest 3.2 and xdist 1.18.2:
    https://travis-ci.org/pymor/pymor/jobs/260181761#L744

  12. RonnyPfannschmidt commented on Aug 2, 2017

    @RonnyPfannschmidt
    RonnyPfannschmidt
    on Aug 2, 2017
    Member

    @renemilk the issue you show is a different problem that comes from your own self-made parametrization system

    all your tests that differ in order may be in a different place based on pythonhashseed
    as such pytest cant know the order itself since the methods share the order

  13. renefritze commented on Aug 2, 2017

    @renefritze
    renefritze
    on Aug 2, 2017

    Ah, ok. I misunderstood then. Setting PYTHONHASHSEED=0 in env indeed fixes the ordering issue for me locally. Thanks.

  14. webknjaz
    added a commit that references this issue on Oct 21, 2018

    Unparallelize pytest for Python 3.5 everywhere

    1778189

  15. liiight commented on Jan 21, 2019

    @liiight
    liiight
    on Jan 21, 2019

    I think I may have the same issue. I dynamically generate parameterization based on a txt file from S3:

    def parametrize_tenant_from_s3_file(metafunc):
    file_path = metafunc.config.getoption('--tenant-list-file')
    profile_name = metafunc.config.getoption('--aws-profile-name')
    aws_session = AWSSession(profile_name=profile_name)
    s3 = S3(aws_session.session)
    parsed_link = parse.urlparse(file_path)
    file_name = Path.cwd() / 'tenant_list.txt'
    s3.download_file(bucket=parsed_link.netloc, key=parsed_link.path.lstrip('/'), path=file_name)
    metafunc.parametrize('tenant_id_from_s3_file', file_name.read_text().splitlines())

    Running this without -n works:

    Launching pytest with arguments -m test_marker --collect-only --aws-profile-name prod /Users/orcarmi/PycharmProjects/shield/tests in /Users/orcarmi/PycharmProjects/shield

============================= test session starts ==============================
platform darwin -- Python 3.6.5, pytest-3.8.2, py-1.5.3, pluggy-0.8.0 -- /Users/orcarmi/PycharmProjects/shield/venv/bin/python
cachedir: .pytest_cache
metadata: {'Python': '3.6.5', 'Platform': 'Darwin-18.2.0-x86_64-i386-64bit', 'Packages': {'pytest': '3.8.2', 'py': '1.5.3', 'pluggy': '0.8.0'}, 'Plugins': {'xdist': '1.22.2', 'rerunfailures': '4.2', 'repeat': '0.5.0', 'metadata': '1.7.0', 'lazy-fixture': '0.4.0', 'jira': '0.3.7', 'instafail': '0.4.0', 'html': '1.17.0', 'forked': '0.2', 'clarity': '0.1.0a1', 'allure-pytest': '2.5.2'}, 'Environment': 'prod-eu-2', 'SHIELD Version': '7.0.0'}
SHIELD version: 7.0.0
Environment: prod-eu-2
rootdir: /Users/orcarmi/PycharmProjects/shield, inifile: pytest.ini
plugins: xdist-1.22.2, rerunfailures-4.2, repeat-0.5.0, metadata-1.7.0, lazy-fixture-0.4.0, jira-0.3.7, instafail-0.4.0, html-1.17.0, forked-0.2, clarity-0.1.0a1, allure-pytest-2.5.2
collecting ... collected 5096 items / 4550 deselected
<Package '/Users/orcarmi/PycharmProjects/shield'>
  <Module 'test_tsd_migration.py'>
    <Function 'test_tsd_customers_migration[tenant_0003a1ef4dfa4360bdcdc175c858e7c6]'>
    <Function 'test_tsd_customers_migration[tenant_001dececa0164f6ba44a1514c6369b42]'>
...

    With -n it does not:

    Launching pytest with arguments -m test_marker -n auto --aws-profile-name prod /Users/orcarmi/PycharmProjects/shield/tests in /Users/orcarmi/PycharmProjects/shield

============================= test session starts ==============================
platform darwin -- Python 3.6.5, pytest-3.8.2, py-1.5.3, pluggy-0.8.0 -- /Users/orcarmi/PycharmProjects/shield/venv/bin/python
cachedir: .pytest_cache
metadata: {'Python': '3.6.5', 'Platform': 'Darwin-18.2.0-x86_64-i386-64bit', 'Packages': {'pytest': '3.8.2', 'py': '1.5.3', 'pluggy': '0.8.0'}, 'Plugins': {'xdist': '1.22.2', 'rerunfailures': '4.2', 'repeat': '0.5.0', 'metadata': '1.7.0', 'lazy-fixture': '0.4.0', 'jira': '0.3.7', 'instafail': '0.4.0', 'html': '1.17.0', 'forked': '0.2', 'clarity': '0.1.0a1', 'allure-pytest': '2.5.2'}, 'Environment': 'prod-eu-2', 'SHIELD Version': '7.0.0'}
SHIELD version: 7.0.0
Environment: prod-eu-2
rootdir: /Users/orcarmi/PycharmProjects/shield, inifile: pytest.ini
plugins: xdist-1.22.2, rerunfailures-4.2, repeat-0.5.0, metadata-1.7.0, lazy-fixture-0.4.0, jira-0.3.7, instafail-0.4.0, html-1.17.0, forked-0.2, clarity-0.1.0a1, allure-pytest-2.5.2
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I
[gw0] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw1] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw2] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw3] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw4] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw5] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw6] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw7] darwin Python 3.6.5 cwd: /Users/orcarmi/PycharmProjects/shield
[gw0] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw1] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw2] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw3] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw4] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw5] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw6] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw7] Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
gw0 [0] / gw1 [0] / gw2 [0] / gw3 [0] / gw4 [0] / gw5 [0] / gw6 [0] / gw7 [0]

scheduling tests via LoadScheduling

 generated xml file: /Users/orcarmi/PycharmProjects/shield/reports/jreport.xml -
 generated html file: /Users/orcarmi/PycharmProjects/shield/reports/report.html 
========================= no tests ran in 6.89 seconds =========================

    I tried setting PYTHONSEEDHASH to no avail.
    Please advise, thanks in advance

    EDIT: Realised I wasn't on latest xdist and pytest, upgrade both to latest, still same issue.

  16. RonnyPfannschmidt commented on Jan 21, 2019

    @RonnyPfannschmidt
    RonnyPfannschmidt
    on Jan 21, 2019
    Member

    @liiight based on what you show its a completely different issue please open new issues instead of recycling related looking but actually fixed issues

  17. liiight commented on Jan 22, 2019

    @liiight
    liiight
    on Jan 22, 2019 via email

  18. maratumba commented on Dec 1, 2021

    @maratumba
    maratumba
    on Dec 1, 2021

    I just encountered this issue while collecting tests with parametrized fixtures (pytest==6.2.5) in Django. The order of tests changes in each collection.

    # Naive datetime objects for edge cases
# values marked with fuzz are ignored by default, only ran in CI pipelines
naive_datetime_fuzz = [
    pytest.param(
        parse_datetime("2021-11-07T01:30:00"), marks=pytest.mark.fuzz
    ),  # Naive datetime during US DST end
    pytest.param(
        parse_datetime("2020-02-29T09:30:00"), marks=pytest.mark.fuzz
    ),  # Leap Year Feb29th
    pytest.param(
        parse_datetime("2021-11-07T00:00:00.000"), marks=pytest.mark.fuzz
    ),  # Exactly midnight
    pytest.param(faker_uk.past_datetime()),  # Recent time
]


@pytest.fixture(params=naive_datetime_fuzz)
def naive_datetime(request):
    return request.param

    pytest.ini

    [pytest]
addopts = 
    --ds=config.settings.test 
    --reuse-db 
    --strict-markers
    -m "not fuzz"
python_files = tests.py test_*.py
markers =
    slow: marks tests as slow (deselect with '-m "not slow"')
    unit
    e2e
    fuzz
    integration
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    plugin: xdistrelated to the xdist external plugintype: bugproblem that needs to be addressed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions