Skip to content

py.test with xdist is not executing tests parametrized with random values #594

Closed
Closed
py.test with xdist is not executing tests parametrized with random values#594
Labels
plugin: xdistrelated to the xdist external plugintype: bugproblem that needs to be addressed

Description

@pytestbot
pytestbot
opened on Sep 22, 2014

Originally reported by: BitBucket: pytry, GitHub: pytry

I have noticed the following strange behaviour for pytest and xdist.

When trying to run the test that is parametrized with some randomly selected values the test are not actually run. The same test is executed without any problems if xdist is not used.

Following code can be used to reproduce this.

#!python
import pytest
import random

PARAMS_NUMBER = 3
PARAMS = []

for i in range(PARAMS_NUMBER):
    PARAMS.append(random.randrange(0, 1000))

@pytest.mark.parametrize('rand_par', PARAMS)
def test_random_param(rand_par):

    assert 500 > rand_par

Without xdists it works fine.

With xdist no test is executed at all with the following output

#!console
============================= test session starts =============================
platform win32 -- Python 2.7.3 -- py-1.4.24 -- pytest-2.6.2
plugins: xdist
gw0 [3] / gw1 [3] / gw2 [3] / gw3 [3]
scheduling tests via LoadScheduling

==============================  in 1.93 seconds ===============================

Versions I'm using:

python 2.7.3
pytest 2.6.2
pytest-xdist 1.11

Additional note:

With some older versions (xdist 1.8 and pytest 2.4.X or 2.5.X do not remember exactly) the xdist was stopping on assertion in dsession.py

#!python
assert collection == col

Thanks in advance for any help :)

Activity

  1. pytestbot commented on Sep 23, 2014

    @pytestbot
    pytestbot
    on Sep 23, 2014
    ContributorAuthor

    Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

    Each subprocess does its own collection of tests. With -n 2 two subprocesses will collect different tests and thus pytest cannot correctly schedule the tests (or not at all as in your case). There should be an error, not sure why nothing shows up, maybe @flub knows.

  2. pytestbot commented on Sep 23, 2014

    @pytestbot
    pytestbot
    on Sep 23, 2014
    ContributorAuthor

    Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

    To work with random values you can do this:

    #!python

import random
import pytest

@pytest.fixture(params=[0,1,2])
def rand_par():
    return random.randrange(0, 1000)

def test_random_param(rand_par):
    assert 500 > rand_par

    With this, the collection itself is non-random, only the execution involves random numbers.

    The only reason i leave this issue open is that we need to check about the error reporting.

  3. pytestbot commented on Sep 23, 2014

    @pytestbot
    pytestbot
    on Sep 23, 2014
    ContributorAuthor

    Original comment by Bruno Oliveira (BitBucket: nicoddemus, GitHub: nicoddemus):

    @hpk42 I think PR9 might be responsible for not reporting anything at all; the purpose of that patch was to not consume the error that happened when xdist collected different number of tests because of some other internal error (for instance that error during .pyc writing, which has been fixed since).

    Because no internal error is happening here as each slave just has a different collection, I think that problem is just being logged but no further message is appearing in the console. If I'm correct, --debug should provide more information.

    I would be happy to work on this if someone can give me a pointer or two how can I produce an error in xdist when we detect that different nodes were collected, but without overwriting any previous error that might have happened in a previous step.

  4. pytestbot commented on Sep 28, 2014

    @pytestbot
    pytestbot
    on Sep 28, 2014
    ContributorAuthor

    Original comment by BitBucket: pytry, GitHub: pytry:

    Thanks @hpk42

    I'll use proposed solution for random values.

    Are You going to add some proper message in such case? So the user does not have to think why tests are not running at all?
    Honestly now it can be quite confusing (as I have written before it was simply crashing on assertion so at least user had info where to start investigation :) )

  5. pytestbot commented on Sep 28, 2014

    @pytestbot
    pytestbot
    on Sep 28, 2014
    ContributorAuthor

    Original comment by Bruno Oliveira (BitBucket: nicoddemus, GitHub: nicoddemus):

    Hi @Pytry,

    This has been fixed by this PR. :)

    Cheers,

  6. pytestbot commented on Sep 28, 2014

    @pytestbot
    pytestbot
    on Sep 28, 2014
    ContributorAuthor

    Original comment by BitBucket: pytry, GitHub: pytry:

    Cool :)
    Thanks for the update @nicoddemus

  7. pytestbot commented on Sep 30, 2014

    @pytestbot
    pytestbot
    on Sep 30, 2014
    ContributorAuthor

    Original comment by Floris Bruynooghe (BitBucket: flub, GitHub: flub):

    @nicoddemus In which case this issue can be closed?

  8. pytestbot commented on Sep 30, 2014

    @pytestbot
    pytestbot
    on Sep 30, 2014
    ContributorAuthor

    Original comment by Bruno Oliveira (BitBucket: nicoddemus, GitHub: nicoddemus):

    Sure, sorry about that. Thanks @flub.

  9. pytestbot
    added
    type: bugproblem that needs to be addressed
    plugin: xdistrelated to the xdist external plugin
    on Jun 15, 2015
  10. pytestbot
    closed this as completedon Jun 15, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    plugin: xdistrelated to the xdist external plugintype: bugproblem that needs to be addressed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions