Enforce serial execution of related tests

I think this is related to #18 but didn't want to clutter the comment history if that's not the case.

I am looking for a way to enforce certain tests, usually the tests within a specific class, to be executed in serial due to that they interfere with each other. I could use -n0 but that would be an overreach - it would apply to the entire test run, not just for a specific class for instance. Would it be possible to just have a decorator directive on the class to 1. instruct xdist to schedule the tests in the class to the same slave as well as 2. instruct them to be run in serial?

If there are other ways of achieving this, I would be interested in that too.

Thanks for a great product.

@betamos

Hey @betamos,

This is what #18 is about, unfortunately nobody got the time to work on it yet.

Here's a solution that works at my workplace:

Mark the tests you want to execute serially with a special mark, say serial:

@pytest.mark.serial
class Test:        
    ...

@pytest.mark.serial    
def test_foo():
    ...

Execute your parallel tests, excluding those with the serial mark:
```
$ py.test -n auto -m "not serial"
```
Next, execute your serial tests in a separate session:
```
$ py.test -n0 -m "serial"
```

This works perfectly fine in C.I. servers, since it is usually trivial to add an additional command-line to the job.

I will close this now since it is a duplicate of #18, but let me know if you have any other questions.

@nicoddemus

Is there additional setup to use the "n" option, @nicoddemus ?:

py.test -n0 .\src\test\test.py
usage: py.test [options] [file_or_dir] [file_or_dir] [...]
py.test: error: unrecognized arguments: -n0

py.test -n 0 .\src\test\test.py
usage: py.test [options] [file_or_dir] [file_or_dir] [...]
py.test: error: unrecognized arguments: -n

py.test -n0 .\src\test\test_entity.py
usage: py.test [options] [file_or_dir] [file_or_dir] [...]
py.test: error: unrecognized arguments: -n0


platform win32 -- Python 2.7.13, pytest-2.8.3, py-1.4.34, pluggy

It seems you need to install pytest-xdist

@nicoddemus

@nicoddemus Any idea if there is a better way to make some tests serial without having to run pytest twice?

There's this post on Stackoverflow which might help:

You can use --dist=loadscope to group all the tests in the same test class. Here is the doc from pytest-xdist on pypi

By default, the -n option will send pending tests to any worker that is available, without any guaranteed order, but you can control this with these options:

--dist=loadscope: tests will be grouped by module for test functions and by class for test methods, then each group will be sent to an available worker, guaranteeing that all tests in a group run in the same process. This can be useful if you have expensive module-level or class-level fixtures. Currently the groupings can’t be customized, with grouping by class takes priority over grouping by module. This feature was added in version 1.19.

--dist=loadfile: tests will be grouped by file name, and then will be sent to an available worker, guaranteeing that all tests in a group run in the same worker. This feature was added in version 1.21.

@ssbarnea

@ssbarnea I struggled with this as well and couldn't find a solution anywhere. It's surprising that very few people have encountered this. I ended up implementing a serial fixture with a session-scoped file lock fixture using the filelock package, and it seems to work well. You can add this to your conftest.py:

import contextlib
import os

import filelock


@pytest.fixture(scope='session')
def lock(tmp_path_factory):
    base_temp = tmp_path_factory.getbasetemp()
    lock_file = base_temp.parent / 'serial.lock'
    yield filelock.FileLock(lock_file=str(lock_file))
    with contextlib.suppress(OSError):
        os.remove(path=lock_file)


@pytest.fixture()
def serial(lock):
    with lock.acquire(poll_intervall=0.1):
        yield

Then inject the serial fixture in any test that requires serial execution. All tests that use the serial fixture are executed serially while any tests that do not use the fixture are executed in parallel.

@ssbarnea

@ssbarnea
Then inject the serial fixture in any test that requires serial execution. All tests that use the serial fixture are executed serially while any tests that do not use the fixture are executed in parallel.

Unfortunately that is not correct. The tests with the serial will be serialized in relation to other tests with the same fixture. Tests without any fixture will be executed in parallel, even at the same time that a serialized test is running.

I need to find a way to have a global lock for my serial tests.

@ssbarnea

@ssbarnea
Then inject the serial fixture in any test that requires serial execution. All tests that use the serial fixture are executed serially while any tests that do not use the fixture are executed in parallel.

Unfortunately that is not correct. The tests with the serial will be serialized in relation to other tests with the same fixture. Tests without any fixture will be executed in parallel, even at the same time that a serialized test is running.

I need to find a way to have a global lock for my serial tests.

@danielfmo You are correct. That is exactly what will happen. This solution solves the original problem where you want to run some tests serially and others in parallel without having to run pytest twice.

The problem that I had was that there were quite a few functional tests in my test suite that had race conditions when being executed in parallel. It didn't matter to me in what order they were executed as long as they weren't executed at the same time. I was fine running the serial tests in parallel with other tests that did not create race conditions.

If your parallel tests cannot run at the same time as the serial tests, then you could have other dependency issues in your suite that my solution certainly would not fix. I'm not quite sure what problem that you are trying to solve?

@christopheraranda

What @christopheraranda describes is almost identical with my use case at https://github.com/ansible-community/molecule/tree/master/molecule/test -- where I want to run all unit tests in parallel and all functional tests in serials.

They happen to be inside their own modules and I am looking for a solution at module level, so I would not have to annotate tons of tests.

If we can find a solution and document it everyone will benefit as it is very common to have a set of tests that cannot run in parallel. How we select them it could be at module level, file level, class level or individual test functions, ideally being able to mention an optional lock name (for those that may have multiple locks).

@christopheraranda

What @christopheraranda describes is almost identical with my use case at https://github.com/ansible-community/molecule/tree/master/molecule/test -- where I want to run all unit tests in parallel and all functional tests in serials.

They happen to be inside their own modules and I am looking for a solution at module level, so I would not have to annotate tons of tests.

If we can find a solution and document it everyone will benefit as it is very common to have a set of tests that cannot run in parallel. How we select them it could be at module level, file level, class level or individual test functions, ideally being able to mention an optional lock name (for those that may have multiple locks).

Did we got solution for this ?
I have same challenge, few tests want to run parallely and few in serial(when no other tests are run parallely) single trigger of pytest.

Hi,

I think --dist=loadscope and group the tests by class already solves this? There's also --dist=loadfile can be used for whole modules (all tests in a module are sent to the same worker).

@nicoddemus

@nicoddemus How about making the serial fixture part of xdist and document it, so anyone could easier reuse?

I do think that the need for serial with xdist is far too popular for not providing support for it, without asking people to DIY their own implementation, which would be more likely to break in various cases.

Add serial marker for tests

@ssbarnea

Hi @ssbarnea,

Unless @RonnyPfannschmidt opposes, I'm OK to add that fixture if:

We document as experimental
It uses an optional filelock dependency
We properly document this limitations: this fixture as proposed ensures the tests using the fixture will not run concurrently in separate workers. It will not force all tests which use the fixture to execute in the same worker (which is a different use case).

While I understand this might not be ideal, we have to face the reality that I don't see this feature being implemented anytime soon, so providing a fixture which works in 90% of the cases out of the box is good for our users.

It's important to note that the file lock solution is limited to single machines

It may be helpful to refractor execnet to help this use case more

It may be helpful to refractor execnet to help this use case more

I agree, but I don't see that happening anytime soon. Just commenting on that to make it clear that we should not block the proposal due to this, and it can be made to work with distributed machines later if someone volunteers to refactor execnet.

Let's first and foremost agree that adding the partial solution is acceptable and should be done sooner rather than later

Let's first and foremost agree that adding the partial solution is acceptable and should be done sooner rather than later

Yeah I agree that we need to agree on that hehehe. I agree 👍

We also have a solution for the other use case I mentioned:

#385 (comment)

@ssbarnea

Hi @ssbarnea,

Unless @RonnyPfannschmidt opposes, I'm OK to add that fixture if:

We document as experimental

It uses an optional filelock dependency

We properly document this limitations: this fixture as proposed ensures the tests using the fixture will not run concurrently in separate workers. It will not force all tests which use the fixture to execute in the same worker (which is a different use case).

While I understand this might not be ideal, we have to face the reality that I don't see this feature being implemented anytime soon, so providing a fixture which works in 90% of the cases out of the box is good for our users.

There have been a few changes to the filelock api which may require changes to the example that I provided. Also, the example implements a *nix file lock and may not work for tests being executed on Windows. Additional logic would need to be implemented in a fixture to consider the platform on which tests are being executed.

@ssbarnea

@ssbarnea I struggled with this as well and couldn't find a solution anywhere. It's surprising that very few people have encountered this. I ended up implementing a serial fixture with a session-scoped file lock fixture using the filelock package, and it seems to work well. You can add this to your conftest.py:
import contextlib
import os

import filelock


@pytest.fixture(scope='session')
def lock(tmp_path_factory):
    base_temp = tmp_path_factory.getbasetemp()
    lock_file = base_temp.parent / 'serial.lock'
    yield filelock.FileLock(lock_file=str(lock_file))
    with contextlib.suppress(OSError):
        os.remove(path=lock_file)


@pytest.fixture()
def serial(lock):
    with lock.acquire(poll_intervall=0.1):
        yield
Then inject the serial fixture in any test that requires serial execution. All tests that use the serial fixture are executed serially while any tests that do not use the fixture are executed in parallel.

Ouch, Python 3.11:

>           warnings.warn(msg, DeprecationWarning, stacklevel=2)
E           DeprecationWarning: use poll_interval instead of poll_intervall

Causes pytest to fail because of warn == error.

Hi,

I think --dist=loadscope and group the tests by class already solves this? There's also --dist=loadfile can be used for whole modules (all tests in a module are sent to the same worker).

Using --dist=loadfile solved my issue as a subset of my tests that require serial running were accordingly collected in files. Thanks!

@ssbarnea

@ssbarnea I struggled with this as well and couldn't find a solution anywhere. It's surprising that very few people have encountered this. I ended up implementing a serial fixture with a session-scoped file lock fixture using the filelock package, and it seems to work well. You can add this to your conftest.py:
import contextlib
import os

import filelock


@pytest.fixture(scope='session')
def lock(tmp_path_factory):
    base_temp = tmp_path_factory.getbasetemp()
    lock_file = base_temp.parent / 'serial.lock'
    yield filelock.FileLock(lock_file=str(lock_file))
    with contextlib.suppress(OSError):
        os.remove(path=lock_file)


@pytest.fixture()
def serial(lock):
    with lock.acquire(poll_intervall=0.1):
        yield
Then inject the serial fixture in any test that requires serial execution. All tests that use the serial fixture are executed serially while any tests that do not use the fixture are executed in parallel.
Ouch, Python 3.11:
>           warnings.warn(msg, DeprecationWarning, stacklevel=2)
E           DeprecationWarning: use poll_interval instead of poll_intervall
Causes pytest to fail because of warn == error.

This is not a Python 3.11 issue but an update to the filelock API that was introduced in version 3.4.0 that was released on November 16, 2021 (which was after my comment on April 22, 2020). You should be able to just update the poll_intervall keyword argument in the fixture function and the warning should go away.

@pytest.fixture()
def serial(lock):
    with lock.acquire(poll_interval=0.1):
        yield

Does this solution lead to unnecessarily stalling other processes that have to wait for a lock? Example: 4 workers in total and if you're unlucky it happens that all 4 of them will want to execute serial tests now, this means that 3 workers will be starving while only one worker actually works, then 2, then 1, etc. This is contrary to an approach where serial tests could be "smartly" scheduled to be executed in a way that they "intersect" as little as possible, while parallel tests keep on running as much as possible.

Does this solution lead to unnecessarily stalling other processes that have to wait for a lock?

Indeed it does.

Navigation Menu

Repository navigation

Enforce serial execution of related tests #84

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Enforce serial execution of related tests #84

Description

Activity

nicoddemus commented on Aug 18, 2016

evbo commented on Jul 5, 2017

nicoddemus commented on Jul 5, 2017

ssbarnea commented on Oct 4, 2019

larsblumberg commented on Nov 29, 2019

christopheraranda commented on Apr 22, 2020

danielfmo commented on Jul 31, 2020

christopheraranda commented on Jul 31, 2020

ssbarnea commented on Jul 31, 2020

bkashigar commented on Nov 26, 2021

nicoddemus commented on Nov 26, 2021

ssbarnea commented on Aug 8, 2022

nicoddemus commented on Aug 10, 2022

RonnyPfannschmidt commented on Aug 10, 2022

nicoddemus commented on Aug 10, 2022

RonnyPfannschmidt commented on Aug 10, 2022

nicoddemus commented on Aug 10, 2022

nicoddemus commented on Aug 10, 2022

christopheraranda commented on Aug 10, 2022

egberts commented on Jul 14, 2024

Zyantist commented on Oct 9, 2024

christopheraranda commented on Oct 15, 2024

Alexander-Shukaev commented on Apr 29, 2025

nicoddemus commented on May 6, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions