Skip to content

OSError: [Errno 39] Directory not empty when doing cache.mkdir() from session fixture + pytest-xdist #12381

Description

@amezin
  • a detailed description of the bug or problem you are having
  • output of pip list from the virtual environment you are using
  • pytest and operating system versions
  • minimal example if possible

cache.mkdir() in a session-level fixture, in combination with xdist, sometimes results in OSError: [Errno 39] Directory not empty when pytest tries to rename the cache directory.

This is a regression in 8.2.0, most likely introduced by #12168 / #12167

On 8.1.x cache.mkdir() works fine in the same case.

In conftest.py:

@pytest.fixture(scope='session')
def my_fixture(request):
    return request.config.cache.mkdir('testdir')

In test_tests.py:

@pytest.mark.parametrize('n', list((n,) for n in range(42)))
def test_test(my_fixture, n):
    pass

Running with pytest -n auto results in:

self = PosixPath('/home/runner/work/pytest-cache-race/pytest-cache-race/.tox/python/pytest-cache-files-8ulls0nm')
target = PosixPath('/home/runner/work/pytest-cache-race/pytest-cache-race/.tox/python/.pytest_cache')

    def rename(self, target):
        """
        Rename this path to the target path.
    
        The target path may be absolute or relative. Relative paths are
        interpreted relative to the current working directory, *not* the
        directory of the Path object.
    
        Returns the new Path instance pointing to the target path.
        """
>       self._accessor.rename(self, target)
E       OSError: [Errno 39] Directory not empty: '/home/runner/work/pytest-cache-race/pytest-cache-race/.tox/python/pytest-cache-files-8ulls0nm' -> '/home/runner/work/pytest-cache-race/pytest-cache-race/.tox/python/.pytest_cache'

Operating System: Ubuntu 22.04.4 LTS
Python 3.10.12, pytest-8.2.1
Virtualenv content: exceptiongroup==1.2.1,execnet==2.1.1,iniconfig==2.0.0,packaging==24.0,pluggy==1.5.0,pytest==8.2.1,pytest-xdist==3.6.1,tomli==2.0.1

https://github.com/amezin/pytest-cache-race

Reproduced in GitHub Actions: https://github.com/amezin/pytest-cache-race/actions/runs/9261012859/job/25475719651

Activity

  1. bluetech commented on Jun 2, 2024

    @bluetech
    Member

    Hmm, so now we have a different race. I think we should be able to ignore this error, i.e. if it happens just continue (while removing the directory). This is because we can assume that the first worker correctly created the directory atomically.

  2. added
    type: bugproblem that needs to be addressed
    plugin: xdistrelated to the xdist external plugin
    plugin: cacherelated to the cache builtin plugin
    type: regressionindicates a problem that was introduced in a release which was working previously
    on Jun 2, 2024
  3. added a commit that references this issue on Jun 2, 2024
    17065cb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    plugin: cacherelated to the cache builtin pluginplugin: xdistrelated to the xdist external plugintype: bugproblem that needs to be addressedtype: regressionindicates a problem that was introduced in a release which was working previously

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions