Repository navigation
Pytest 8.0.2 may collect tests multiple times under Windows #12039
Description
Activity
Hi @mrbean-bremen, thanks for the report!
Probably it is related to the fix for short paths... perhaps it is collecting the soft-links that
tmpdircreates? 🤔Lines 206 to 225 in 98b008f
- added a commit that references this issue
on Feb 28, 2024 @nicoddemus - could be... I have unsucessfully tried to reproduce this locally (using junctions), I may have another look later, maybe tomorrow.
- added a commit that references this issue
on Feb 29, 2024 I can confirm that removing the fix for short paths (e.g. the line that uses
os.path.samefilefor the match) fixes the problem in the CI. Still no luck to reproduce this locally.@nicoddemus: I think the test using
samefileis too broad, as it also finds symlinks and junctions, which we don't wont.
I tested to adapt the check to filter out links and junctions to regular directories, which fixes the problem in the CI:if sys.platform == "win32" and not is_match: # In case the file paths do not match, # account for short-paths on Windows (#11895). same_file = os.path.samefile(node.path, matchparts[0]) # we don't want to find links or junctions, so we at least # exclude links/junctions to regular directories is_match = ( same_file and os.path.islink(node.path) == os.path.islink(matchparts[0]) and _is_junction(node.path) == _is_junction(matchparts[0]) )
(
is_junctionis basically a copy of the Python 3.12 implementation ofos.path.junction).
This does not not exclude links to other links to the same directory, but I would say that these are very unlikely cases.
Directly checking the short paths usingctypesandGetShortPath(which might be more correct) is possible, but ugly...Please let me know if this makes sense...
Hmm it does make sense.
Just the
islinkcheck is enough to fix your CI? If so, I think we can easily just add this extra check. Junctions are more rare (I believe), so I think we can leave them out of this fix for now, as it requires extra steps (or we can just check for them in 3.12 at least).I wouldn't say that junctions are rare (I use them all the time under Windows, as you need admin rights to create symlinks there), but maybe they are indeed rare for use cases with pytest. I will check tomorrow, if the check for links is sufficient for the CI.
Ok, I checked. You were right - the problem is indeed solved with the
islinkcheck alone.Thanks @mrbean-bremen.
However I'm having trouble reproducing this in a test, this is what I came up with:
import pytest def test_double_collect_symlinks(tmpdir) -> None: test_path = tmpdir.join("test_double_collect_symlinks.py") test_path.write("def test(): pass") result = pytest.main([str(tmpdir)]) assert result == 0
However this executes just 1 test in several pytest versions I tried:
7.4.4,8.0.1,8.0.2,main:λ pytest .tmp\test_double_collect.py -s ================================================= test session starts ================================================= platform win32 -- Python 3.10.10, pytest-8.0.2, pluggy-1.4.0 rootdir: e:\projects\pytest\.tmp configfile: pytest.ini plugins: hypothesis-6.84.0, replay-1.4.0 collected 1 item .tmp\test_double_collect.py ================================================= test session starts ================================================= platform win32 -- Python 3.10.10, pytest-8.0.2, pluggy-1.4.0 rootdir: E:\.tmp\pytest-of-Pichau\pytest-375\test_double_collect_symlinks0 plugins: hypothesis-6.84.0, replay-1.4.0 collected 1 item ..\..\.tmp\pytest-of-Pichau\pytest-375\test_double_collect_symlinks0\test_double_collect_symlinks.py . [100%] ================================================== 1 passed in 0.01s ================================================== . ================================================== 1 passed in 0.09s ==================================================Can you try to reproduce this problem outside
pytest-order's test suite?Looks like the problem happens only withpytester, e.g. if I run tests usingpytester.
EDIT:pytesteris not involved, actually.I have set up a minimal repository to show this.
The test is trivial:import pytest def test_collecting_symlinks(tmpdir): testname = str(tmpdir.join("first_test.py")) with open(testname, "w") as fi: fi.write( """ def test_1(): pass """ ) args = ["-v", str(tmpdir)] ret = pytest.main(args, []) assert ret == 0
and under Linux the output in the CI is as expected:
collecting ... collected 1 item ../../../../../tmp/pytest-of-runner/pytest-0/test_collecting_symlinks0/first_test.py::test_1 PASSED [100%]while under Windows I get:
collecting ... collected 8 items ::::Documents and Settings::runneradmin::AppData::Local::Temp::pytest-of-runneradmin::pytest-0::test_collecting_symlinks0::first_test.py::test_1 PASSED [ 12%] ::::Documents and Settings::runneradmin::AppData::Local::Temp::pytest-of-runneradmin::pytest-0::test_collecting_symlinkscurrent::first_test.py::test_1 PASSED [ 25%] ::::Documents and Settings::runneradmin::AppData::Local::Temp::pytest-of-runneradmin::pytest-current::test_collecting_symlinks0::first_test.py::test_1 PASSED [ 37%] ::::Documents and Settings::runneradmin::AppData::Local::Temp::pytest-of-runneradmin::pytest-current::test_collecting_symlinkscurrent::first_test.py::test_1 PASSED [ 50%] first_test.py::test_1 PASSED [ 62%] ::::Users::runneradmin::AppData::Local::Temp::pytest-of-runneradmin::pytest-0::test_collecting_symlinkscurrent::first_test.py::test_1 PASSED [ 75%] ::::Users::runneradmin::AppData::Local::Temp::pytest-of-runneradmin::pytest-current::test_collecting_symlinks0::first_test.py::test_1 PASSED [ 87%] ::::Users::runneradmin::AppData::Local::Temp::pytest-of-runneradmin::pytest-current::test_collecting_symlinkscurrent::first_test.py::test_1 PASSED [100%]which is quite a lot of duplication...
Actually I wrote a test very similar to yours, and it passed locally, but failed in the CI (though with 2 tests passed, not 8, in this case):
@pytest.mark.skipif(not sys.platform.startswith("win"), reason="Windows only") def test_collect_symlinks_windows(pytester: Pytester, tmpdir) -> None: """Regression test for #12039: Tests collected multiple times under Windows.""" test_file = Path(tmpdir) / "symlink_collection_test.py" test_file.write_text("def test(): pass", encoding="UTF-8") result = pytester.runpytest(tmpdir) > assert result.parseoutcomes() == {"passed": 1} E AssertionError: assert {'passed': 2} == {'passed': 1} E E Differing items: E {'passed': 2} != {'passed': 1} E Use -v to get more diffI made a PR for this.
- added a commit that references this issue
on Mar 3, 2024
I'm not quite sure if this is a bug, or a problem with the test or GitHub Actions, but since version 8.0.2 a test in
pytest-orderfails in the CI under Windows, which worked fine in 8.0.1 and previous versions. Linux and macOS tests are not affected.Pytest run environment in the CI:
The same happens for Python 3.8 - 3.12.
The actual test should collect 6 items (3 tests per test module, in 2 modules) that are created in the directory provided by the
tempdirfixture,but collects 24 tests in the CI instead. Locally, it works as expected.
Checking the failed tests revealed that the tests have been collected from different locations. The tempdir had been at:
C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-9\\test_xdist_ordering0(for a specific run), but the same tests have been discovered with a test ID with just the qualified test name, and additionally with the test IDs:
::::Users::runneradmin::AppData::Local::Temp::pytest-of-runneradmin::pytest-9:::xxxand::::Documents and Settings::runneradmin::AppData::Local::Temp::pytest-of-runneradmin::pytest-9::xxx,which are most likely aliases.
I haven't dug into this yet, but I suspect it may have something to do with the fix for the problem with short paths.