Repository navigation
Pytest 8.2.0 breaks Jenkins unittesting [Python 3.10] #12266
Description
Activity
we also have other repos that use pytest and are ok with 8.2.0 that contain DeprecationWarning
But, I cant see any PytestCollectionWarning in them.
Was there a change in 8.2.0 that makes PytestCollectionWarning make the command exit code != 0?
A warning would only cause an exit code != 0 if warnings are turned into errors (with
filterwarnings=error).However the return code
-1073741819is not issued by pytest, seems like something else is causing the interpreter to crash... 🤔-1073741819in Windows is "access violation" AKA "segmentation fault". pytest itself, written in pure Python, cannot trigger such a thing. You need to run the program under a (C) debugger to find out why it happens. You can also try changingpytest ...topython -X dev -m pytest ...to enablesfaulthandlerfor the entire duration and might catch the issue if it's coming from Python.Im going to run some experiments. Basically keep disabling tests until it stops crashing. Then see what is different with that particular test. We have many other repos using the same jenkins nodes with same python that were running fine with latest pytest
Script and instruction to reproduce below.
""" Python 3.10.9 - Windows X64 pip install wxpython==4.2.1 pytest==8.2.0 --prefer-binary pytest test_abc.py Fatal Python error: PyThreadState_Get: the function must be called with the GIL held, but the GIL is released (the current Python thread state is NULL) Python runtime state: finalizing (tstate=0x0000020576a43c00) running unittest directly OK running with pytest 8.1.0 OK running with pytest 8.2.0 + Python 3.12.3 OK """ import wx import unittest class Test_ABC(unittest.TestCase): def setUp(self): self.app = wx.App() wx.CallAfter(lambda: print("hello")) def test_a(self): pass # need two tests as the issue is triggered when setUp called the 2nd time def test_b(self): pass if __name__ == "__main__": unittest.main()we have fixed this by creating the app in class setup instead
@classmethod def setUpClass(cls): cls.app = wx.App()However, thought you may still be interested to investigate why 8.2.0 started making it crash :)
@bluetech did you see the above?
Not sure if it's related, but just to mention in case it is and is useful to know. Our GitHub Actions tests for
dopedstarted crashing in certain cases (ubuntu-latest,python=3.10) upon updating topytest==8.2.0, whereas fixing it to the previous8.1.2had all tests running fine, before and after.
The output from the GitHub CI wasn't particularly informative, but seemed to be due to some memory/resource overload issue with our plotting tests, which seems like it could be related to @matthuisman's issues (moving the app initialisation tosetUpClassto reduce memory/resource demand?).For the section of our tests that runs plotting tests with:
pytest --mpl -m "mpl_image_compare" tests # all plotting teststhe log output was:
2024-05-11T08:23:16.2962832Z ============================= test session starts ============================== 2024-05-11T08:23:16.2969685Z platform linux -- Python 3.10.14, pytest-8.2.0, pluggy-1.5.0 2024-05-11T08:23:16.2979464Z Matplotlib: 3.8.4 2024-05-11T08:23:16.2982774Z Freetype: 2.6.1 2024-05-11T08:23:16.2985341Z rootdir: /home/runner/work/doped/doped 2024-05-11T08:23:16.2988990Z configfile: pyproject.toml 2024-05-11T08:23:16.2989605Z plugins: mpl-0.17.0 2024-05-11T08:23:16.2992918Z collected 197 items / 133 deselected / 64 selected 2024-05-11T08:23:16.2993313Z 2024-05-11T08:42:54.3094183Z tests/test_analysis.py ...................... [ 34%] 2024-05-11T08:46:03.0040480Z tests/test_corrections.py ....... [ 45%] 2024-05-11T08:46:07.4215628Z tests/test_displacements.py ... [ 50%] 2024-05-11T09:02:25.4387407Z tests/test_plotting.py ............................. [ 95%] 2024-05-11T09:04:29.0892314Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled. 2024-05-11T09:04:30.7922723Z tests/test_thermodynamics.py 2024-05-11T09:04:30.8393348Z ##[error]The operation was canceled.Full logs of failed and passed runs attached if useful
passed_GH_Actions_log_pytest_8.1.1.txt
failed_GH_Actions_log_pytest_8.2.0.txt@matthuisman Thanks for providing a reproduction. I bisected the segfault to commit 0dc0360.
Backtrace:
Thread 1 "python" received signal SIGSEGV, Segmentation fault. 0x00007ffff799d901 in _PyInterpreterState_GET () at ./Include/internal/pycore_pystate.h:133 Downloading source file /usr/src/debug/python/Python-3.12.3/./Include/internal/pycore_pystate.h 133 return tstate->interp; #0 0x00007ffff799d901 in _PyInterpreterState_GET () at ./Include/internal/pycore_pystate.h:133 #1 handle_func_event (new_value=<optimized out>, func=<optimized out>, event=<optimized out>) at Objects/funcobject.c:65 #2 func_dealloc (op=0x7fffef9d5d00) at Objects/funcobject.c:844 #3 0x00007ffff54411d6 in Py_DECREF (op=<optimized out>) at /usr/include/python3.12/object.h:706 #4 wxPyCallback::~wxPyCallback (this=0x555555da1a10, this=<optimized out>) at ../../../../sip/cpp/sip_corewxEvtHandler.cpp:43 #5 0x00007ffff544124e in wxPyCallback::~wxPyCallback (this=0x555555da1a10, this=<optimized out>) at ../../../../sip/cpp/sip_corewxEvtHandler.cpp:44 #6 0x00007ffff4378fb6 in wxEvtHandler::~wxEvtHandler (this=0x555555628100, this=<optimized out>) at /usr/src/debug/wxwidgets/wxWidgets/src/common/event.cpp:1242 #7 0x00007ffff56ea79d in sipwxPyApp::~sipwxPyApp (this=0x555555628100, this=<optimized out>) at ../../../../sip/cpp/sip_corewxPyApp.cpp:457 #8 release_wxPyApp (sipState=<optimized out>, sipCppV=0x555555628100) at ../../../../sip/cpp/sip_corewxPyApp.cpp:2557 #9 dealloc_wxPyApp (sipSelf=<optimized out>) at ../../../../sip/cpp/sip_corewxPyApp.cpp:2580 #10 dealloc_wxPyApp (sipSelf=<optimized out>) at ../../../../sip/cpp/sip_corewxPyApp.cpp:2573 #11 0x00007ffff17f7394 in forgetObject (sw=sw@entry=0x7fffef9c0b90) at ../../../../sip/siplib/siplib.c:11424 #12 0x00007ffff17f7487 in sipWrapper_dealloc (self=0x7fffef9c0b90) at ../../../../sip/siplib/siplib.c:11043 #13 0x00007ffff79d6aa6 in subtype_dealloc (self=0x7fffef9c0b90) at Objects/typeobject.c:2051 #14 0x00007ffff7982571 in _Py_Dealloc (op=<optimized out>) at Objects/object.c:2625 #15 Py_DECREF (op=<optimized out>) at ./Include/object.h:705 #16 Py_XDECREF (op=<optimized out>) at ./Include/object.h:798 #17 free_keys_object (keys=0x7ffff6472290, interp=0x7ffff7dc51c8 <_PyRuntime+76392>) at Objects/dictobject.c:673 #18 dictkeys_decref (dk=0x7ffff6472290, interp=0x7ffff7dc51c8 <_PyRuntime+76392>) at Objects/dictobject.c:333 #19 dict_dealloc (mp=0x7ffff6404e40) at Objects/dictobject.c:2374 #20 0x00007ffff79d6c5d in _Py_Dealloc (op=0x7ffff6404e40) at Objects/object.c:2625 #21 Py_DECREF (op=0x7ffff6404e40) at ./Include/object.h:705 #22 Py_XDECREF (op=0x7ffff6404e40) at ./Include/object.h:798 #23 subtype_dealloc (self=0x7ffff6fbd250) at Objects/typeobject.c:2020 #24 0x00007ffff798266f in dict_dealloc (mp=0x7fffef9da700) at Objects/dictobject.c:2367 #25 0x00007ffff7a75d14 in subtype_clear (self=0x7ffff6482740) at Objects/typeobject.c:1863 #26 0x00007ffff798fc7a in delete_garbage (old=0x7ffff7dc5280 <_PyRuntime+76576>, collectable=0x7fffffffdb80, gcstate=0x7ffff7dc5238 <_PyRuntime+76504>, tstate=0x7ffff7e22ae8 <_PyRuntime+459656>) at Modules/gcmodule.c:1029 #27 gc_collect_main (tstate=0x7ffff7e22ae8 <_PyRuntime+459656>, generation=generation@entry=2, n_collected=n_collected@entry=0x0, n_uncollectable=n_uncollectable@entry=0x0, nofail=nofail@entry=1) at Modules/gcmodule.c:1303 #28 0x00007ffff7a759fc in _PyGC_CollectNoFail (tstate=<optimized out>) at Modules/gcmodule.c:2135 #29 0x00007ffff7a5e406 in Py_FinalizeEx () at Python/pylifecycle.c:1889 #30 0x00007ffff7a6ccf2 in Py_RunMain () at Modules/main.c:711 #31 0x00007ffff7a28fab in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:763 #32 0x00007ffff7639c88 in __libc_start_call_main (main=main@entry=0x555555555120 <main>, argc=argc@entry=4, argv=argv@entry=0x7fffffffdf88) at ../sysdeps/nptl/libc_start_call_main.h:58 #33 0x00007ffff7639d4c in __libc_start_main_impl (main=0x555555555120 <main>, argc=4, argv=0x7fffffffdf88, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdf78) at ../csu/libc-start.c:360 #34 0x0000555555555045 in _start ()@matthuisman I will try to see what in pytest has started causing this, however I do not consider the issue to be caused by pytest, because the test does not isolate the
CallAfterproperly. I don't know anything about wxpython, but you need to somehow terminate theappon test teardown, e.g. if you add this then pytest doesn't segfault:def tearDown(self): del self.app
@matthuisman I will try to see what in pytest has started causing this, however I do not consider the issue to be caused by pytest, because the test does not isolate the
CallAfterproperlyAs per comment here: #12266 (comment)
"we have fixed this by creating the app in class setup instead.
However, thought you may still be interested to investigate why 8.2.0 started making it crash"Its upto you guys if you want to investigate why it used to not crash but now does.
Memory layout changes are a common cause for surfacing memory errors with 3rd parties
As pytest itself is pure python, I don't see a need to follow up
I consider this a bug in wx-python
So the problem in this issue is that the test doesn't tear down the resources it sets up. So I will close this issue again.
However, I found the reason this (and couple other recent reports) started coming up in pytest 8.2. I will open a separate issue for this as it is a sort of regression that we should fix (hopefully for 8.2.2).
Thats great :)
i was hoping even know our test was being silly, that the issue it brought up may be of some help
[some text/paths removed below]
Note some warnings and then the "script returned exit code -1073741819"
I can't reproduce locally - i always get exit code 0
If I go back to 8.1.2 - jenkins works fine without any other changes.
I get the same warnings etc, but the exit code is OK
Any help / ideas to try would be greatly appreciated :)
Command that runs the below:
pytest --splits 1 --group 1 --splitting-algorithm least_duration --store-durations --clean-durations --durations-path test_durations.json --junitxml=junit.xml