Skip to content

pytest with multiprocessing & "--import-mode=importlib" EOFError #13690

Description

Issue Type: Bug

Pytest raises:


    def test_funcInTestFile():
        p = multiprocessing.Process(target=sleepSomeTimeInTestFile)
>       p.start()

test_example.py:24: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
W:\Python38-64bit\lib\multiprocessing\process.py:121: in start
    self._popen = self._Popen(self)
W:\Python38-64bit\lib\multiprocessing\context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
W:\Python38-64bit\lib\multiprocessing\context.py:327: in _Popen
    return Popen(process_obj)
W:\Python38-64bit\lib\multiprocessing\popen_spawn_win32.py:93: in __init__
    reduction.dump(process_obj, to_child)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <Process name='Process-1' parent=19324 initial>
file = <_io.BufferedWriter name=5>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       _pickle.PicklingError: Can't pickle <function sleepSomeTimeInTestFile at 0x0000019EF622A3A0>: it's not the same object as test_example.sleepSomeTimeInTestFile

W:\Python38-64bit\lib\multiprocessing\reduction.py:60: PicklingError
  File "W:\Python38-64bit\lib\multiprocessing\spawn.py", line 116, in spawn_main
- generated xml file: C:\Users\DOMENJ~1.ISY\AppData\Local\Temp\tmp-1116815mH4iv3Rg16.xml -
    exitcode = _main(fd, parent_sentinel)

... error when:

  • "--import-mode=importlib" arg is used
  • multiprocessing Process() target=<function> function is defined in the same (test) file

Without "--import-mode=importlib" test works (but not when test files have the same names.

pytestImporlibTest.zip

Extension version: 2020.8.105045
VS Code version: Code 1.48.2 (a0479759d6e9ea56afa657e454193f72aef85bd0, 2020-08-25T10:13:11.295Z)
OS version: Windows_NT x64 10.0.19041

System Info
Item Value
CPUs Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz (4 x 3193)
GPU Status 2d_canvas: enabled
flash_3d: enabled
flash_stage3d: enabled
flash_stage3d_baseline: enabled
gpu_compositing: enabled
multiple_raster_threads: enabled_on
oop_rasterization: disabled_off
protected_video_decode: unavailable_off
rasterization: enabled
skia_renderer: disabled_off_ok
video_decode: enabled
viz_display_compositor: enabled_on
viz_hit_test_surface_layer: disabled_off_ok
webgl: enabled
webgl2: enabled
Load (avg) undefined
Memory (System) 15.89GB (6.98GB free)
Process Argv W:\AsystLib\isys\python\VSCodeDevWorkspace_isys.code-workspace
Screen Reader no
VM 0%

Activity

  1. ghost added
    triage-neededNeeds assignment to the proper sub-team
    on Aug 31, 2020
  2. nicoddemus commented on Aug 31, 2020

    @nicoddemus

    I assume multiprocessing either tries to:

    1. Dump only module name + function name, and in the other process, it uses that information to obtain the function object and call it.

    2. If the function object doesn't have a module attribute (which is the case with --import-mode=importlib), then it tries to pickle the actual function object, but then fails.

    (I don't have time to look up the code, so I'm just assuming the above is true)

    If that's the case, it is not something that is possible to fix: the importlib module's purpose is to not pollute the sys.modules dict with the test modules being imported.

  3. karthiknadig commented on Aug 31, 2020

    @karthiknadig
    Member

    Bruno Oliveira (@nicoddemus) Thank you for the response.

    Domen Jurkovic (@schperplata) The python extension is not involved in this at all. If you do pickle.dumps on sleepSomeTimeInTestFile it does not match what it expects in the test process. Actual __main__.sleepSomeTimeInTestFile, pickle expects test_example.sleepSomeTimeInTestFile.

    Closing this since it occurs without the python extension being involved.

  4. ghost removed
    triage-neededNeeds assignment to the proper sub-team
    on Aug 31, 2020
  5. locked as resolved and limited conversation to collaborators on Oct 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions