Repository navigation
Crash on OS X when accessing keyring from multiple processes #281
Description
OS: Mac OS Sierra, 10.12.5
Version: 10.3
Description: Crash happens (with -11 exit code) when trying to ask for a password both from the main process and from a subprocess. See the repro case below.
After some digging, it seems that the issue happens in the SecKeychainFindGenericPassword() call of backends/_OS_X_API.py: find_generic_password().
import keyring
from multiprocessing import Process
from_address = '[email protected]'
def my_subprocess():
keyring.get_password('myapp', from_address)
print('ok')
# comment out the line below to get things work(around)
keyring.get_password('myapp', from_address)
proc1 = Process(target=my_subprocess)
proc1.start()
proc1.join()
print(proc1.exitcode)
proc2 = Process(target=my_subprocess)
proc2.start()
proc2.join()
print(proc2.exitcode)
# expected output:
# ok
# 0
# ok
# 0
#
# current output:
# -11
# -11
Activity
renatahodovan commentedon Jun 20, 2017
The crash is also valid on the latest master (3ed3cae).
jaraco commentedon Jun 25, 2017
Nice work characterizing the issue and providing a repro.
I'm not sure what the next steps are to investigate, so I welcome you or others to continue to pursue a fix. Or I'll surely look into it more as the time permits.
Add new test capturing failure. Ref #281.
jaraco commentedon Dec 6, 2018
Based on the reproducer you supplied, I've added a test to a branch that replicates the reported failure. I was able to distill the reproducer even more to show that only one subprocess is necessary. I'm fairly sure that what's going on here is multiprocessing is using
forkto create the new processes... and something about the way ctypes or keyring's use of ctypes doesn't work well under a forking model.jaraco commentedon Dec 6, 2018
This issue/answer seems related. Probably keyring needs to have its OS_X_API module reinitialized in the subprocesses.
jaraco commentedon Dec 6, 2018
When the process crashes, I see this in the "Problem Report" that pops up due to the segmentation fault:
jaraco commentedon Dec 6, 2018
I tried adding this patch, but it did not help:
jaraco commentedon Dec 6, 2018
I thought maybe forcing all the modules including
_OS_X_APIto reset in the subprocess might workaround the issue, so I applied this patch:But the error still occurs. I'm now inclined to think the issue may be with the API itself... that it may not allow invocation by a process and its forked child.
I'm out of time to work on this, but I would recommend someone try to put together a reproducer that doesn't use keyring, but simply calls the ctypes calls as found in _OS_X_API.py. If the segfault can be recreated that way, it might implicate fundamental issues in the API. If not, that suggests there's something that keyring could do differently to address the condition.
akosthekiss commentedon Dec 12, 2018
I've been trying to come up with a keyring-free test case, this is how far I got (nokeyring.py):
Running this gives:
Commenting out the call in main results in:
I hope this helps.
jaraco commentedon Dec 13, 2018
Yes, that definitely helps. You've reproduced the issue without keyring. Next will be to figure out if the issue is in Python/ctypes or inherent in the API.
akosthekiss commentedon Dec 13, 2018
It seems that it is an API problem :/. Here is an Objective-C(-ish?) repro (nokeyring.m).
Build and run:
Build and run with call in main process commented out:
Not sure how to move forward if the native code/API behaves like this.
Workaround for keyring crash on OS X.
bhedavivek commentedon Sep 9, 2019
I think I am facing a similar issue when keyring tries to store a generic password in a subprocess.
fails for me particularly in
keyring.backend._OS_X_APIline 235. Keyring version 13.2.1jaraco commentedon Sep 11, 2019
I've posted this question in the forums, hoping they can shed some light on the issue.
jaraco commentedon Sep 11, 2019
Aha. Ned Deily pointed me to the macOS man page for fork which states clearly:
So that pretty definitively settles it. You can't use keychain after fork.
I do have some good news, however. With Python 3.8, the multiprocessing module prefers exec/spawn for multiprocess.Process, meaning the issue no longer manifest.
I just confirmed that this script executes without error on Python 3.8:
Skip tests in environments known to fail. Ref #281.