Skip to content

Python crashes on macOS after fork with no exec #77906

Closed
Closed
Python crashes on macOS after fork with no exec#77906
Labels
3.8 (EOL)end of lifeOS-mactype-crashA hard crash of the interpreter, possibly with a core dump

Description

@kapilt
kapilt
mannequin
opened on Jun 1, 2018
BPO 33725
Nosy @ronaldoussoren, @ned-deily
PRs
  • bpo-33725: skip test_multiprocessing_fork on macOS #11043
  • [3.7] bpo-33725: skip test_multiprocessing_fork on macOS (GH-11043) #11044
  • [3.6] bpo-33725: skip test_multiprocessing_fork on macOS (GH-11043) #11045
  • bpo-33725: multiprocessing uses spawn by default on macOS #13603
  • [3.7] bpo-33725: multiprocessing uses spawn by default on macOS (GH-13603) #13626
  • bpo-33725, multiprocessing doc: rephase warning against fork on macOS #13841
  • [3.8] bpo-33725, multiprocessing doc: rephase warning against fork on macOS (GH-13841) #13849
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2020-05-29.18:09:22.220>
created_at = <Date 2018-06-01.00:53:06.418>
labels = ['OS-mac', '3.8', 'type-crash']
title = 'Python crashes on macOS after fork with no exec'
updated_at = <Date 2021-11-04.14:32:41.053>
user = 'https://github.com/kapilt'

    bugs.python.org fields:

    activity = <Date 2021-11-04.14:32:41.053>
actor = 'eryksun'
assignee = 'none'
closed = True
closed_date = <Date 2020-05-29.18:09:22.220>
closer = 'barry'
components = ['macOS']
creation = <Date 2018-06-01.00:53:06.418>
creator = 'kapilt'
dependencies = []
files = []
hgrepos = []
issue_num = 33725
keywords = ['patch']
message_count = 70.0
messages = ['318352', '318361', '318396', '318397', '318528', '318529', '318708', '329871', '329880', '329885', '329919', '329922', '329923', '329926', '329927', '329933', '329941', '331101', '331406', '331407', '331409', '331411', '331435', '331438', '331459', '331610', '331733', '331735', '337587', '337591', '337733', '338819', '338873', '341452', '341455', '341475', '342042', '342071', '342412', '343704', '343773', '343779', '343782', '343807', '343826', '343828', '343830', '343832', '343833', '343838', '343841', '343842', '343844', '343895', '343898', '344590', '344608', '344710', '344762', '344763', '345841', '365249', '365251', '365252', '365262', '365263', '365266', '365281', '370296', '370331']
nosy_count = 2.0
nosy_names = ['ronaldoussoren', 'ned.deily']
pr_nums = ['11043', '11044', '11045', '13603', '13626', '13841', '13849']
priority = 'critical'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue33725'
versions = ['Python 3.8']

    Activity

    1. kapilt commented on Jun 1, 2018

      @kapilt
      kapiltmannequin
      on Jun 1, 2018
      MannequinAuthor

      This issue seems to be reported a few times on various githubs projects. I've also reproduced using a brew install of python 2.7.15. I haven't been able to reproduce with python 3.6. Note this requires a framework build of python.

      Background on the underlying issue cause due to a change in high Sierra
      http://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html
      A ruby perspective on the same issue exhibiting for some apps
      https://blog.phusion.nl/2017/10/13/why-ruby-app-servers-break-on-macos-high-sierra-and-what-can-be-done-about-it/

      The work around seems to be setting an environment variable OBJC_DISABLE_INITIALIZE_FORK_SAFETY prior to executing python.

      Other reports

      https://bugs.python.org/issue30837
      ansible/ansible#32499
      MikeChongCan/scylla#22
      elastic/beats-tester#73
      jhaals/ansible-vault#60

    2. kapilt
      added
      OS-mac
      on Jun 1, 2018

    3. ronaldoussoren commented on Jun 1, 2018

      @ronaldoussoren
      ronaldoussoren
      on Jun 1, 2018
      Contributor

      A better solution is to avoid using fork mode for multiprocessing. The spawn and fork server modes should work fine.

      The underlying problem is that macOS system frameworks (basically anything higher level than libc) are not save wrt fork(2) and fixing that appears to have no priority at all at Apple.

    4. ned-deily commented on Jun 1, 2018

      @ned-deily
      ned-deily
      on Jun 1, 2018
      Member

      (As a side note, the macOS Pythons provided by python.org installers should not behave differently on macOS 10.13 High Sierra since none of them are built with a 10.13 SDK.)

    5. pitrou commented on Jun 1, 2018

      @pitrou
      pitrou
      on Jun 1, 2018
      Member

      I understand that Apple, with their limited resources, cannot spend expensive engineer manpower on improving POSIX support in macOS </snark>.

      In any case, I'm unsure this bug can be fixed at the Python level. If macOS APIs don't like fork(), they don't like fork(), point bar. As Ronald says, on 3.x you should use "forkserver" (for multiple reasons, not only this issue). On 2.7 you're stuck dealing with the issue by yourself.

    6. ronaldoussoren commented on Jun 3, 2018

      @ronaldoussoren
      ronaldoussoren
      on Jun 3, 2018
      Contributor

      Antoine, the issue is not necessarily related to POSIX compliance, AFAIK strictly POSIX compliant code should work just fine. The problem is in higher-level APIs (CoreFoundation, Foundation, AppKit, ...), and appears to be related to using multi-threading in those libraries without spending effort on pre/post fork handlers to ensure that new processes are in a sane state after fork(). In older macOS versions this could result in hard to debug issues, in newer versions APIs seem to guard against this by aborting when the detect that the pid changed.

      Anyways... I agree that we shouldn't try to work around this in CPython, there's bound to more problems that are hidden with the proposed workaround.

      ---

      <http://www.sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html\> describes what the environment variable does, and this "just" changes behavior of the ObjC runtime, and doesn't make using macOS system frameworks after a fork saver.

    7. ronaldoussoren commented on Jun 3, 2018

      @ronaldoussoren
      ronaldoussoren
      on Jun 3, 2018
      Contributor

      @ned: In the long run the macOS installers should be build using the latest SDK, primarily to get full API coverage and access to all system APIs.

      AFAIK building using the macOS 10.9 SDK still excludes a number of libSystem APIs that would be made available through the posix module when building with a newer SDK.

      That's something that would require some effort though to ensure that the resulting binary still works on older versions of macOS (basically similar to the work I've done in the post to weak link some other symbols in the posix module).

    8. ned-deily commented on Jun 4, 2018

      @ned-deily
      ned-deily
      on Jun 4, 2018
      Member

      {Note: this is not particularly relevant to the issue here.)

      Ronald:

      In the long run the macOS installers should be build using the latest SDK [...] That's something that would require some effort though to ensure that the resulting binary still works on older versions of macOS

      I agree that being able to build with the latest SDK would be nice but it's also true it would require effort on our part, both one-time and ongoing, at least for every new macOS SDK release and update to test with each older system. It would also require that the third-party libraries we build for an installer also behave correctly. And to make full use of it, third-party Python packages with extension modules would also need to behave correctly. I see one of the primary use cases for the python.org macOS installers as being for Python app developers who want to provide apps that run on a range of macOS releases. It seems to me that the safest and simplest way to guarantee that python.org macOS Pythons fulfill that need is to continue to always build them on the oldest supported system. Yes, that means that users may miss out on a few features only supported on the more recent macOS releases but I think that's the right trade-off until we have the resources to truly investigate and decide to support weak linking from current systems.

    9. warsaw commented on Nov 13, 2018

      @warsaw
      warsaw
      on Nov 13, 2018
      Member

      bpo-35219 is where I've run into this problem. I'm still trying to figure out all the details in my own case, but I can confirm that setting the environment variable does not always help.

    10. warsaw
      added
      3.7 (EOL)end of life
      3.8 (EOL)end of life
      on Nov 13, 2018
    11. warsaw
      changed the title [-]High Sierra hang when using multi-processing[/-] [+]macOS crashes after fork with no exec[/+] on Nov 13, 2018
    12. warsaw
      changed the title [-]macOS crashes after fork with no exec[/-] [+]Pytho crashes on macOS after fork with no exec[/+] on Nov 13, 2018
    13. warsaw
      changed the title [-]Pytho crashes on macOS after fork with no exec[/-] [+]Python crashes on macOS after fork with no exec[/+] on Nov 14, 2018

    14. warsaw commented on Nov 14, 2018

      @warsaw
      warsaw
      on Nov 14, 2018
      Member

      Hoo boy. I'm not sure I have the full picture, but things are starting to come into focus. After much debugging, I've narrowed down at least one crash to urllib.request.getproxies(). On macOS (darwin), this ends up calling _scproxy.get_proxies() which calls into the SystemConfiguration framework. I'll bet dollars to donuts that that calls into the ObjC runtime. Thus it is unsafe to call between fork and exec. This certainly seems to be the case even if the environment variable is set.

      The problem is that I think requests.post() probably also ends up in here somehow (still untraced), because by removing our call to urllib.requests.getproxies(), we just crash later on when requests.post() is called.

      I don't know what, if anything can be done in Python, except perhaps to document that anything that calls into the ObjC runtime between fork and exec can potentially crash the subprocess.

    15. 66 remaining items

    16. ahmedsayeed1982
      added
      3.7 (EOL)end of life
      and removed
      3.8 (EOL)end of life
      on Nov 4, 2021
    17. eryksun
      added
      OS-mac
      3.8 (EOL)end of life
      and removed
      stdlibStandard Library Python modules in the Lib/ directory
      3.7 (EOL)end of life
      on Nov 4, 2021
    18. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    19. digantdesai
      added 2 commits that reference this issue on Nov 27, 2024

      Use platform dependent start method

      4e5fb89

      Handle platform dependent start method

      751f712
    20. minjang
      added a commit that references this issue on Nov 27, 2024

      Handle platform dependent start method (#179)

      b3e811b
    21. MiCurry
      added a commit that references this issue on Mar 20, 2025

      Set Python multiprocess launch method to be fork

      8ccd751
    22. jwilk
      added a commit that references this issue on Sep 23, 2026

      Support non-"fork" multiprocessing start methods.

      1b619de
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.8 (EOL)end of lifeOS-mactype-crashA hard crash of the interpreter, possibly with a core dump

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions