Py_SetStandardStreamEncoding leads to a memory error in debug mode #74004

Activity

1. ncoghlan commented on Mar 15, 2017

   

   ncoghlan

   on Mar 15, 2017

   ContributorAuthor

   More actions

   For PEP-538, setting PYTHONIOENCODING turned out to have undesirable side effects on Python 2 instances in subprocesses, since Python 2 has no 'surrogateescape' error handler.

   So I switched to using the "Py_SetStandardStreamEncoding" API defined in http://bugs.python.org/issue16129 instead, but this turns out to have problematic interactions with the dynamic memory allocator management, so it fails with a fatal exception in debug mode. An example of the error can be seen here: https://travis-ci.org/python/cpython/jobs/211293576

   The problem appears to be that between the allocation of the memory with _PyMem_RawStrdup in Py_SetStandardStreamEncoding and the release of that memory in initstdio, the active memory manager has changed (at least in a debug build), so the deallocation as part of the interpreter startup fails.

   That interpretation is based on this comment in Programs/python.c:

       /* Force again malloc() allocator to release memory blocks allocated
          before Py_Main() */
       (void)_PyMem_SetupAllocators("malloc");
   

   The allocations in Py_SetStandardStreamEncoding happen before the call to Py_Main/Py_Initialize, but the deallocation happens in Py_Initialize.

   The "fix" I applied to the PEP branch was to make the default allocator conditional in Programs/python.c as well:

   #ifdef Py_DEBUG
       (void)_PyMem_SetupAllocators("malloc_debug");
   #  else
       (void)_PyMem_SetupAllocators("malloc");
   #  endif
   

   While that works (at least in the absence of a PYTHONMALLOC setting) it seems fragile. It would be nicer if there was a way for Py_SetStandardStreamEncoding to indicate which allocator should be used for the deallocation.
2. ncoghlan

   added

   type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

   3.7 (EOL)end of lifeend of life

   on Mar 15, 2017
3. vstinner commented on May 15, 2019

   

   vstinner

   on May 15, 2019

   Member

   More actions

   I fixed this issue in Python 3.7. Py_SetStandardStreamEncoding() now uses:

   PyMemAllocatorEx old_alloc;
   _PyMem_SetDefaultAllocator(PYMEM_DOMAIN_RAW, &old_alloc);
   
   ... _PyMem_RawStrdup() ...
   

       PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &old_alloc);
4. vstinner

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   on May 15, 2019
5. vstinner

   closed this as completedon May 15, 2019
6. ezio-melotti

   transferred this issue fromon Apr 10, 2022