Repository navigation
Usage of API method Py_SetPath causes errors in Py_Initialize() (Posix ony)) #55529
Description
| BPO | 11320 |
|---|---|
| Nosy | @warsaw, @ncoghlan, @pitrou, @vstinner, @tarekziade, @ericsnowcurrently, @zooba, @melvyn-sopacua |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2019-05-27.15:33:59.760>
created_at = <Date 2011-02-25.14:32:02.620>
labels = ['interpreter-core', 'type-bug']
title = 'Usage of API method Py_SetPath causes errors in Py_Initialize() (Posix ony))'
updated_at = <Date 2019-05-27.15:33:59.759>
user = 'https://bugs.python.org/palmkevin'bugs.python.org fields:
activity = <Date 2019-05-27.15:33:59.759>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-05-27.15:33:59.760>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2011-02-25.14:32:02.620>
creator = 'palm.kevin'
dependencies = []
files = []
hgrepos = []
issue_num = 11320
keywords = []
message_count = 22.0
messages = ['129372', '129395', '129396', '131101', '131131', '131137', '131292', '131293', '131297', '131301', '131316', '131317', '131318', '131324', '144244', '249276', '249311', '249321', '249342', '249383', '276050', '343647']
nosy_count = 11.0
nosy_names = ['barry', 'ncoghlan', 'pitrou', 'vstinner', 'tarek', 'srid', 'palm.kevin', 'eric.snow', 'dbzhang800', 'steve.dower', 'msopacua']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue11320'
versions = ['Python 3.2', 'Python 3.3', 'Python 3.4', 'Python 3.5']
Activity
palmkevin commentedon Feb 25, 2011
The raised exception is the following:
(perhaps linked to issue bpo-10743 ?!)
srid commentedon Feb 25, 2011
This issue is potentially breaking virtualenv5,
http://code.google.com/p/virtualenv5/issues/detail?id=6
pitrou commentedon Feb 25, 2011
Can you explain why this is a problem in Python?
Can't lib/python3.2/config-3.2m/Makefile simply be provided by virtualenv (by copying it, I guess)?
palmkevin commentedon Mar 16, 2011
@sridhar: Could you please provide input for the question asked by Antoine?
I'd love to have this issue fixed for next Python release.
ncoghlan commentedon Mar 16, 2011
Note that this will be fixed for 3.3, as siteconfig will be updated to use static data (generated during the build process) rather than relying on build artifacts being available at runtime.
However, virtualenv (et al) will still need to provide the relevant static config file in the isolated environments.
srid commentedon Mar 16, 2011
[pitrou]
Yes, I believe virtualenv already does that (or symlinks to it). Python 3.2 changed the path to config and include directories for some reason, viz.
and:
It is possible that virtualenv is hardcoding the relative path to 'config' (and 'include') directories and thus failing to find the new 'config-3.2m' dir. If that is the case, this is not a problem with Python.
Although msg129372 does point to a Python bug, it may or may not be related to the virtualenv issue noted earlier. Given that virtualenv doesn't officially Python 3 and virtualenv5 is more of a hack, I haven't investigated into this much.
Does that answer your question?
pitrou commentedon Mar 17, 2011
Sridhar: I'm sorry, but at this point you should investigate a bit more into the actual causes of the problem. I'm not going to dig in virtualenv myself. Also, since there's already bpo-10743 for the virtualenv issue, perhaps further virtualenv-related comments should be posted there.
(Palm, I take it your issue doesn't have anything to do with virtualenv?)
pitrou commentedon Mar 17, 2011
Ok, Palm's example even segfaults here:
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff72452bc in free () from /lib64/libc.so.6
(gdb) bt
#0 0x00007ffff72452bc in free () from /lib64/libc.so.6
#1 0x000000000042a591 in Py_SetPath (
path=0x88c330 L"/home/antoine/cpython/default/usr/lib/python33.zip:/home/antoine/cpython/default/usr/lib/python3.3/:/home/antoine/cpython/default/usr/lib/python3.3/plat-linux2:/home/antoine/cpython/default/usr/lib/py"...)
at ./Modules/getpath.c:729
#2 0x0000000000416600 in main ()
The problem is calling free() on a pointer to statically allocated memory.
pitrou commentedon Mar 17, 2011
Crash fixed in a791dd7d51f3 (3.2) and b4104ffd5127 (3.3), but the original problem remains, or a variant of it. I now get:
Init
Fatal Python error: Py_Initialize: Unable to get the locale encoding
LookupError: no codec search functions registered: can't find encoding
Abandon
pitrou commentedon Mar 17, 2011
Ok, the issue here is that you can't call Py_SetPath() on the point returned by Py_GetPath(), since the memory refered to that pointer is free()d at the beginning of Py_SetPath(). The following works, though:
Perhaps we could modify Py_SetPath() so that it copies the new path first before deallocating the old one, but I'm not sure I see the point of calling Py_SetPath() with the pointer returned by Py_GetPath().
[-]Usage of API method Py_SetPath causes errors in Py_Initialize() (Posix ony))[/-][+]Can't call Py_SetPath() on pointer returned by Py_GetPath()[/+]palmkevin commentedon Mar 18, 2011
Antoine,
Your guess that my issue initially wasn't related to virtualenv is correct (I've never heard about that project before posting this issue...)
As for passing the output of Py_GetPath directly to Py_SetPath: You are right, there is no point in doing this...
Now, I remember that the initial problem I had was the one you reported:
Fatal Python error: Py_Initialize: Unable to get the locale encoding
LookupError: no codec search functions registered: can't find encoding
Shall I create a separate issue to report this problem?
palmkevin commentedon Mar 18, 2011
Furthermore I would propose to rename this issue: The problem is not that Py_SetPath cannot be called on pointer returned by Py_GetPath. I think that the problem is more general: Calling Py_SetPath NEVER works.
palmkevin commentedon Mar 18, 2011
As for this error:
Fatal Python error: Py_Initialize: Unable to get the locale encoding
LookupError: no codec search functions registered: can't find encoding
It seems to me that this error appears if the path passed to Py_SetPath does not point to a valid python lib folder.
pitrou commentedon Mar 18, 2011
Indeed, it does. The error message could perhaps get improved.
What Python does is try to find the current locale's encoding in the
"encodings" package and load it. If it can't find it, it assumes that's
because the encoding is unknown, not because the path is wrong.
dbzhang800 commentedon Sep 18, 2011
Hello everyone,
I have found the reason for the problem.
From the manual http://docs.python.org/py3k/c-api/init.html#Py_SetPath ,
we can see that:
After we call Py_SetPath,both sys.prefix and sys.exec_prefix will be empty.
However, sys.prefix will be used in the sysconfig.py to generate the makefile' s name, and the empty sys.prefix will cause the wrong path: lib/python3.2/config-3.2m/Makefile
sysconfig.py imported by site.py, and site.py used in the Py_InitializeEx.
So ...
palmkevin commentedon Aug 28, 2015
The problem seems still not resolved in Python 3.2.6 :-(
The reason has been found by Debao (msg144244). Isn't there anybody motivated to fix this bug?
[-]Can't call Py_SetPath() on pointer returned by Py_GetPath()[/-][+]Usage of API method Py_SetPath causes errors in Py_Initialize() (Posix ony))[/+]ncoghlan commentedon Aug 29, 2015
Unfortunately, the answer to the question "Isn't there anybody motivated to fix this bug?" is "No, not really". As far as I am aware, all of the currently active core developers are primarily interested in the use of the default runtime interpreter as is, rather than embedding it in larger applications (which is the main case where PySys_SetPath is needed).
I'm *personally* interested in that area (hence my intermittent updates to PEP-432), but it's purely on my own time rather than being particularly work related.
That said, Steve Dower did do some significant work to provide an embedding friendly variant of the Windows builds for 3.5, so I've added him to the nosy list here in case he might be able to take a look.
zooba commentedon Aug 29, 2015
Seems like a fairly obvious bug. From https://docs.python.org/3/c-api/init.html#c.Py_SetPath
Apparently you can't set
sys.[exec_]prefixbefore calling Py_Initialize, and you can't call Py_Initialize without settingsys.[exec_]prefix(much like bpo-20891, where you can't initialize threads without holding the GIL, and you can't acquire the GIL without having initialized threads.)I don't know how to go about resolving this though (my changes were limited to getpathp.c - and I really need to go add the same changes to the non-Windows getpath.c too...). The best way seems to be forcing Nick to finish PEP-432, but unfortunately I have no leverage over him :)
ncoghlan commentedon Aug 29, 2015
I think the current situation is mainly an artifact of our relative dearth
of regression testing for embedded configurations, and the fact that we
have very few core developers working for companies embedding CPython in
larger applications.
I do care about that space (hence PEP-432), but operating system
integration, software distribution tools, and improved modularisation take
precedence on work time, and the science & education sectors on my personal
time.
I did recently file bpo-24932 to propose investigating and adopting a C
level unit testing framework for the embedding tests, as one of the things
my preliminary work on PEP-432 highlighted is how limited our current
direct testing capabilities for the embedding API are - at this point,
we're mostly limited to testing it the way CPython uses it, which makes it
unfortunately fragile for embedders (as issues like this one show).
palmkevin commentedon Aug 31, 2015
@ncoghlan: Not PySys_SetPath, but Py_SetPath is causing the problem (you mentionned PySys_SetPath in your message msg249311)
I discovered PySys_SetPath only last Friday.
In fact, in my case, the usage of PySys_SetPath (after Py_Initialize) instead of Py_SetPath (before Py_Initialize) is sufficient to respond to my needs!
melvyn-sopacua commentedon Sep 12, 2016
The cause for this error is any cause for not being able to load a module. This includes version conflicts, so when trying to load a virtualenv with python 3.3 interpreter by a uWSGI embedded python3.4 interpreter the action fails.
This is - bluntly overstated - a design flaw or overly confident claim with respect to virtualenv support in uWSGI. This configuration can never work and becomes a configuration error.
It really deserves a less generic error message, indicating the version conflict and possibly the path to the encoding module it tried to load. This helps in debugging the cause and possibly abandon the idea before wading through the list of reports generated by this error message (only one of which I found to hint at the version conflict and most don't have a clear solution or different root cause).
vstinner commentedon May 27, 2019
I understand that this bug has been fixed by the implementation of the PEP-587 in bpo-36763: Py_SethPath() now uses dynamically allocated memory and can be called multiple times. Moreover, the PEP-587 now provides a better API for the Path Configuration. I close the issue.
Reopen/comment the issue if I misunderstood it :-)