Repository navigation
Can not import modules if sys.prefix contains DELIM #75393
Description
| BPO | 31210 |
|---|---|
| Nosy | @brettcannon, @pfmoore, @ncoghlan, @vstinner, @tjguk, @cedk, @bitdancer, @ericsnowcurrently, @zware, @eryksun, @zooba |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2019-09-26.00:55:34.746>
created_at = <Date 2017-08-15.12:29:44.531>
labels = ['interpreter-core', 'invalid', 'type-bug', '3.7']
title = 'Can not import modules if sys.prefix contains DELIM'
updated_at = <Date 2019-09-26.00:55:34.745>
user = 'https://github.com/cedk'bugs.python.org fields:
activity = <Date 2019-09-26.00:55:34.745>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-09-26.00:55:34.746>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2017-08-15.12:29:44.531>
creator = 'ced'
dependencies = []
files = []
hgrepos = []
issue_num = 31210
keywords = []
message_count = 17.0
messages = ['300293', '300294', '300295', '300296', '300302', '300304', '300307', '300308', '300329', '300360', '300361', '300370', '300374', '300375', '300388', '300403', '353247']
nosy_count = 11.0
nosy_names = ['brett.cannon', 'paul.moore', 'ncoghlan', 'vstinner', 'tim.golden', 'ced', 'r.david.murray', 'eric.snow', 'zach.ware', 'eryksun', 'steve.dower']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31210'
versions = ['Python 3.7']
Activity
cedk commentedon Aug 15, 2017
If the path on which Python is installed contains the DELIM, the resulted sys.path is split.
I think maybe there should be a escape mechanism for the PYTHONPATH.
bitdancer commentedon Aug 15, 2017
By DELIM, you mean the shell ':'? As far as I've been able to determine there is no way defined in posix to escape that character. If you can find one, please let us know. (I think the same is true for the Windows semicolon but I'm not sure.)
cedk commentedon Aug 15, 2017
Yes I mean ':' for posix.
Indeed I do not known a posix way to escape the colon (it was discussed here: https://stackoverflow.com/questions/14661373/how-to-escape-colon-in-path-on-unix).
But it does not mean that the method makepathobject could not support one. I'm thinking about path single-quoted.
bitdancer commentedon Aug 15, 2017
I'm not sure there is anything we should do here, then, because we are conforming to the posix parsing for PATH in our PYTHONPATH implementation.
I think if you want to pursue this further you should take it to the python-ideas mailing list. I'm going to close the issue, but you can reopen it if python-ideas thinks it is worth doing something (and there is something we can reasonably do :)
cedk commentedon Aug 15, 2017
A last comment, I do not think it is an issue to follow posix way to parse PATH. But for me, the problem is that Python adds without sanitation the sys.prefix to the PYTHONPATH. So I think internally Python should not use PATH notation to extend the PYTHONPATH because it is not a robust notation.
bitdancer commentedon Aug 15, 2017
You mean to create the entries on sys.path that do not come from the PYTHONPATH?
cedk commentedon Aug 15, 2017
On 2017-08-15 17:32, R. David Murray wrote:
Yes because such path could contain ':'.
bitdancer commentedon Aug 15, 2017
OK, that seems reasonable to me. I'll reopen the issue. Assuming other developers agree that this should be changed, I'm not sure if it will qualify as a bug or an enhancement, so I'm leaving versions unselected for now :)
ncoghlan commentedon Aug 16, 2017
I'm marking this as Python 3.7 only, not because I don't think it's a bug, but because getpath.c is a maintainability nightmare and I strongly prefer we avoid going anywhere near it in maintenance releases :)
Targeting Python 3.7+ also means we may be able to take advantage of the initial phase of PEP-432 changes and look at switching getpath.c to use higher level CPython data structures (specifically list objects) internally.
I've also added the Windows folks to the nosy list to check if Windows might have a similar problem with semi-colons appearing in sys.prefix.
zooba commentedon Aug 16, 2017
We don't. Semicolons are not valid path characters, so you can't have them in a single path (unless you're deliberately trying to break your entire machine...)
bitdancer commentedon Aug 16, 2017
Is this post wrong then?:
https://superuser.com/questions/584870/how-can-i-add-a-folder-containing-a-semicolon-to-the-windows-path
("I noticed that the semicolon ; is a valid character for Windows (NTFS) file and directory names.")
eryksun commentedon Aug 16, 2017
No, it's not wrong that semicolon is a valid character in Windows NTFS and FAT32 filesystems.
However, the answer by Kevin Edwards that recommends using double quotes needs qualification. It only works in CMD. It doesn't work in PowerShell nor the runtime library functions used by CreateProcess or SearchPath (i.e. RtlDosSearchPath_U[str]) or the loader's LdrpSearchPath function. These all split PATH on semicolon, with no supported escape character or quoting. In fact they retain double quotes in the computed filename when testing for existence, so quoting paths in PATH is wrong in general.
bitdancer commentedon Aug 16, 2017
All right. So the challenge here for windows is: if python is installed on a path that has a semicolon in one of the directory names, is it even possible to populate sys.path such that modules can be imported from the lib directory that is under that path? (IMO we shouldn't jump through big hoops to support this, but if it can be done and the refactoring Nick is contemplating makes it *relatively* easily, it would probably be good to do, to be consistent).
[-]Can not import site from sys.prefix containing DELIM[/-][+]Can not import modules if sys.prefix contains DELIM[/+]eryksun commentedon Aug 16, 2017
It's simple to work around using a junction or symbolic link, so is this worth pursuing?
cedk commentedon Aug 16, 2017
I'm wondering if it could have security implications and be used to fool user by changing the PYTHONPATH.
ncoghlan commentedon Aug 17, 2017
If you have access to modify PYTHONPATH at all, you can already shadow almost all standard library modules:
$ PYTHONPATH=/MY_CHOSEN_DIRECTORY python3 -m site sys.path = [ '/home/ncoghlan', '/MY_CHOSEN_DIRECTORY', '/usr/lib64/python36.zip', '/usr/lib64/python3.6', '/usr/lib64/python3.6/lib-dynload', '/home/ncoghlan/.local/lib/python3.6/site-packages', '/usr/lib64/python3.6/site-packages', '/usr/lib/python3.6/site-packages', ]The only ones you can't shadow that way are builtin and frozen modules, and any modules that get imported even before PYTHONPATH is processed. So no, this doesn't open up any new attack vectors that weren't already present by design.
As far as whether or not it's worth fixing goes, yes, I think so - one of my original motivations for writing PEP-432 was to allow the use of CPython data structures when calculating the initial value of sys.path, and this is a nice concrete example of a bug arising from the current implementation.
vstinner commentedon Sep 26, 2019
I consider that the current behavior is correct. They are other ways to put a path which contains ":" into sys.path. I close the issue.