Can not import modules if sys.prefix contains DELIM

@brettcannon

BPO	31210
Nosy	@brettcannon, @pfmoore, @ncoghlan, @vstinner, @tjguk, @cedk, @bitdancer, @ericsnowcurrently, @zware, @eryksun, @zooba

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-09-26.00:55:34.746>
created_at = <Date 2017-08-15.12:29:44.531>
labels = ['interpreter-core', 'invalid', 'type-bug', '3.7']
title = 'Can not import modules if sys.prefix contains DELIM'
updated_at = <Date 2019-09-26.00:55:34.745>
user = 'https://github.com/cedk'

bugs.python.org fields:

activity = <Date 2019-09-26.00:55:34.745>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-09-26.00:55:34.746>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2017-08-15.12:29:44.531>
creator = 'ced'
dependencies = []
files = []
hgrepos = []
issue_num = 31210
keywords = []
message_count = 17.0
messages = ['300293', '300294', '300295', '300296', '300302', '300304', '300307', '300308', '300329', '300360', '300361', '300370', '300374', '300375', '300388', '300403', '353247']
nosy_count = 11.0
nosy_names = ['brett.cannon', 'paul.moore', 'ncoghlan', 'vstinner', 'tim.golden', 'ced', 'r.david.murray', 'eric.snow', 'zach.ware', 'eryksun', 'steve.dower']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31210'
versions = ['Python 3.7']

If the path on which Python is installed contains the DELIM, the resulted sys.path is split.
I think maybe there should be a escape mechanism for the PYTHONPATH.

By DELIM, you mean the shell ':'? As far as I've been able to determine there is no way defined in posix to escape that character. If you can find one, please let us know. (I think the same is true for the Windows semicolon but I'm not sure.)

Yes I mean ':' for posix.
Indeed I do not known a posix way to escape the colon (it was discussed here: https://stackoverflow.com/questions/14661373/how-to-escape-colon-in-path-on-unix).
But it does not mean that the method makepathobject could not support one. I'm thinking about path single-quoted.

I'm not sure there is anything we should do here, then, because we are conforming to the posix parsing for PATH in our PYTHONPATH implementation.

I think if you want to pursue this further you should take it to the python-ideas mailing list. I'm going to close the issue, but you can reopen it if python-ideas thinks it is worth doing something (and there is something we can reasonably do :)

A last comment, I do not think it is an issue to follow posix way to parse PATH. But for me, the problem is that Python adds without sanitation the sys.prefix to the PYTHONPATH. So I think internally Python should not use PATH notation to extend the PYTHONPATH because it is not a robust notation.

You mean to create the entries on sys.path that do not come from the PYTHONPATH?

On 2017-08-15 17:32, R. David Murray wrote:

You mean to create the entries on sys.path that do not come from the PYTHONPATH?

Yes because such path could contain ':'.

OK, that seems reasonable to me. I'll reopen the issue. Assuming other developers agree that this should be changed, I'm not sure if it will qualify as a bug or an enhancement, so I'm leaving versions unselected for now :)

I'm marking this as Python 3.7 only, not because I don't think it's a bug, but because getpath.c is a maintainability nightmare and I strongly prefer we avoid going anywhere near it in maintenance releases :)

Targeting Python 3.7+ also means we may be able to take advantage of the initial phase of PEP-432 changes and look at switching getpath.c to use higher level CPython data structures (specifically list objects) internally.

I've also added the Windows folks to the nosy list to check if Windows might have a similar problem with semi-colons appearing in sys.prefix.

We don't. Semicolons are not valid path characters, so you can't have them in a single path (unless you're deliberately trying to break your entire machine...)

Is this post wrong then?:

https://superuser.com/questions/584870/how-can-i-add-a-folder-containing-a-semicolon-to-the-windows-path

("I noticed that the semicolon ; is a valid character for Windows (NTFS) file and directory names.")

Is this post wrong then?

No, it's not wrong that semicolon is a valid character in Windows NTFS and FAT32 filesystems.

However, the answer by Kevin Edwards that recommends using double quotes needs qualification. It only works in CMD. It doesn't work in PowerShell nor the runtime library functions used by CreateProcess or SearchPath (i.e. RtlDosSearchPath_U[str]) or the loader's LdrpSearchPath function. These all split PATH on semicolon, with no supported escape character or quoting. In fact they retain double quotes in the computed filename when testing for existence, so quoting paths in PATH is wrong in general.

All right. So the challenge here for windows is: if python is installed on a path that has a semicolon in one of the directory names, is it even possible to populate sys.path such that modules can be imported from the lib directory that is under that path? (IMO we shouldn't jump through big hoops to support this, but if it can be done and the refactoring Nick is contemplating makes it *relatively* easily, it would probably be good to do, to be consistent).

It's simple to work around using a junction or symbolic link, so is this worth pursuing?

I'm wondering if it could have security implications and be used to fool user by changing the PYTHONPATH.

If you have access to modify PYTHONPATH at all, you can already shadow almost all standard library modules:

$ PYTHONPATH=/MY_CHOSEN_DIRECTORY python3 -m site
sys.path = [
    '/home/ncoghlan',
    '/MY_CHOSEN_DIRECTORY',
    '/usr/lib64/python36.zip',
    '/usr/lib64/python3.6',
    '/usr/lib64/python3.6/lib-dynload',
    '/home/ncoghlan/.local/lib/python3.6/site-packages',
    '/usr/lib64/python3.6/site-packages',
    '/usr/lib/python3.6/site-packages',
]

The only ones you can't shadow that way are builtin and frozen modules, and any modules that get imported even before PYTHONPATH is processed. So no, this doesn't open up any new attack vectors that weren't already present by design.

As far as whether or not it's worth fixing goes, yes, I think so - one of my original motivations for writing PEP-432 was to allow the use of CPython data structures when calculating the initial value of sys.path, and this is a nice concrete example of a bug arising from the current implementation.

I consider that the current behavior is correct. They are other ways to put a path which contains ":" into sys.path. I close the issue.

Navigation Menu

Repository navigation

Can not import modules if sys.prefix contains DELIM #75393

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Can not import modules if sys.prefix contains DELIM #75393

Description

Activity

cedk commented on Aug 15, 2017

bitdancer commented on Aug 15, 2017

cedk commented on Aug 15, 2017

bitdancer commented on Aug 15, 2017

cedk commented on Aug 15, 2017

bitdancer commented on Aug 15, 2017

cedk commented on Aug 15, 2017

bitdancer commented on Aug 15, 2017

ncoghlan commented on Aug 16, 2017

zooba commented on Aug 16, 2017

bitdancer commented on Aug 16, 2017

eryksun commented on Aug 16, 2017

bitdancer commented on Aug 16, 2017

eryksun commented on Aug 16, 2017

cedk commented on Aug 16, 2017

ncoghlan commented on Aug 17, 2017

vstinner commented on Sep 26, 2019

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions