Skip to content

Mpy modules cannot be imported (sys.path missing '') when running file outside of cwd #3509

Open
Open
Mpy modules cannot be imported (sys.path missing '') when running file outside of cwd#3509
Labels
py-coreRelates to py/ directory in source

Description

@ryannathans
ryannathans
opened on Dec 21, 2017

Found on unix port - protostuffer is a single .py file in the modules folder, built in at compile time

./micropython
MicroPython v1.9.3-7-g46e9ef6-dirty on 2017-12-21; linux version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> import protostuffer
>>>

./micropython ../../tests/protostuffer-tests/header.py
Traceback (most recent call last):
  File "../../tests/protostuffer-tests/header.py", line 1, in <module>
ImportError: no module named 'protostuffer'

To fix

./micropython -i ../../tests/protostuffer-tests/header.py
Traceback (most recent call last):
  File "../../tests/protostuffer-tests/header.py", line 1, in <module>
ImportError: no module named 'protostuffer'
MicroPython v1.9.3-7-g46e9ef6-dirty on 2017-12-21; linux version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> import protostuffer
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: no module named 'protostuffer'
>>> import sys
>>> sys.path.append('')
>>> import protostuffer
>>>

Activity

  1. dpgeorge commented on Dec 22, 2017

    @dpgeorge
    dpgeorge
    on Dec 22, 2017
    Member

    I can confirm this issue.

    The way CPython works is that the first entry in sys.path is the directory of the script being executed, and the second entry in sys.path is the current directory. uPy doesn't have the latter and it probably should, but the docs (https://docs.python.org/3/library/sys.html#sys.path) need to be understood to see exactly what values to put in sys.path (it looks like it should be '' instead of the explicit current dir, found by running CPython with the -S option).

  2. wnienhaus commented on Aug 11, 2019

    @wnienhaus
    wnienhaus
    on Aug 11, 2019
    Contributor

    (As suggested, I tested the difference between Micropython and python3, specifically around adding '' to sys.path and MicroPython applies the exact same logic as python3 (see https://gist.github.com/wnienhaus/65727fff5f4954d0e9418ffeaf9bbb6d), but python3 does not have frozen modules like MicroPython. See below.)

    I wanted to give this issue a go, but ran into a logical problem: There is an ambiguity around the meaning of '' in sys.path, in that it refers both to the current directory and to the "root directory" of frozen modules.

    If '' is not part of sys.path for a script given as a command line argument, frozen modules cannot be loaded at all (what this issue is about). When appending '' to sys.path (the workaround proposed by @ryannathans), it "works" but also allows loading modules/scripts from the current directory, which "real" python does not allow.

    The ambiguity around '' also means one cannot define a different relative preference between frozen modules and the current directory. For example, this issue can be "solved" as mentioned before, by adding'' to the end of sys.path. Now frozen modules can be loaded, however it also means that the current directory is now searched last during import (even after frozen modules as per mp_import_stat_any()).

    This becomes relevant when the REPL interface for example (or with code via stdin) "officially" has '' as the first entry in sys.path (also in "real" python), resulting in frozen modules being searched before library paths. Since frozen code feels "built-in" perhaps it should be searched after library paths in sys.path? However, if one moves the '' to the end of sys.path (so frozen modules are searched last), it would also mean that the current directory is searched after library paths, which is usually not desired.

    One option could be to append '' to sys.path only for the case of scripts called as command line arguments. Since "real" python does not allow running scripts from the current directory anyway, perhaps it does not matter so much that the current directory is searched after library paths? But... now we would have an inconsistency between the search order of directly called scripts vs the REPL (or code via stdin / -c).

    (It would also open up a potential security issue, where intentionally frozen modules could be overridden by an adversary by placing modified versions of those modules into the current directory - which can be a directory of the adversary's choosing, not necessarily the directory containing the called script.)

    I dont really like any of the above issues, and any option with '' in sys.path would not match "real" python, but then again, python3 does not have frozen modules. The closest in real python I suppose could be the support for archives (e.g. zip), so perhaps that can lead to a nice approximate solution?

    I can see 3 alternatives:

    1. add '' immediately after the script base directory

    i.e. for ./micropython path/to/script.py we would have:
    sys.path == ['path/to', '', '~/.micropython/lib/', '/usr/lib/micropython']

    1. add '' to the end of sys.path

    i.e. for ./micropython path/to/script.py we would have:
    sys.path == ['path/to', '~/.micropython/lib/', '/usr/lib/micropython', '']

    1. Mimic python3's support for archives in sys.path, by inventing a mechanism to address frozen files differently from files in the current directory. For example we could invent a fake filename /.frozen.zip - but it's not a zip, or we could invent a token referring to the frozen that would never be a file by itself, e.g. <mpy>, or even borrow from urls and invent a scheme mpy://. In this case we should also add the identifier to the end of sys.path for cases where no script was passed (such as the REPL case or passing code via stdin).

    Option 1 and 2 are very easy to implement (I have a patch for 1 - see https://gist.github.com/wnienhaus/13010c88ef89514d4de2a8b4902d8513). Option 3 is more effort and requires adaptations in a number places.

    I would like to ask for comment/input on the above. I am happy to give complex (but better) solutions a go and make a Pull Request, but I do not want to waste time if we can find simpler solutions or we agree on Option 1 or 2. (between those two, I would favour option 1 for consistency with the REPL).

  3. dpgeorge commented on Aug 17, 2019

    @dpgeorge
    dpgeorge
    on Aug 17, 2019
    Member

    @wnienhaus thanks for looking into this.

    But, I think there is some confusion as to what issue is what. Although the title of this issue has "mpy" in it (probably that should be changed to just "py") it's actually about pure .py files, that MicroPython does not insert the current directory as the second element in sys.path.

    Searching for frozen modules, and how sys.path affects that, is a separate issue: #2322 . And for frozen files I do prefer option 3 above, to provide a way to specify the search order of frozen files via a meta path name.

  4. wnienhaus commented on Aug 17, 2019

    @wnienhaus
    wnienhaus
    on Aug 17, 2019
    Contributor

    Thank you for linking to #2322. Wonder how I did not find that. Anyway, it's exactly the issue I am trying to resolve. My option 3 is the same as option 2 in #2322 and I found it is actually very easy to implement, but I will continue the discussion around this in #2322.

    About this issue: It seemed like the original poster did refer to a module "built in at compile time" (which I interpreted as frozen). If so, then this is definitely a duplicate of #2322 and should likely be closed.

    If this issue was about importing a script (.py file) from the current directory, I feel MicroPython already behaves correctly (compared to CPython), and nothing should be changed. If anyone wants this behaviour, they can always have their script append '' to sys.path right at the top of their script.

  5. dpgeorge commented on Aug 17, 2019

    @dpgeorge
    dpgeorge
    on Aug 17, 2019
    Member

    About this issue: It seemed like the original poster did refer to a module "built in at compile time" (which I interpreted as frozen). If so, then this is definitely a duplicate of #2322 and should likely be closed.

    Yes it does indeed seem like that.

    If this issue was about importing a script (.py file) from the current directory, I feel MicroPython already behaves correctly (compared to CPython), and nothing should be changed.

    In my comment above (second post) I must have misunderstood the original report, because what I was talking about there was the following problem (which has nothing to do with frozen modules): if you go into tests/basics and do:

    $ python -i ../float/float1.py
(runs float1.py then drops to prompt)
>>> import array1
(runs array1.py then drops to prompt)

    but:

    $ micropython -i ../float/float1.py
(runs float1.py then drops to prompt)
>>> import array1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: no module named 'array1'

    CPython works because it puts the current directory second in sys.path, while uPy does not (both put the directory of the executing script first in the path).

    Anyway, I guess that issue just described is not that much of an issue, since no one really reported it. Rather #2322 is a real problem to be fixed.

  6. wnienhaus commented on Sep 1, 2019

    @wnienhaus
    wnienhaus
    on Sep 1, 2019
    Contributor

    I cannot repeat the output you had with CPython. When I use -i it fails in the same way as MicroPython does. I don't see a situation where CPython puts the directory of the executing script first in sys,path, followed by the current directory.

    I have now submitted a PR (#5057) to attempt to fix this. There's already a discussion there, so perhaps the final fix will be different, but happy to explore options.

    Btw, I noticed that -i is undocumented in MicroPython (at least in the -h output) - I will submit a PR to address that.

  7. dpgeorge
    added a commit that references this issue on Dec 17, 2021

    py: Only search frozen modules when '.frozen' is found in sys.path.

    e0bf461
  8. dpgeorge
    added
    py-coreRelates to py/ directory in source
    on Dec 17, 2021

  9. dpgeorge commented on Dec 17, 2021

    @dpgeorge
    dpgeorge
    on Dec 17, 2021
    Member

    .frozen in sys.path was implemented in e0bf461, but I'm not sure that fully fixes this issue.

  10. leifbirger
    added a commit that references this issue on Jun 14, 2023

    py: Only search frozen modules when '.frozen' is found in sys.path.

    aec0f4b
  11. Taccart
    added a commit that references this issue on May 23, 2026

    py: Only search frozen modules when '.frozen' is found in sys.path.

    5803181
  12. jakub-vesely
    added a commit that references this issue on Sep 21, 2026

    py: Only search frozen modules when '.frozen' is found in sys.path.

    a0e05c3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    py-coreRelates to py/ directory in source

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions