ImportWarning when importing a PEP 420 namespace package

The pkg_resources code is raising an ImportWarning when it attempts to import a PEP 420 implicit namespace package:

ImportError: Failed to import test module: test.test_methods
Traceback (most recent call last):
  File "/usr/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/home/bignose/Projects/debian/dput/trunk/test/test_methods.py", line 26, in <module>
    import testscenarios
  File "/usr/lib/python3/dist-packages/testscenarios/__init__.py", line 57, in <module>
    from testscenarios.scenarios import (
  File "/usr/lib/python3/dist-packages/testscenarios/scenarios.py", line 34, in <module>
    from testtools import iterate_tests
  File "/usr/lib/python3/dist-packages/testtools/__init__.py", line 124, in <module>
    from pbr.version import VersionInfo
  File "/usr/lib/python3/dist-packages/pbr/version.py", line 25, in <module>
    import pkg_resources
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3037, in <module>
    @_call_aside
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3021, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3065, in _initialize_master_working_set
    for dist in working_set
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3065, in <genexpr>
    for dist in working_set
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2595, in activate
    declare_namespace(pkg)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2163, in declare_namespace
    _handle_ns(packageName, path_item)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2083, in _handle_ns
    loader = importer.find_module(packageName)
  File "<frozen importlib._bootstrap_external>", line 445, in _find_module_shim
ImportWarning: Not importing directory /usr/lib/python3/dist-packages/keyrings: missing __init__

(To reproduce this, enable the -W error option to python3. I did this to try to catch warnings in my code, assuming that libraries would not cause them :-/)

The “Not importing directory […] missing init” contradicts the explicit specification from PEP 420:

Namespace packages [as specified in this PEP] cannot contain an init.py .

I am not very familiar with namespace packages, so I could be wrong about where the fault lies. I am sure, though, that there is a fault, and it needs to be resolved to avoid the ImportWarning.

I'm seeing this as well. It is causing loads of noise in my logs when Python warnings are enabled. Here is a minimal bash script to demonstrate the error:

#!/bin/bash

rm -rf venv
python3 -m venv venv
venv/bin/pip install sphinxcontrib-websupport
venv/bin/python -Werror -c 'import pkg_resources'

Actual output:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/jon/warning-test/venv/lib64/python3.5/site-packages/pkg_resources/__init__.py", line 2958, in <module>
    @_call_aside
  File "/home/jon/warning-test/venv/lib64/python3.5/site-packages/pkg_resources/__init__.py", line 2944, in _call_aside
    f(*args, **kwargs)
  File "/home/jon/warning-test/venv/lib64/python3.5/site-packages/pkg_resources/__init__.py", line 2985, in _initialize_master_working_set
    dist.activate(replace=False)
  File "/home/jon/warning-test/venv/lib64/python3.5/site-packages/pkg_resources/__init__.py", line 2521, in activate
    declare_namespace(pkg)
  File "/home/jon/warning-test/venv/lib64/python3.5/site-packages/pkg_resources/__init__.py", line 2103, in declare_namespace
    _handle_ns(packageName, path_item)
  File "/home/jon/warning-test/venv/lib64/python3.5/site-packages/pkg_resources/__init__.py", line 2037, in _handle_ns
    loader = importer.find_module(packageName)
  File "<frozen importlib._bootstrap_external>", line 445, in _find_module_shim
ImportWarning: Not importing directory /home/jon/warning-test/venv/lib64/python3.5/site-packages/sphinxcontrib: missing __init__

Resolve warning filter issues

Ignore ruamel import warnings

As noted above, the lack of init.py is fine according to PEP420. The error itself comes from the builtin imp module in python2.7:

/usr/lib/python2.7/pkgutil.py:186: ImportWarning: Not importing directory '/home/kleptog/venv/lib/python2.7/site-packages/zc': missing __init__.py
  file, filename, etc = imp.find_module(subname, path)

So the real fix for this is for python itself to stop complaining (given it just works), or barring that suppress the warning in the package that calls it, which is setuptools.

One plan that addresses this issue is to replace the vast majority of uses of pkg_resources with stdlib functionality (and backports), such as with importlib_resources, importlib_metadata and entrypoints.

a minimal bash script to demonstrate the error

I tried replicating the minimal script on Python 3.7, but it fails first on another warning:

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 943, in <module>
    collections.MutableMapping.register(ParseResults)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/collections/__init__.py", line 52, in __getattr__
    DeprecationWarning, stacklevel=2)
DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working

I've reported that in #1480.

I tried replicating the issue on Python 3.6 but couldn't:

~ $ python3.6 -m venv venv
~ $ venv/bin/pip install sphinxcontrib-websupport
Collecting sphinxcontrib-websupport
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/52/69/3c2fbdc3702358c5b34ee25e387b24838597ef099761fc9a42c166796e8f/sphinxcontrib_websupport-1.1.0-py2.py3-none-any.whl
Installing collected packages: sphinxcontrib-websupport
Successfully installed sphinxcontrib-websupport-1.1.0
You are using pip version 9.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
~ $ venv/bin/python -c 'import pkg_resources'; echo no problem
no problem

I should note that if you have rwt (and you should), it's easier and cleaner to test with this one liner (and doesn't leave you with ./venv):

~ $ python3.6 -m rwt sphinxcontrib-websupport -- -Werror -c 'import pkg_resources'; echo no problem
Collecting sphinxcontrib-websupport
  Using cached https://files.pythonhosted.org/packages/52/69/3c2fbdc3702358c5b34ee25e387b24838597ef099761fc9a42c166796e8f/sphinxcontrib_websupport-1.1.0-py2.py3-none-any.whl
Installing collected packages: sphinxcontrib-websupport
Successfully installed sphinxcontrib-websupport-1.1.0
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
no problem

Let's see if we can ascertain why the issue exists in your environment but not in mine.

I upgraded my setuptools and pip and still no issue:

~ $ python3.6 -m pip install -q -U setuptools pip
~ $ python3.6 -m rwt -q sphinxcontrib-websupport -- -Werror -c 'import pkg_resources'; echo no problem
no problem

I tested on Python 3.5.2 and 3.6.4 and 3.6.6, but none of those exhibit the reported failure. Maybe the issue has gone away?

@bignose-debian

@bignose-debian Would you consider creating a Dockerfile that replicates the issue? With that, I could troubleshoot in the environment where the issue occurs.

@jaraco

@jaraco wrote:

Would you consider creating a Dockerfile that replicates the issue?

Yes, though I haven't a clue how to do that (nor how to know whether the result would correctly replicate the behaviour).

I'll take a stab at it.

I was able to replicate thus:

draft $ cat > Dockerfile
from python:3.6
RUN python -m venv venv
RUN venv/bin/python --version
RUN venv/bin/python -m pip install sphinxcontrib-websupport
CMD venv/bin/python -W error -c 'import pkg_resources'
draft $ docker build .
Sending build context to Docker daemon  13.04MB
Step 1/5 : from python:3.6
3.6: Pulling from library/python
05d1a5232b46: Pull complete
5cee356eda6b: Pull complete
89d3385f0fd3: Pull complete
80ae6b477848: Pull complete
28bdf9e584cc: Pull complete
dec1a1f0462b: Pull complete
a4670d125615: Pull complete
547b45a875f5: Pull complete
102a0247b454: Pull complete
Digest: sha256:0c3b6384565b8524a82096bee143174a1f07a28cfdc090ca63c040251ab10604
Status: Downloaded newer image for python:3.6
 ---> 4f13b7f2138e
Step 2/5 : RUN python -m venv venv
 ---> Running in 31b6974b096e
Removing intermediate container 31b6974b096e
 ---> 48aa5b2ac9ae
Step 3/5 : RUN venv/bin/python --version
 ---> Running in 46a4c6c2e3a2
Python 3.6.6
Removing intermediate container 46a4c6c2e3a2
 ---> 5d232b495a62
Step 4/5 : RUN venv/bin/python -m pip install sphinxcontrib-websupport
 ---> Running in 134e7c778b19
Collecting sphinxcontrib-websupport
  Downloading https://files.pythonhosted.org/packages/52/69/3c2fbdc3702358c5b34ee25e387b24838597ef099761fc9a42c166796e8f/sphinxcontrib_websupport-1.1.0-py2.py3-none-any.whl
Installing collected packages: sphinxcontrib-websupport
Successfully installed sphinxcontrib-websupport-1.1.0
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Removing intermediate container 134e7c778b19
 ---> 86e837b01253
Step 5/5 : CMD venv/bin/python -W error -c 'import pkg_resources'
 ---> Running in 0ce64945506a
Removing intermediate container 0ce64945506a
 ---> d90cede4387c
Successfully built d90cede4387c
draft $ docker run d90cede4387c
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3088, in <module>
    @_call_aside
  File "/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3072, in _call_aside
    f(*args, **kwargs)
  File "/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3116, in _initialize_master_working_set
    for dist in working_set
  File "/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3116, in <genexpr>
    for dist in working_set
  File "/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2639, in activate
    declare_namespace(pkg)
  File "/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2176, in declare_namespace
    _handle_ns(packageName, path_item)
  File "/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2096, in _handle_ns
    loader = importer.find_module(packageName)
  File "<frozen importlib._bootstrap_external>", line 426, in _find_module_shim
ImportWarning: Not importing directory /venv/lib/python3.6/site-packages/sphinxcontrib: missing __init__

Not sure if this will be of any help but the warning seems to occur on any namespace packages present in sys.modules that have a "namespace_packages.txt" file in their metadata.

So I've confirmed the .pth file is present and that causes sphinxcontrib to exist in sys.modules before any import of pkg_resources, which is good.

Modules that have namespace_packages.txt in their metadata are those that use setuptools to declare namespace_packages, a deprecated but common technique (and difficult to migrate away from).

I tried replicating the issue on Python 3.6 but couldn't

Glad I pasted my exact commands, because I started replicating it on my local workstation and couldn't figure out what I'd done wrong. I see now I neglected to include the important -W error when I tried to replicate the issue above.

I learned that removing the .pth file works around the issue. That explains why the issue didn't replicate when under rwt (because the .pth files aren't honored in that mode).

draft $ rm venv/lib/python*/site-packages/sphinxcontrib_websupport*.pth
draft $ venv/bin/python -W error -c 'import pkg_resources'; echo no problem
no problem

Digging deeper, I see why the absence of the .pth file bypasses the issue. It's because in pkg_resources.Distribution.activate, the package is not in sys.modules, so the declare_namespace never gets called.

Removing setuptools from the equation, I can replicate the failure with this Dockerfile:

from python:3.6
RUN python -m venv venv
RUN venv/bin/python -m pip install -U pip setuptools
RUN venv/bin/python -m pip install sphinxcontrib-websupport
CMD venv/bin/python -W error -c 'import pkgutil; importer = pkgutil.get_importer("venv/lib/python3.6/site-packages/"); importer.find_module("sphinxcontrib")'

emits

draft $ docker run -it 07e6f2229b27
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap_external>", line 426, in _find_module_shim
ImportWarning: Not importing directory venv/lib/python3.6/site-packages/sphinxcontrib: missing __init__

Suppress warnings in importer.find_module. Fixes #1111.

I'm glad you got such good diagnosis. The Dockerfile doesn't help me much because I just get a bunch of permission errors trying to use it. (I don't have Docker set up on any of my machines so I'm probably not best suited for going down that path for diagnosis.)

Navigation Menu

Repository navigation

ImportWarning when importing a PEP 420 namespace package #1111

1 remaining item

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

ImportWarning when importing a PEP 420 namespace package #1111

Description

Activity

bignose-debian commented on Jul 29, 2017

jdufresne commented on Aug 1, 2017

kleptog commented on Sep 14, 2018

jaraco commented on Sep 14, 2018

jaraco commented on Sep 14, 2018

jaraco commented on Sep 14, 2018

1 remaining item

jaraco commented on Sep 14, 2018

bignose-debian commented on Sep 14, 2018

jaraco commented on Sep 14, 2018

jaraco commented on Sep 14, 2018

orlnub123 commented on Sep 14, 2018

jaraco commented on Sep 14, 2018

jaraco commented on Sep 14, 2018

jaraco commented on Sep 14, 2018

jaraco commented on Sep 14, 2018

jaraco commented on Sep 14, 2018

jaraco commented on Sep 14, 2018

bignose-debian commented on Sep 15, 2018

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions