Skip to content

sysconfig data inconsistencies caused by the disabling the site initialization on virtual environments #126789

Closed
Closed
sysconfig data inconsistencies caused by the disabling the site initialization on virtual environments#126789
Labels
topic-sysconfigtype-bugAn unexpected behavior, bug, or error

Description

@FFY00
FFY00
opened on Nov 13, 2024

Bug report

Bug description:

Currently, running the Python interpreter with -S (disabling the site module initialization) causes several inconsistencies sysconfig.get_paths() depends on whether sysconfig was imported before or after the site initialization

Currently, sysconfig calculates the paths at import time and caches them. This results in it resulting incorrect paths if the site initialization happens after sysconfig was imported.

$ python -S
Python 3.14.0a1+ experimental free-threading build (heads/main:de0d5c6e2e1, Oct 23 2024, 15:37:46) [GCC 14.2.1 20240910] on linux
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14t', 'platstdlib': '/usr/local/lib/python3.14t', 'purelib': '/usr/local/lib/python3.14t/site-packages', 'platlib': '/usr/local/lib/python3.14t/site-packages', 'include': '/usr/local/include/python3.14td', 'platinclude': '/usr/local/include/python3.14td', 'scripts': '/usr/local/bin', 'data': '/usr/local'}
>>> import site
>>> site.main()
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14t', 'platstdlib': '/usr/local/lib/python3.14t', 'purelib': '/usr/local/lib/python3.14t/site-packages', 'platlib': '/usr/local/lib/python3.14t/site-packages', 'include': '/usr/local/include/python3.14td', 'platinclude': '/usr/local/include/python3.14td', 'scripts': '/usr/local/bin', 'data': '/usr/local'}

Expected output when the site module has been initialized:

$ python
Python 3.14.0a1+ experimental free-threading build (heads/main:de0d5c6e2e1, Oct 23 2024, 15:37:46) [GCC 14.2.1 20240910] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14t', 'platstdlib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t', 'purelib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t/site-packages', 'platlib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t/site-packages', 'include': '/usr/local/include/python3.14td', 'platinclude': '/usr/local/include/python3.14td', 'scripts': '/home/anubis/.virtualenvs/test-sysconfig-paths/bin', 'data': '/home/anubis/.virtualenvs/test-sysconfig-paths'}

And just to make sure this is not dependent on the interpreter being run with -S, and rather on the sysconfig import time.

$ python -S
Python 3.14.0a1+ experimental free-threading build (heads/main:de0d5c6e2e1, Oct 23 2024, 15:37:46) [GCC 14.2.1 20240910] on linux
>>> import site
>>> site.main()
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14t', 'platstdlib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t', 'purelib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t/site-packages', 'platlib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t/site-packages', 'include': '/usr/local/include/python3.14td', 'platinclude': '/usr/local/include/python3.14td', 'scripts': '/home/anubis/.virtualenvs/test-sysconfig-paths/bin', 'data': '/home/anubis/.virtualenvs/test-sysconfig-paths'}

CPython versions tested on:

3.9, 3.10, 3.11, 3.12, 3.13, 3.14

Operating systems tested on:

Linux

Linked PRs

Activity

  1. FFY00
    added
    type-bugAn unexpected behavior, bug, or error
    on Nov 13, 2024

  2. FFY00 commented on Nov 13, 2024

    @FFY00
    FFY00
    on Nov 13, 2024
    MemberAuthor

    Things are a bit messy, as the virtual environment mechanism is technically part of the site customization. In reality, the pyvenv.cfg detection is implemented on the interpreter initialization, but the customization itself is done by site. I feel like this should probably be reconsidered.

    Anyway, to match this model, if site has been initialized, sysconfig.get_paths() should always return the virtual environment paths. So, the correct result should be as follows:

    $ python -S
Python 3.14.0a1+ experimental free-threading build (heads/main:de0d5c6e2e1, Oct 23 2024, 15:37:46) [GCC 14.2.1 20240910] on linux
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14t', 'platstdlib': '/usr/local/lib/python3.14t', 'purelib': '/usr/local/lib/python3.14t/site-packages', 'platlib': '/usr/local/lib/python3.14t/site-packages', 'include': '/usr/local/include/python3.14td', 'platinclude': '/usr/local/include/python3.14td', 'scripts': '/usr/local/bin', 'data': '/usr/local'}
>>> import site
>>> site.main()
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.14t', 'platstdlib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t', 'purelib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t/site-packages', 'platlib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t/site-packages', 'include': '/usr/local/include/python3.14td', 'platinclude': '/usr/local/include/python3.14td', 'scripts': '/home/anubis/.virtualenvs/test-sysconfig-paths/bin', 'data': '/home/anubis/.virtualenvs/test-sysconfig-paths'}

  3. FFY00 commented on Nov 13, 2024

    @FFY00
    FFY00
    on Nov 13, 2024
    MemberAuthor

    Correction: sysconfig does not cache the sysconfig.get_paths(), it caches sysconfig.get_config_vars(), which is used to calculate the paths.

  4. FFY00
    added a commit that references this issue on Nov 13, 2024

    pythonGH-126789: fix some sysconfig data on late site initializations

    06356b0
  5. FFY00
    changed the title [-]Fix `sysconfig.get_paths()` inconsistencies caused by the disabling the `site` initialization on virtual environments[/-] [+]`sysconfig` data inconsistencies caused by the disabling the `site` initialization on virtual environments[/+] on Nov 13, 2024
  6. FFY00
    added
    topic-sysconfig
    on Nov 13, 2024

  7. FFY00 commented on Nov 13, 2024

    @FFY00
    FFY00
    on Nov 13, 2024
    MemberAuthor

    While fixing the main bug, I have noticed that the prefix and exec_prefix variables get overwritten by _init_posix, so on virtual environments sysconfig.get_config_vars('prefix') and sys.prefix will have different values, the same goes for exec_prefix.

    IMO this is worth to fix, but we probably shouldn't backport it as it is a significant behavior change, and there's a possibility it would break existing user code.

  8. FFY00 commented on Nov 13, 2024

    @FFY00
    FFY00
    on Nov 13, 2024
    MemberAuthor

    Correction: sysconfig does not cache the sysconfig.get_paths(), it caches sysconfig.get_config_vars(), which is used to calculate the paths.

    Just some more background on this, sysconfig.get_paths() uses an incomplete copy of the config vars directory to expand the path placeholder, which is what threw me off initially. This makes it a little bit more tricky to fix the issue, as we need to make sure we update the config vars cache without creating a recursive loop, but wasn't too difficult to get the implementation right.

  9. FFY00
    added a commit that references this issue on Nov 17, 2024

    GH-126789: fix some sysconfig data on late site initializations

    acbd5c9
  10. miss-islington
    added 2 commits that reference this issue on Nov 17, 2024

    pythonGH-126789: fix some sysconfig data on late site initializations

    a3e0c59

    pythonGH-126789: fix some sysconfig data on late site initializations

    d4f985b
  11. FFY00
    added 2 commits that reference this issue on Nov 17, 2024

    [3.12] GH-126789: fix some sysconfig data on late site initializations

    50e42b9

    [3.13] GH-126789: fix some sysconfig data on late site initializations

    ec9cbcb
  12. FFY00
    closed this as completedon Nov 17, 2024
  13. merlinz01
    added a commit that references this issue on Nov 17, 2024

    pythonGH-126789: fix some sysconfig data on late site initializations

    002d692

  14. 5 remaining items

  15. encukou
    added a commit that references this issue on Nov 18, 2024

    [3.13] gh-126789: Correct sysconfig test exclusions for iOS and Andro…

    da7e93d
  16. FFY00
    added 2 commits that reference this issue on Nov 19, 2024

    Revert "pythonGH-126789: fix some sysconfig data on late site initial…

    9cc5c44

    fixup! Revert "pythonGH-126789: fix some sysconfig data on late site …

    fa3adba
  17. encukou
    added a commit that references this issue on Nov 19, 2024

    [3.12] gh-126789: Correct sysconfig test exclusions for iOS and Andro…

    1f911ba

  18. FFY00 commented on Nov 20, 2024

    @FFY00
    FFY00
    on Nov 20, 2024
    MemberAuthor

    @hugovk, sorry I just say this now. I should have mentioned it in the issue text, but as per the title this only affects virtual environments.

    The issue is that currently site is responsible for the virtual environment initialization (setting sys.prefix and sys.exec_prefix), so when you run a virtual environment with -S, sys.prefix and sys.exec_prefix point to the base installation, meaning that, canonically, you are not in a virtual environment. In site.main(), sys.prefix and sys.exec_prefix are updated to point to the virtual environment.
    If the sysconfig.get_config_vars() cache, which is used when calculating sysconfig.get_paths(), is created before the site initialization, it still holds the old prefix values, resulting in sysconfig reporting different paths depending on whether the cache was generated before or after the site initialization.
    The issue only affects virtual environments, because that's the only case where sys.prefix and sys.exec_prefix change after initialization.

    I hope that helps clarify things. Let me know if you have any more questions.

  19. hugovk commented on Nov 20, 2024

    @hugovk
    hugovk
    on Nov 20, 2024
    Member

    Thanks for the clarification!

  20. thesamesam commented on Dec 6, 2024

    @thesamesam
    thesamesam
    on Dec 6, 2024
    Contributor

    The change for this seems to have regressed building Firefox with Python 3.12.8 and Python 3.13.1: https://bugzilla.mozilla.org/show_bug.cgi?id=1935621

    I don't know enough about Firefox's build system, mach, to know if it is an issue on their side or not.

  21. FFY00 commented on Dec 7, 2024

    @FFY00
    FFY00
    on Dec 7, 2024
    MemberAuthor

    @thesamesam, I investigated the issue, the source of the issue is that they are overwriting sys.prefix and were relying on sysconfig data using the original prefix. GH-126812 fixed sysconfig data not matching sys.prefix/sys.exec_prefix if they were updated after import. sysconfig used to cache their values at import, so if they were updated afterwards, the data wouldn't reflect the change.

    In GH-126987, we moved the virtual environment activation from site to the interpreter path initialization, which means that the reproducible above would no longer trigger the sysconfig cache issue, as sys.prefix/sys.exec_prefix are no longer updated by the site module. Because of this, I also reverted the cache bugfix, but seeing that some folks are updating sys.prefix manually, I think we should re-apply the bugfix.

  22. FFY00
    added 2 commits that reference this issue on Dec 7, 2024

    pythonGH-126789: fix some sysconfig data on late site initializations

    e9d8a6d

    GH-126789: fix some sysconfig data on late site initializations (#127729

    7015485
  23. ziyao233
    added a commit that references this issue on Dec 19, 2024

    [firefox] 132.0.2-3: fix build with python 3.13

    1bcff3b
  24. hack3ric
    added a commit that references this issue on Dec 20, 2024

    [firefox] 132.0.2-3: fix build with python 3.13

    224a26c
  25. srinivasreddy
    added a commit that references this issue on Jan 8, 2025

    pythonGH-126789: fix some sysconfig data on late site initializations (…

    2ec336b
  26. ebonnal
    added 2 commits that reference this issue on Jan 12, 2025

    pythonGH-126789: fix some sysconfig data on late site initializations

    71f5423

    pythongh-126789: Correct sysconfig test exclusions for iOS and Androi…

    5b5c194
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    topic-sysconfigtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions