Activity

1. FFY00

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Nov 13, 2024
2. zware commented on Nov 13, 2024

   

   zware

   on Nov 13, 2024

   Member

   More actions

   I'm not sure I agree with the assertion that purelib and platlib should be considered unavailable when running under -S. All that should mean is that they're not actually added to sys.path automatically, not that they're totally off limits. In fact, there's nothing stopping one from running python3 -i -S -c 'import site; site.main()' as a weird way to start up a regular REPL with no headers :)
3. FFY00 commented on Nov 13, 2024

   

   FFY00

   on Nov 13, 2024

   MemberAuthor

   More actions

   I would agree with that if the -S flag was as simple as you describe it, but it isn't 😅

   What makes things messy is that on top of the customizations you mentioned, the site module is also responsible for implementing virtual environments. Disabling the site initialization also disables the virtual environment activation mechanism.
   If we were to think conceptually of an environment resultant of -S simply as a "locked down" version of another existing environment, then that existing environment would be the installation-wide environment (eg. /usr/lib/python3.14t/site-packages), rather than the virtual environment (eg. ~/.virtualenvs/test-sysconfig-paths/lib/python3.14t/site-packages).

   In practice, this means that python -S -m ensurepip, python -S -m pip install ..., or other commands that customize a "fully-fledged" environment would be acting on the installation-wide environment, even when running these commands inside a virtual environment. IMO this is a major footgun and, to me, is an undesirable behavior left over from of historic implementation decisions.

   I think we should think conceptually of -S more as an environment-less way of running Python, than a limited environment.
   I am happy to hear other points of view though.
4. picnixz

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   on Nov 14, 2024
5. FFY00

   added

   topic-sysconfig

   on Nov 16, 2024
6. zooba commented on Nov 18, 2024

   

   zooba

   on Nov 18, 2024

   Member

   More actions

   I think we should think conceptually of -S more as an environment-less way of running Python, than a limited environment.

   This sounds reasonable to me, however...

   Disabling the site initialization also disables the virtual environment activation mechanism.

   Not entirely (a lot is handled in getpath, IIRC independent of the -S option), but if the above definition sticks, then it should disable it entirely. Effectively you should only get the stdlib (and any explicit additions), regardless of whether you launch through a venv or not. And so...

   this means that python -S -m ensurepip, python -S -m pip install ..., or other commands that customize a "fully-fledged" environment would be acting on the installation-wide environment, even when running these commands inside a virtual environment

   Here's where it gets complicated. There's strictly nothing wrong with ensurepip or pip install always installing into the default "site", regardless of whether it's active or not, but that is going to mean something weird if we don't clearly define what launching a venv with -S means. Hopefully we can define that in a way that makes it consistent and doesn't actually change existing behaviour.

   Perhaps this works:

   • launching a venv overrides the default site directory
   • site.main() adds the default site directory (and etc...) when called
   • -S skips calling site.main()
   • [ensure]pip installs to the default site directory regardless of whether it's active or not

   I'm sure there are new edge cases introduced by a definition like that (if it works at all, I haven't thought it all the way through yet), but perhaps those are easier to fix/handle?
7. FFY00 commented on Nov 18, 2024

   

   FFY00

   on Nov 18, 2024

   MemberAuthor

   More actions

   Not entirely (a lot is handled in getpath, IIRC independent of the -S option), but if the above definition sticks, then it should disable it entirely. Effectively you should only get the stdlib (and any explicit additions), regardless of whether you launch through a venv or not. And so...

   (I realized I went a bit off track when writing the reply below, but I am keeping it since it's useful)

   Right, I am aware that getpath has some code that covers virtual environments, but AFAIK nothing that changes any other aspects of the initialization, at least not in any meaningful way anymore. That may have been the case in the past, but the only code left now is for finding sys._base_executable in niche situations where it is unknown.

   Here's my understanding of what happens:

   • getpath looks for a pyvenv.cfg (the indicator of a virtual environment) near the executable or, if unknown, the current directory
   • If base_executable from PyConfig is not set a home key is specified in pyvenv.cfg, it searches for the interpreter executable there and sets it base_executable to its path if found
   • If no process executable was provided, base_executable is used instead when looking for the ._pth file
   • sys set sys._base_executable to the value in PyConfig

   So, there may be a possibility for sys._base_executable to be set to an interpreter from a virtual environment, though this looks like a very niche case, and am not sure if there's are any relevant real world use-case.
   Even in that possibility, it only seems to affect ._pth file results in -E and -S, I don't think it's relevant here.
   Considering this,

   If there's anything else that I am missing, let me know.

   Here's where it gets complicated. There's strictly nothing wrong with ensurepip or pip install always installing into the default "site", regardless of whether it's active or not, but that is going to mean something weird if we don't clearly define what launching a venv with -S means. Hopefully we can define that in a way that makes it consistent and doesn't actually change existing behaviour.

   Perhaps this works:

   * launching a venv overrides the default site directory
   
   * `site.main()` adds the default site directory (and etc...) when called
   
   * `-S` skips calling `site.main()`
   
   * `[ensure]pip` installs to the default site directory regardless of whether it's active or not
   

   I'm sure there are new edge cases introduced by a definition like that (if it works at all, I haven't thought it all the way through yet), but perhaps those are easier to fix/handle?

   I think what we are running into here is the fact that virtual environments probably shouldn't be implemented by the site module. IMO, the base mechanism (prefix relocation via pyvenv.cfg) should be provided as a core feature, and site could still keep being responsible for the environment customization.

   My proposal to move forward:

   • Start setting prefix and base_prefix to the pyvenv.cfg directory in getpath again
   • Document the pyvenv.cfg detection as part of the interpreter initialization, instead of part of the site import
   • Update the all other documentation to reflect this change
8. zooba commented on Nov 18, 2024

   

   zooba

   on Nov 18, 2024

   Member

   More actions

   Even in that possibility, it only seems to affect ._pth file results in -E and -S, I don't think it's relevant here.

   FWIW, the mere presence of a ._pth file implies both -E and -S (and an import site in the ._pth file removes -S again).
9. FFY00 commented on Nov 26, 2024

   

   FFY00

   on Nov 26, 2024

   MemberAuthor

   More actions

   This is no longer relevant, as GH-126985 made virtual environments no longer dependent on the site module.
10. FFY00

    closed this as completedon Nov 26, 2024