Repository navigation
Calling Py_Initialize() twice now triggers a fatal error (Python 3.7) #78113
Description
Activity
It would help to document that calling Py_Initialize() twice in Python 3.7 now triggers a fatal error, whereas previous the second call did nothing. Document it at:
https://docs.python.org/dev/whatsnew/3.7.html#changes-in-the-c-api... Or should it be considered as a regression?
--
My colleague Miro Hrončok reported a Python crash (SIGABRT) when running https://github.com/konlpy/konlpy test suite on Python 3.7:
"Fatal Python error: _Py_InitializeCore: main interpreter already initialized"
konlpy uses the JPype project, the bug is in JPype initialization function (it's a C extension):
I set the priority to release blocker to make sure that someone looks at this issue, even if I'm not sure that it's a huge regression.
The behaviour changed has been introduced by bpo-22257:
commit 1abcf67
Author: Eric Snow <[email protected]>
Date: Tue May 23 21:46:51 2017 -0700bpo-22257: Private C-API for core runtime initialization (PEP-432). (bpo-1772) (patch by Nick Coghlan)Simplified change:
- if (initialized)
-
return; - initialized = 1;
+ if (_Py_Initialized) {
+ Py_FatalError("Py_InitializeCore: main interpreter already initialized");
+ }Note: the doc doesn't say that the function must only be called once, nor document the (current) new Python 3.7 restriction (must only be called once):
https://docs.python.org/dev/c-api/init.html#c.Py_InitializeNote: the doc doesn't say that the function must only be called once, nor document the (current) new Python 3.7 restriction (must only be called once):
But the doc says:
This is a no-op when called for a second time (without calling Py_FinalizeEx() first).
no-op means it could be called twice?
While I agree the documentation issue means that this should be made to work again for Python 3.7.1, I'd like for Python 3.7 to at least deprecate calling Py_Initialize twice without an intervening Py_Finalize.
Otherwise the public multi-phase interpreter initialization API is going to have to cope with somebody starting the initialization process, and then calling a full Py_Initialize to finish it (instead of the new "complete initialization" API, whatever we end up calling that), and I'd prefer to keep the permitted state transitions more linear than that.
11 remaining items
Fixed in 3.7.0. Thanks, everyone!
I reopen the issue because fontforge still fails on Python 3.7.0 final. fontforge calls Py_Initialize() and then calls Py_Main() which fails with an assertion: https://bugzilla.redhat.com/show_bug.cgi?id=1595421
The same code works well on Python 3.6, so should we define this issue as a regression?
Call stack:
- Py_Main()
- pymain_main()
- _Py_InitializeCore() which fails
The problem is that the first call of Py_Initialize() sets a first configuration, but then Py_Main() sets a different configuration.
Is it correct to call Py_Initialize() before Py_Main()?
I looked quickly at _Py_InitializeCore() and I'm not sure that it's designed to replace an existing configuration.
Example:
int Py_HashRandomizationFlag = 0; /* for -R and PYTHONHASHSEED */
(...)
if (!core_config->use_hash_seed || core_config->hash_seed) { /* Random or non-zero hash seed */ Py_HashRandomizationFlag = 1; }
If Py_Initialize() sets Py_HashRandomizationFlag to 1, but Py_Main() doesn't use an hash seed, Py_HashRandomizationFlag value remains 1 which is wrong.
pymain_read_conf() fills _PyCoreConfig from global configuration flags, but then global configuration flags are supposed to be set from the global configuration flags.
So maybe for this specific case, _Py_InitializeCore() should always set Py_HashRandomizationFlag. But it was just one example to show that right now, _Py_InitializeCore() doesn't seem to support to be called twice with two different configurations.
Ok, now with my regression hat.
In Python 3.6, Py_Main() calls Py_Initialize(). If Py_Initialize() is called twice, the second call does nothing. So it's fine to call Py_Main() after Py_Initialize() in Python 3.6.
I looked quickly at _Py_InitializeCore() and I'm not sure
that it's designed to replace an existing configuration.
...
So maybe for this specific case, _Py_InitializeCore() should
always set Py_HashRandomizationFlag.+1
At the same time, the use of Py_*Flag variables is a little unclear in the PEP-432 world. Under the PEP I'd expect code in CPython to use the core config from the current PyInterpreterState (or even _PyRuntimeState), rather than the global flag variables. So presumably (the PEP doesn't talk about it) the intent of keeping them is to help folks that currently make use of the flags. However, they shouldn't be modifying them after initialization, and under PEP-432 they don't need to set them beforehand (they set values on the config), right?
Part of the problem here is that Py_Main() results in all the flag variables getting set (before runtime initialization). However, with the current PEP-432 [internal] implementation those variables are (mostly) never getting set, right? Shouldn't we set all of them, but only at the end of _Py_InitializeCore() (for the sake of extensions/embedders that use them)? In that case we'd need to replace our internal use of Py_HashRandomizationFlag with field on the config struct on _PyRuntimeState.
But it was just one example to show that right now,
_Py_InitializeCore() doesn't seem to support to be called
twice with two different configurations.Are there other things (than how we set that flag) that break in that case? Regardless, what would ensure that it works properly?
Victor, could you file a separate issue for Py_Main(). It's a different problem, since Py_Main() is an entry point for an *application that embeds Python*. It's not a CPython API in the tradition sense.
I filed https://bugs.python.org/issue34008 to cover the fact that "Py_Initialize(); Py_Main();" doesn't work in 3.7.0 whereas it used to sort of work (by only partially applying the settings read from the CLI and environment by the Py_Main() call).
Since "Py_Initialize(); Py_Initialize();" *does* work in 3.7.0, closing this one again.
Answering Eric's question about the flag variables though:
- pymain_get_global_config copies the global variable flags to their respective struct entries
- pymain_set_global_config copies the various struct entries to their respective global variable flags
One of the first things that Py_Main() does in 3.7 is to:
- read the global variable flags into the relevant structs
- update the structs based on the CLI options
- write the struct values back to the global flags
This mostly fakes the 3.6-and-earlier behaviour of using the global variables directly.
For anything listed in https://docs.python.org/3/c-api/init.html#global-configuration-variables, we also mostly left any code reading the global variable directly alone, since they're considered a public API for embedding applications to dynamically adjust behaviour and we don't have a good way of deprecating using them for that purpose :(
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields: