Repository navigation
[C API] Private functions used by mypyc #121489
Description
Activity
_PyBytes_Join defined in pycore_bytesobject.h
You can call something like:
PyObject *empty = PyBytes_FromString(""); PyObject *res = PyObject_CallMethod(empty, "join", ...); Py_DECREF(empty);
_PyObject_CallMethodIdObjArgs defined in pycore_call.h
We are trying to get rid of _Py_IDENTIFIER() API. You should use your own mecanism for constant strings.
You can call
PyObject_CallMethodObjArgs()._PyType_CalculateMetaclass defined in pycore_object.h
For this one, I don't know. You might reimplement it, use the internal C API, or we should expose it as a public C API.
_PyUnicode_EQ defined in pycore_unicodeobject.h
You can use PyUnicode_Compare(). Maybe we should expose it.
_PyUnicode_FastCopyCharacters defined in pycore_unicodeobject.h (this could be replaced with PyUnicode_CopyCharacters but it's used in a simplified copy of _PyUnicode_JoinArray which also uses _PyUnicode_FastCopyCharacters).
Would you mind to elaborate your use case? Can you use the new https://docs.python.org/dev/c-api/unicode.html#pyunicodewriter API instead?
_PyUnicode_FastCopyCharacters defined in pycore_unicodeobject.h (this could be replaced with PyUnicode_CopyCharacters but it's used in a simplified copy of _PyUnicode_JoinArray which also uses _PyUnicode_FastCopyCharacters).
Would you mind to elaborate your use case? Can you use the new https://docs.python.org/dev/c-api/unicode.html#pyunicodewriter API instead?
It's used here https://github.com/python/mypy/blob/4e3346ee1dee83868adc2411c4a0f4050cf2f95a/mypyc/lib-rt/str_ops.c#L120 in a function which basically reimplements
_PyUnicode_JoinArray(). Guess it would work if we replaced it withPyUnicode_CopyCharacters.It's used here https://github.com/python/mypy/blob/4e3346ee1dee83868adc2411c4a0f4050cf2f95a/mypyc/lib-rt/str_ops.c#L120 in a function which basically reimplements _PyUnicode_JoinArray().
IMO PyUnicodeWriter would be a good fit for this function, especially if you can compute the exact output string length in advance.
cc @erlend-aasland @serhiy-storchaka
Reacted by Erlend E. Aasland- added a commit that references this issue
on Jul 25, 2024 @cdce8p: Python 3.13rc1 has been released. Which functions do you still need?
I believe we've now resolved all issues here. There is one open PR on the mypy side which needs to get merged but that should be it https://github.com/python/mypy/pull/17526.
What's the mypyc status?
Not strictly related to this issue. There are still two test issues with mypyc before we can support
3.13. Maybe you've an idea.- mypyc still uses some functions from
internal/pycore_*.h. I've added#define Py_CORE_BUILDwhich works. However, there seems to be an issue with linking mimalloc. This is one of the error messages.pythonsupport.his part of mypyc.
/usr/include/python3.13/internal/mimalloc/mimalloc.h:433:1: error: template with C linkage 433 | template<class T> struct _mi_stl_allocator_common { | ^~~~~~~~ In file included from /usr/include/python3.13/internal/pycore_runtime.h:17, from /usr/include/python3.13/internal/pycore_pystate.h:12, from /usr/include/python3.13/internal/pycore_call.h:12, from pythonsupport.h:21, from CPy.h:12, from test_capi.cc:5: /usr/include/python3.13/internal/pycore_interp.h:4:1: note: ‘extern "C"’ linkage started here 4 | extern "C" { | ^~~~~~~~~~ In file included from /usr/include/python3.13/internal/pycore_mimalloc.h:39, from /usr/include/python3.13/internal/pycore_interp.h:31, from /usr/include/python3.13/internal/pycore_runtime.h:17, from /usr/include/python3.13/internal/pycore_pystate.h:12, from /usr/include/python3.13/internal/pycore_call.h:12, from pythonsupport.h:21, from CPy.h:12, from test_capi.cc:5:- The other category of errors is
extension module '...' is already cached. I've bisected that to https://github.com/python/cpython/pull/118532. It might be something which can be fixed in the mypyc test runner.
Traceback (most recent call last): (diff) File "driver.py", line 1, in <module> (diff) from native import f1 (diff) File "<frozen importlib._bootstrap>", line 1360, in _find_and_load (diff) File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked (diff) File "<frozen importlib._bootstrap>", line 921, in _load_unlocked (diff) File "<frozen importlib._bootstrap>", line 819, in module_from_spec (diff) File "<frozen importlib._bootstrap>", line 782, in _init_module_attrs (diff) SystemError: extension module 'native' is already cached (diff)- mypyc still uses some functions from
SystemError: extension module 'native' is already cached (diff)
Please open a separated issue for that.
I've been working on adding support for
3.13to mypyc. After updating the codebase, there are still some functions left we depend on which are now private after #106320. Would it make sense to export these again? That would make adopting3.13a bit easier. /CC @vstinner_PyBytes_Joindefined inpycore_bytesobject.h_PyObject_CallMethodIdObjArgsdefined inpycore_call.h_PyType_CalculateMetaclassdefined inpycore_object.h_PyUnicode_EQdefined inpycore_unicodeobject.h_PyUnicode_FastCopyCharactersdefined inpycore_unicodeobject.h(this could be replaced withPyUnicode_CopyCharactersbut it's used in a simplified copy of_PyUnicode_JoinArraywhich also uses_PyUnicode_FastCopyCharacters).Other functions include these two. However, as they are inlined, it isn't a problem to include them.
_PyObject_CallMethodIdNoArgsand_PyObject_CallMethodIdOneArgdefined inpycore_call.hIt would be fine to keep all of them in
internal/pycore_*and just addPyAPI_FUNC./CC @hauntsaninja
Linked PRs