Using the limited Python API in astropy

When building a Python package with C extensions, it is possible to build it against only the limited Python C API (more details at https://docs.python.org/3/c-api/stable.html), which is an API that does not change across minor versions of Python. This makes it possible to build a wheel on e.g. 3.11 and have it also work on 3.12 and 3.13. As an example, Sunpy uses this, and they only have to provide a few wheels, one per platform/architecture:

https://pypi.org/project/sunpy/6.1.1/

Extension-helpers will soon include a change that makes it possible to easily opt in to using this by just adding a single configuration item in pyproject.toml, namely:

[tool.distutils.bdist_wheel]
py-limited-api = "cp311"

This also requires using Cython 3.1.

Currently this does not work out of the box for astropy core, and there are a few errors we need to resolve:

column_mixins.pyx

It looks like Cython support may not be complete here, we should try and see if we can make a minimal example of the failure here and report it upstream.

x86_64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -DPy_LIMITED_API=0x030B0000 -DNPY_TARGET_VERSION=NPY_1_23_API_VERSION -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/home/tom/python/dev/lib/python3.11/site-packages/numpy/_core/include -I/home/tom/python/dev/include -I/usr/include/python3.11 -c astropy/table/_column_mixins.c -o build/temp.linux-x86_64-cpython-311/astropy/table/_column_mixins.o
astropy/table/_column_mixins.c: In function ‘__pyx_f_7astropy_5table_14_column_mixins_column_getitem’:
astropy/table/_column_mixins.c:5217:80: error: invalid use of incomplete typedef ‘PyTypeObject’ {aka ‘struct _typeobject’}
 5217 |   __pyx_t_1 = ((PyTypeObject *)__pyx_mstate_global->__pyx_ptype_5numpy_ndarray)->tp_as_mapping->mp_subscript(__pyx_v_self, __pyx_v_item); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 74, __pyx_L1_error)
      |                                                                                ^~
error: command '/usr/lib/ccache/x86_64-linux-gnu-gcc' failed with exit code 1

storage size unknown

There are several similar errors:

astropy/utils/xml/src/iterparse.c: At top level:
astropy/utils/xml/src/iterparse.c:1105:21: error: storage size of ‘IterParserType’ isn’t known
 1105 | static PyTypeObject IterParserType =
      |                     ^~~~~~~~~~~~~~
error: command '/usr/lib/ccache/x86_64-linux-gnu-gcc' failed with exit code 1

astropy/wcs/src/astropy_wcs.c:753:21: error: storage size of ‘WcsType’ isn’t known
  753 | static PyTypeObject WcsType = {
      |                     ^~~~~~~
astropy/wcs/src/astropy_wcs.c:753:21: error: storage size of ‘WcsType’ isn’t known
error: command '/usr/lib/ccache/x86_64-linux-gnu-gcc' failed with exit code 1

astropy/cosmology/_src/signature_deprecations.c: At top level:
astropy/cosmology/_src/signature_deprecations.c:200:21: error: storage size of ‘DeprKwsWrap’ isn’t known
  200 | static PyTypeObject DeprKwsWrap = {
      |                     ^~~~~~~~~~~
cc1: all warnings being treated as errors
error: command '/usr/lib/ccache/x86_64-linux-gnu-gcc' failed with exit code 1

Aside from these issues, everything builds fine, so it's not completely impossible to get there!

Need to investigate further what this is due to (similar to astropy.utils error above)

Linked PRs

astropy/table/_column_mixins.c is generated from the pyx, so that is extra layer to wade through for debugging. 🤪

Could the debugger be more specific on exactly what in this one isn't defined properly?

astropy/astropy/wcs/src/astropy_wcs.c

Line 753 in f442813

static PyTypeObject WcsType = {

For now I just wanted to make sure I opened an issue to keep track of known problems, but I'll try and dig deeper into this.

Just a note about the failure in _column_mixins.pyx - the problematic line is:

    return (<PyTypeObject *>np.ndarray).tp_as_mapping.mp_subscript(self, item)

If we use only the limited API, we can't call methods on PyTypeObject directly (it is an opaque struct). I'm not sure how we can work around this though.

Extension-helpers will soon include a change that makes it possible to easily opt in to using this by just adding a single configuration item in pyproject.toml

Any idea how this option will behave in free-threaded Python (#16916) ?
At the moment, Limited API is only supported for GIL-enabled builds, so most build systems I know of (meson, maturin) will just ignore the option in a free-threaded build.
To paraphrase Sam Gross' talk at PyConUS 2025, the current goal is to define a new stable ABI (abi4) for Python 3.15 (oct 2026) that will work with both builds (compile once, install in against GIL-enabled or free-threaded Python).

I wanted to take a shot a fixing the existing errors, and to my surprise, I don't seem to get the same ones your reported.
I do get the 3 following errors (alongside 10 warnings that I'll skip for brevity):

(...)
astropy/wcs/src/astropy_wcs.c:94:16: error: incomplete definition of type 'PyTypeObject' (aka 'struct _typeobject')
   94 |   Py_TYPE(self)->tp_free((PyObject*)self);
      |   ~~~~~~~~~~~~~^
/Users/clm/.local/share/uv/python/cpython-3.11.12-macos-aarch64-none/include/python3.11/pytypedefs.h:20:16: note: forward declaration of 'struct _typeobject'
   20 | typedef struct _typeobject PyTypeObject;
      |                ^
astropy/wcs/src/astropy_wcs.c:104:20: error: incomplete definition of type 'PyTypeObject' (aka 'struct _typeobject')
  104 |   self = (Wcs*)type->tp_alloc(type, 0);
      |                ~~~~^
/Users/clm/.local/share/uv/python/cpython-3.11.12-macos-aarch64-none/include/python3.11/pytypedefs.h:20:16: note: forward declaration of 'struct _typeobject'
   20 | typedef struct _typeobject PyTypeObject;
      |                ^

(...)
astropy/wcs/src/astropy_wcs.c:753:21: error: variable has incomplete type 'PyTypeObject' (aka 'struct _typeobject')
  753 | static PyTypeObject WcsType = {
      |                     ^
/Users/clm/.local/share/uv/python/cpython-3.11.12-macos-aarch64-none/include/python3.11/pytypedefs.h:20:16: note: forward declaration of 'struct _typeobject'
   20 | typedef struct _typeobject PyTypeObject;
      |                ^

All these errors live in astropy_wcs.c, note in Cython code.

Also noting as I start digging:
the root issue in astropy_wcs.c appears to be that we rely on implementation details of the PyTypeObject struct, but it's actually only available, through the limited API, as an opaque struct. I'm not sure yet how to refactor these details out. More inquiry required.

Yes I realized since that the errors I posted above were not the main errors, and other errors happened higher up.

Just to make sure we don't duplicate effort, I started looking into the astropy.wcs issues and I made progress on fixing the errors but it's a bit tedious as quite a few files need fixing. Nevertheless I think it should be possible to fix. For now, the trickiest ones to fix as far as I can tell are the column_mixin.pyx one and the cosmology one, and I was a bit stuck trying to fix them.

In case it's useful, I can try and push my changes for astropy.wcs shortly.

Definitely useful. And thanks for the heads up !

Just a note about the failure in _column_mixins.pyx - the problematic line is:
    return (<PyTypeObject *>np.ndarray).tp_as_mapping.mp_subscript(self, item)
If we use only the limited API, we can't call methods on PyTypeObject directly (it is an opaque struct). I'm not sure how we can work around this though.
Possibly it is time to check if this mixin is still necessary - it replaced a python overwrite because of speed, but perhaps things are better now... Alternatively, would have to see if just using Python C getitem would work.

The other option is PyType_GetSlot but be very wary of the behaviour change before 3.10 (it'll just crash though, so you'll find out pretty quickly if it's an issue).

Oh great, thanks for pointing it out !
the change in 3.10 won't affect us because we already require 3.11+

@da-woods

@da-woods I tried the following patch:

diff --git a/astropy/table/_column_mixins.pyx b/astropy/table/_column_mixins.pyx
index 5ab4fe66d3..f9227e533d 100644
--- a/astropy/table/_column_mixins.pyx
+++ b/astropy/table/_column_mixins.pyx
@@ -26,7 +26,7 @@ import sys
 
 import numpy as np
 cimport numpy as np
-
+from cpython.ref cimport PyTypeObject
 np.import_array()
 
 
@@ -44,8 +44,9 @@ cdef extern from "Python.h":
     ctypedef struct PyMappingMethods:
         binaryfunc mp_subscript
 
-    ctypedef struct PyTypeObject:
-        PyMappingMethods* tp_as_mapping
+    void *PyType_GetSlot(PyTypeObject *type, int slot)
+
+    int Py_tp_as_mapping
 
 
 ctypedef object (*item_getter)(object, object)
@@ -71,7 +72,7 @@ cdef inline object base_getitem(object self, object item, item_getter getitem):
 
 
 cdef inline object column_getitem(object self, object item):
-    return (<PyTypeObject *>np.ndarray).tp_as_mapping.mp_subscript(self, item)
+    return (<PyMappingMethods *>PyType_GetSlot(<PyTypeObject *>np.ndarray, Py_tp_as_mapping)).mp_subscript(self, item)

but compilation fails with

astropy/table/_column_mixins.c:5195:118: error: use of undeclared identifier 'Py_tp_as_mapping'
 5195 |   __pyx_t_1 = ((PyMappingMethods *)PyType_GetSlot(((PyTypeObject *)__pyx_mstate_global->__pyx_ptype_5numpy_ndarray), Py_tp_as_mapping))->mp_subscript(__pyx_v_self, __pyx_v_item); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 75, __pyx_L1_error)
      |                                                                                                                      ^
1 error generated

I'm guessing I'm not doing it right. What's the correct way to include a constant like Py_tp_as_mapping from Python.h in Cython ?

The slot is Py_mp_subscript. You don't need to care that it's in as_mapping

For the others e.g.

static PyTypeObject WcsType = {

you'll need to convert them into heaptypes (i.e. define them in a PyType_Spec and then call PyType_FromSpec. Unfortunately there doesn't seem to be a hugely good tutorial on this, so you might be best browsing the Python tests for how to do it: https://github.com/python/cpython/blob/main/Modules/_testcapi/heaptype.c

@astrofrog

@astrofrog I took the liberty to add a "Linked PRs" section to the issue.

Thank you!

@neutrinoceros

@neutrinoceros - see #18236 for another PR, which I've added to the list.

I think all refactor PRs are now stable and up for review !
once they're all in, we can start infrastructural changes to actually leverage abi3 wheels in the release process.

@neutrinoceros

@neutrinoceros - thank you for all your work on this!

We're getting there ! #18231 is the last remaining open PR to get limited-api compliance !

Navigation Menu

Repository navigation

Using the limited Python API in astropy #18163

Linked PRs

1 remaining item

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Using the limited Python API in astropy #18163

Description

Linked PRs

Activity

pllim commented on May 20, 2025

pllim commented on May 20, 2025

astrofrog commented on May 20, 2025

astrofrog commented on May 20, 2025

neutrinoceros commented on May 22, 2025

neutrinoceros commented on May 29, 2025

neutrinoceros commented on May 29, 2025

astrofrog commented on May 29, 2025

neutrinoceros commented on May 30, 2025

1 remaining item

da-woods commented on Jun 6, 2025

neutrinoceros commented on Jun 6, 2025

neutrinoceros commented on Jun 6, 2025

da-woods commented on Jun 6, 2025

da-woods commented on Jun 6, 2025

neutrinoceros commented on Jun 9, 2025

astrofrog commented on Jun 9, 2025

astrofrog commented on Jun 9, 2025

neutrinoceros commented on Jun 13, 2025

astrofrog commented on Jun 13, 2025

neutrinoceros commented on Jul 7, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions