Repository navigation
DEP: should we vendorize pythoncapi-compat ? #16839
Description
This was brought up in the context of #16834 (specifically #16834 (review)):
Let me summarize the motivation here:
With the uprising of free-threading Python, part of Python's C API is now being discouraged as not thread safe. Now, alternative, thread-safe APIs exist to resolve this tension, but they are new in Python 3.13, which makes maintenance a bit cumbersome (for an example of how quickly adapting code in this context because tedious, see #16834 itself). A solution, adopted for instance by NumPy and recommended in the reference porting guide, is to vendorize pythoncapi-compat, which would allow us to use the latest C APIs without breaking build-compatibility for older Pythons.
I feel that #16834 is probably not enough justification in itself to do this, though I would expect that later developments of free-threaded Python 3.14+ might amplify this tension so it might be worth considering doing it now. Implementation wise, NumPy does this using a git submodule, and that's probably the easiest solution, but since astropy doens't have submodules at the moment, maybe we'd want to do something different: the code itself is a single C header, but comes with a Python script to upgrade it, which seems to adhere to our existing practices more closely.
I think it could live in astropy/extern.
Activity
pllim commentedon Aug 16, 2024
I agree.
While free-threading is all the rage now (in theory anyway), we should wait and see. What use is us being compatible if most upstream packages aren't or downstream doesn't care? To vendor this ourselves up the maintenance cost and would need devs who really understand the CPython C API + threading if any problem surfaces.
mhvk commentedon Aug 16, 2024
Thanks for the summary! I also feel that a bit of wait-and-see is the right approach, especially since I think #16834 showed how tricky it can be to get things right -- it seems reasonable to wait for demand, and then hope some of the people clamouring have enough skill that they can also make the changes and write tests that check that these indeed work (as is being done in numpy;
numpy/_core/tests/test_multithreading).Note that the hope for people clamouring also making PRs is not entirely wishful thinking: earlier threading/locking problems involving global state in python itself were really nicely solved that way (in ways that I would not easily have found, though I could just about review the contributions...).
Anyway, my sense is to leave issue open as a reminder, but take no action now.
neutrinoceros commentedon Aug 31, 2024
fine by me !
I just wanted to point out that the
API changelabel isn't accurate as far as I understand: this would have zero implication on the user facing API whatsoever.