Repository navigation
Decide how to backport 3.7 async context management support #12
Description
Python 3.7 added async/await support to the contextlib module (e.g. asyncontextmanager and AsyncExitStack), and then Python 3.10 added more (e.g. aclosing).
This raised the question of how to backport those features to 3.6+ (the first version with support for yield inside async def).
The answer turned out to be "procrastinate on the question for 4+ years, so it became feasible to simply drop support for Python 3.5 and earlier, and use the async generator code as-is".
(Issue description updated Jun 2021 to describe what actually happened, rather than the more complicated alternatives I was considering to allow adding the new async features without dropping support for Python 3.5 and earlier)
Activity
sashgorokhov commentedon Mar 8, 2017
Can't imagine how you can manage this to work on 3.5 without yield support inside coroutines. I'll subscribe to this thread to see any useful ideas for that.
sashgorokhov commentedon Mar 8, 2017
But supporting at least 3.6 would be great. I am already using personal package for this, and could provide a PR for
asynccontextmanagerJelleZijlstra commentedon Mar 8, 2017
Like Nick, I'd prefer keeping a single package. Having fewer packages with similar purposes makes it easier for people to decide what package to install.
The implementation would probably involve making contextlib2 into a package, organized something like this:
__init__.pydoesfrom _contextlib import *andif sys.version >= (3, 6): from _contextlib_async import asynccontextmanager. Thus, _contextlib_async would only be imported on Python versions where it's syntactically valid.Yes, this probably can't be backported to 3.5 (maybe using https://github.com/njsmith/async_generator would work?). However, other future async enhancements to contextmanager, like AsyncExitStack, could be backported to 3.5 (and even earlier if you want to use yield from).
sashgorokhov commentedon Mar 8, 2017
Using another libraries with their own logic (
yield_) will make it a pain in python 3.7 to migrate from this package to a standart one. Also, it won't be a clear solution for end-usersncoghlan commentedon Mar 8, 2017
For now, I've reworded the issue to just target 3.6+. However, we can keep the possibility of supporting earlier versions in mind as we consider possible solutions.
the first option would be something like @JelleZijlstra suggested, only with the "minimum compatible version" files named after the earliest version that can import them (
_contextlib_needs35.py,_contextlib_needs36.py, etc). I think this is the cleanest approach from an install time and runtime perspective, but the trickiest from the point of view of actually merging changes from the stdlib into the backport (but, as noted, we don't need to do that very often)the second option that comes to mind would be to include full copies of the versions of interest (so
contextlib2._contextlib_37.pywould be an actual pre-release copy of the 3.7 version of the module). Aside from the inevitable code duplication, the main argument against this is that it wouldn't help with cases likeAsyncExitStack, where that's likely to be compatible with 3.5, even thoughasynccontextmanagerwill need 3.6. That's enough of a downside that I don't think any simplification of future maintenance could make it worthwhile to accept as a limitationnjsmith commentedon Jun 15, 2017
FWIW, I use a home grown version of this all the time on 3.5, via https://github.com/njsmith/async_generator. For example,
trio.open_nurseryis implemented this way. Theasynccontextmanagercode itself doesn't need to define async generators, so there's no trouble importing it on 3.5; it just depends on the user to provide something that quacks like an async generator function, and it's up to them to decide if they want to useasync_generatoror whatever to make that function. (Why do I care about 3.5 support? PyPy.)This code is sometimes a bit annoying to maintain, so I'd be happy to delegate to contextlib2 :-).
OTOH there might be a few challenges. The homegrown changes in my version are:
Nice error message if someone accidentally uses
withwhen they meantasync with: Clearer error message if async context manager used synchronously python-trio/trio#212Adding some metadata so that functions returning async generators are recognizable: https://github.com/python-trio/trio/blob/e4c8ce7b81669f3e886c2ee434c7249aba7f55ed/trio/_util.py#L147-L148 (see bpo-30359, though here there's the additional wrinkle that AFAIK we don't yet have a way to express "async context manager" in
__annotations__)(Not yet but someday) Right now, there's a race condition in signal handling when exiting from an async context manager: if control-C arrives just as we're entering the
__aexit__method, it causes it not to be run. In general trio protects from this using some complicated annotations, but I haven't bothered trying to solve this race condition yet, because even if I fixed it here then essentially the same race condition exists in the interpreter itself, so what's the point really. But if bpo-29988 got fixed then I would be motivated to figure out how to fix the part in the context manager as well, and it might be trickier if it lives in another library. Of course,contextmanagerhas the same problem, and I can probably solve it using some disgusting hack like noticing that all(async)contextmanagerobjects use the samecodeobjects for their__(a)exit__methods.I don't think these are showstoppers, just figured put them on the radar.
sorcio commentedon May 7, 2018
FYI, since the only piece I was missing from async_generator was an
AsyncExitStackand since I only need Python 3.6, I packaged a "backport" for 3.5+ (more or less a verbatim copy from 3.7) here https://github.com/sorcio/async_exit_stackI'd be happy to help with a proper backport.
ncoghlan commentedon Jun 8, 2018
I think between @sorcio's
async_exit_stackmodule and the simple passage of time, the idea of copying in the Python 3.7 version of the modules ascontextlib2._contextlib_36_compat.py, moving the current code that implements stdlib APIs tocontextlib2._contextlib_26_compat.py, and then populatingcontextlib2.__init__.pywith code to choose the newest importable option and then add the extracontextlib2specific backwards compatibility code is a lot more viable now than it was when I first opened this issue.While there a few APIs it would be nice to extract and make universally available, that could be considered as a separate issue after the initial 3.6+ option was implemented.
ncoghlan commentedon Sep 21, 2019
With @jayvdb submitting the synchronous enhancements as a separate PR (which I'm about to release as 0.6.0) that opens up a new alternative: what if contextlib2 0.7.0 were to just drop support for all versions prior to 3.6, leaving users of old Python versions on 0.6.0 indefinitely?
graingert commentedon Sep 16, 2020
@ncoghlan does the Python 3.5 EOL change anything?
ncoghlan commentedon Jun 23, 2021
@graingert It makes me even happier with the idea of 0.7.0 being 3.6+ only, which means it should be possible to just sync the latest version over from CPython 3.10 (modulo keeping the contextlib2 APIs that never graduated to the stdlib version)
ncoghlan commentedon Jun 26, 2021
I've merged the changes from #29 (switch to CalVer, set the minimum version to 3.6), clearing the way for the Python 3.10 version to be sync'ed over. With the 2.x compatibility code gone, I'm hoping that will now be a lot simpler than it used to be.
Issue jazzband#12: sync contextlib from Python 3.10
ncoghlan commentedon Jun 26, 2021
PR is up #32
CI mostly looks good, just need to check what's up with PyPy3
Merge pull request #32 from ncoghlan/issue-12-sync-module-from-python…
ncoghlan commentedon Jun 26, 2021
As far as I can tell, the PyPy3 issue was a bug in the stdlib test suite (assuming the use of a refcounted GC): https://bugs.python.org/issue44515
CPython PR submitted at python/cpython#26910
ncoghlan commentedon Jun 26, 2021
Bah, forgot to include the docs updates for the new APIs
Issue jazzband#12: Sync API documentation from Python 3.10
Merge pull request #35 from ncoghlan/issue-12-backport-py3.10-context…
ncoghlan commentedon Jun 26, 2021
OK, docs are also updated now.