Repository navigation
PEP 654: Exception groups and except *, leading to asyncio TaskGroup #8508
Description
PEP 654 was recently added to CPython 3.11. It adds exceptions propagated together and a new except* syntax to handle them. Tracebacks handle this as well. This was considered a prerequisite to adding TaskGroup to asyncio, and delayed TaskGroup for about three years.
I am very fond of TaskGroup, and think it makes asyncio a lot easier to use. I think it will also be the style going forward, though it will take a while to catch on.
@dpgeorge and @jimmo, have you thought about when or if you might add TaskGroup, and whether you would add it compatibly (implementing PEP 654)? 654 seems pretty complicated and might add a lot of code, but I haven't looked at it in detail. Or might you add TaskGroup but not make it quite compatible?
BTW, the person who did async/await for CircuitPython added __await__. I think you are still using __iter__, but I wonder about whether you feel the need to add __await__ as well.
Activity
m-u-xyz commentedon Jun 20, 2022
You want taskgroups, you get them. 😁
m-u-xyz commentedon Jun 20, 2022
Personally I think the
except *stuff is seriously over-engineered, at least for the micropython context where we don't have a__traceback__attached to the exception. My solution is to dump the sub-errors to the console, don't raise anExceptionGroupwhen there's only one exception in it (Trio works that way), and call it a wrap.dhalbert commentedon Jun 20, 2022
@smurfix Thank you!!
I have not looked at the
except *stuff in detail, except to see that is complicated. I don't know if it is possible to write CPython task group code without it (that is, with plainexcept). Making MicroPython's support of TaskGroups be upward compatible would be great, so that at least code could be written in a subset style that would run on CPython and MicroPython.But I am motivated by CircuitPython's general goals, not MicroPython's.
m-u-xyz commentedon Jun 20, 2022
If we really want CPython compatible exception groups, there's compatibility code to unravel
ExceptionGroups for Py3.9+ out there which we probably could adapt. It basically implements all theexcept *syntactic sugar (which admittedly is very nice sugar when you need it, given that we don't havenonlocal).The question is, do we need to bother? In an embedded context the focus should be on reliably getting back to a working state, which you can easily do (in most cases) with plain vanilla "except Exception: … raise" and "finally:" handlers. As my backport explicitly use an
ExceptionGroupinstead of aBaseExceptionGroupwhen all of the embedded exceptions do descend fromExceptionand skips using an exception group in the first place if there's only one exception to be reported, I'd like to see code that actually requires the complications and all the syntactic sugar before spending a heap of effort to implement them.If we do want to fix this we'd also need
ExceptionGroupto multiply-inherit from bothBaseExceptionGroupandException. This is not possible without C-level surgery, which I am not about to attempt any time soon.Not to mention: Backporting a working
TaskGrouptook half a day, more or less. Backporting a "real" exception group would require roughly the same time, on top of fixing the inheritance issue. Implementingexcept *handling in µPy would be more than a week's effort for somebody who knows their way around MicroPython's C source and coding conventions, i.e. definitely not me. Frankly I'd rather spend the time on something more productive, esp. given 289 open pull requests and 1200+ issues.IMHO.
dpgeorge commentedon Jun 21, 2022
nonlocalis supported in MicroPython.dpgeorge commentedon Jun 21, 2022
I have not put any thought into this and never used TaskGroup. Also not looked at PEP 654. It sounds like a lot of work for not much gain.
I agree that CPython compatibility is a good goal, and maybe we can do it (at least a subset) in just pure Python as @smurfix has done. That sounds like a really good approach, at least to start with.
I also haven't thought much about this, and there hasn't really been a need for it from what I've seen (although maybe @peterhinch has other experiences).
dhalbert commentedon Jun 21, 2022
The only thing this has meant for CircuitPython is that we don't/can't use
yieldfor task switching, and instead useawait asyncio.sleep(0), as would be done in CPython.dpgeorge commentedon Jun 21, 2022
We don't use (or at least don't encourage use of)
yield.yieldis only used in very low level code, eg stream yielding using_io_queueto wait for stream readiness.peterhinch commentedon Jun 21, 2022
The only time I've had to engage with this issue is in writing awaitable classes. There is a minor bear-trap in making an awaitable class portable between MP and CPython. As far as I'm concerned this is a minor issue. It's documented here including the workround needed for CPython compatibility.
2 remaining items
m-u-xyz commentedon Jun 22, 2022
Bah. Sorry about the noise.
In any case, I tried to teach
_uasyncio.Taskto be hashable, which seems to cause spurious (IMHO) test failues. Can somebody look at what might possibly be wrong with my second commit? If there is, I don't see it …extmod/uasyncio: Backport py3.11 asyncio's taskgroups.
extmod/uasyncio: Backport py3.11 asyncio's taskgroups.
Merge pull request micropython#8508 from dhalbert/v1.21-merge
extmod/uasyncio: Backport py3.11 asyncio's taskgroups.
extmod/uasyncio: Backport py3.11 asyncio's taskgroups.