RFC py/objgenerator: Allow to pend an exception for next execution. - #3380

This PR addresses (some) challenges described in micropython/micropython-lib#215.

Travis failure due to size check (the new functionality of course needs to be ifdef'ed).

Added returning of previous value by .pend_throw(), motivation described in micropython/micropython-lib#215 (comment)

Added config option and test. This should be pretty ready to merge.

@dpgeorge: Ping

@peterhinch: If you tested this with micropython/micropython-lib#221 , please ack.

I've successfully tested the PR for regressions by running applications which make substantial use of uasyncio. I'll try to find time to test the new functionality this weekend.

@peterhinch : Thanks.

I've had a look at this and it's awesome. I can now cancel coroutines and trap the cancellation!

A couple of observations. This is surely a typo.

Secondly, and this is perhaps fairly self-evident.
If you have a coro which is subject to a timeout, but the timeout occurs when that coro is waiting (via yield millisecs or await asyncio.sleep(secs)), the coro is not cancelled until it is next scheduled. It is obvious why this occurs and in terms of the execution path of that coro cancellation is immediate, so I don't regard it as an issue. But it should be documented as there may be circumstances in which this behaviour will need to be coded around - something which is easily done.

[EDIT]
I've now tested this fairly extensively and failed to break it! I think this great patch is good to go.

One observation. I cancel a coroutine by issuing coro.pend_throw(CancelError) which the coro traps and then terminates. If by that time the coro has already terminated, pend_throw() fails silently and returns None regardless of success status.

This has no discernible ill effects and causes me no problem, but I thought it worth recording.

This is surely a typo.

Yep. Was even fixed in a local clean-up which I forgot to push to that PR, now micropython/micropython-lib#221 updated.

If you have a coro which is subject to a timeout, but the timeout occurs when that coro is waiting (via yield millisecs or await asyncio.sleep(secs)), the coro is not cancelled until it is next scheduled.

Well, yeah, the way sleep() is implemented, it won't be easy to make it cancellable. Hopefully, at least on first look, that shouldn't be very needed. Other cases of cancellation are expected to be covered, as I mentioned, micropython/micropython-lib#224 was added specifically to have (non-I/O) case for cancellation.

One observation. I cancel a coroutine by issuing coro.pend_throw(CancelError) which the coro traps and then terminates. If by that time the coro has already terminated, pend_throw() fails silently and returns None regardless of success status.

I'm not sure about "fail silently" valuation, I'd say "it has no effect". And as this patch explains, return value from pend_throw() is a previous value which was pended, which is initially None, so that's the reason why you get None.

Now, here's a further idea:

As I mentioned, we wouldn't be able to implement CPython's asyncio Task.cancel() method, because there's no Task's in uasyncio, only coro's. Instead, it would need to be a main loop method.

But, we could achieve better asyncio compatibility by renaming this new .pend_throw() method to .cancel(), and making it default to CancelledError if no arg is passed (the rest of semantics of .pend_throw() needs to stay). One caveat to this is that CancelledError now needs to be defined on C level.

I think this idea makes sense, would need to wait for @dpgeorge's feedback.

I'm agnostic about the naming scheme but it would be good to see this merged.

This is in review for quite some time, no concerns were raised. The latest version of uasyncio depends on this functionality (won't work without it). So, I'm merging this to my fork, and while I'm able to, propagating it to upstream.

@pfalcon I'd be very grateful if you could explain the purpose of the return value from pend_throw(), in particular the circumstance in which it can return False.

It just returns previous value pended (or None if nothing). The idea is that it's possible to associate an additional flags-like value with a coro - as long as it's not running. A usecase can be seen in uasyncio. And if it doesn't make sense, that's ok - .pend_throw() is an internal function not supposed to be used by end users (only by coro schedulers).

Thanks for that. My interest was partly promoted by the behaviour of the following code; a coro which is waiting on a lock does not respond to a timeout. I'm unclear if this is by design - perhaps you can advise.

import uasyncio as asyncio
from uasyncio.synchro import Lock

async def needs_lock(lock):
    try:
        await lock.acquire()
        print('Acquired lock in needs_lock')  # Never happens (correctly)
    except asyncio.TimeoutError:
        print('Got timeout')  # But nor does this (???)

async def main():
    lock = Lock()
    await lock.acquire()  # Succeeds
    print('Acquired lock in main')
    await asyncio.wait_for(needs_lock(lock), 5)  # hangs
    print('Finishing main()')

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

This prints 'Acquired lock in main' and hangs indefinitely.

Test conditions

MicroPython v1.9.3-181-g02d2a0f on 2017-12-16; PYBv1.1 with STM32F405RG

a coro which is waiting on a lock does not respond to a timeout

That's not implemented yet. As I mentioned, I implemented lock specifically to see if it would fit with the cancellation framework, but didn't yet implement it.