V1.22 creating new thread leads to hangup

Dear micropython developers,

this morning I found about micropython V1.22 and tried this on my Raspberry Pico (non W). But unfortunately I had to find out that my programs stop when the second thread is created.

consider this boot.py to reproduce the issue:

import _thread
import machine 
import asyncio

from time import sleep, sleep_ms


async def thread1task():
    print("thread 1 started")
    while True:
        print("thread 1 talking ... ")
        await asyncio.sleep(1)
        
def thread2task():
    print("thread 2 started")
    while True:
        print("thread 2 talking ... ")
        sleep(1)


if __name__ == "__main__":
    try:
        loop = asyncio.get_event_loop()
        messagestart = const(">> starting script in ... ")
        print(messagestart + f"3")
        sleep(1)
        print(messagestart + f"2")
        sleep(1)
        print(messagestart + f"1")
        sleep(1)
        print(">> Loop is starting")


        taskobj = loop.create_task(thread1task())
        print(">> starting second thread...")
        _thread.start_new_thread(thread2task,())
        print(">> running loop ...")

        loop.run_forever()
    except KeyboardInterrupt:
        loop.stop()
        _thread.exit() # does not work, code hangs
        print(f"INTERRUPTED")

The hangup is not always at the exact same spot, but the "most" debug output I could receive was:

MPY: soft reboot
>> starting script in ... 3
>> starting script in ... 2
>> starting script in ... 1
>> Loop is starting
>> starting second thread...
>> running loop ...thread 2 started

thread 2 talking ...

After that the serial port is blocked, device does not react to CTRL+C or any other keys.

Executing the code above with V1.21 has no issues and runs forever as expected.

@jimmo

I can confirm this. It is a regression: the script works with (e.g)

MicroPython v1.21.0 on 2023-10-06; Raspberry Pi Pico with RP2040

but fails with

MicroPython v1.22.0 on 2023-12-27; Raspberry Pi Pico with RP2040

@jimmo ?
[EDIT]
This is a minimal repro (paste at the REPL):

import _thread
from time import sleep

def foo(st):
    while True:
        sleep(1)
        print(st)

_thread.start_new_thread(foo,("core 2",))
foo("core 1")

I happened to have an older preview sitting around. This script works fine in RPI_PICO-20231104-v1.22.0-preview.123.g47ed06bda.uf2, so the regression happened after 20231104. Time for binary search... It fails in RPI_PICO-20231221-v1.22.0-preview.278.g74fd7b3d3.uf2.

Is there any way to get older preview builds than a day or so off the server, so I don't have to start compiling revisions myself?

@dpgeorge

@dpgeorge @jimmo OK, bbc5a18 is good, and bcbdee2 is bad for this regression.

bcbdee2 is bad for this regression.

Thanks for doing the bisection.

I guess the issue is that TinyUSB is being run on the second core, when it should be run only on the first core; see daff597 for the original fix to this issue.

A potential fix could be this:

--- a/py/scheduler.c
+++ b/py/scheduler.c
@@ -88,7 +88,7 @@ static inline void mp_sched_run_pending(void) {
 
     #if MICROPY_SCHEDULER_STATIC_NODES
     // Run all pending C callbacks.
-    while (MP_STATE_VM(sched_head) != NULL) {
+    while (mp_thread_is_main_thread() && MP_STATE_VM(sched_head) != NULL) {
         mp_sched_node_t *node = MP_STATE_VM(sched_head);
         MP_STATE_VM(sched_head) = node->next;
         if (MP_STATE_VM(sched_head) == NULL) {

@dpgeorge

@dpgeorge This patch didn't seem to fix it. It still hangs.

I've tried to fix this problem in #13312.

@dpgeorge

@dpgeorge @TRadigk Fixed! At least under simple testing.

This issue was a tricky one to figure out, but it seems that the problem is a deadlock in mp_thread_begin_atomic_section(), when obtaining the atomic_mutex, in the following situation:

main thread calls mp_thread_begin_atomic_section() (for whatever reason, doesn't matter)
the second core is running so the main thread grabs the mutex via the call mp_thread_mutex_lock(&atomic_mutex, 1);, and this succeeds
before the main thread has a chance to run save_and_disable_interrupts() a USB IRQ comes in and the main thread jumps off to process this IRQ
that USB processing triggers a call to the dcd_event_handler() wrapper from commit bcbdee2
that then calls mp_sched_schedule_node()
that then attempts to obtain the atomic section, calling mp_thread_begin_atomic_section()
that call then blocks trying to obtain atomic_mutex
core0 is now deadlocked on itself, because the main thread has the mutex but the IRQ handler (which preempted the main thread) is blocked waiting for the mutex, which will never be free

So the second core is not playing any role here, it's just the fact that mp_thread_begin_atomic_section() is obtaining a mutex with interrupts still enabled. And it can't first disable interrupts because obtaining the mutex may block for some period (eg waiting for core1). So the solution is to use a recursive mutex.

Fixed by dc2a4e3

Awesome finding/fixing. Thank for your quick response and support. When I would like to test the fix, do I have to make sure to take a 1.23 preview build or will this be included in any 1.22 nightly? Since this is a rather big issue (for applications making use of multithreading) will 1.22 be removed? Or replaced by 1.22.1?

The v1.x nightly builds have been renamed to v1.x+1 preview. So 1.23 preview is what you should use. Today's merges and the previous ones since v1.22 should be in the preview from tomorrow on.

rp2/mpthreadport: Fix race with IRQ when entering atomic section.

Hello, I’ve been using v1.20 with some issues similar to this with the threads, I was able to control it by adding try-except to retry when starting a thread, but now using v1.22.2 that is not working anymore. I’ve figured out that the problem appears when using several threads and activating BLE. Without BLE I can still start several threads without issues, but just adding these three lines to my code, the threads start failing:

 from bluetooth import BLE
 myble = BLE()
 myble.activate(True)

Specifically the third line is the one that triggers the issues. If I comment it out, all the threads run well.
Is there anyway to avoid this issues to use BLE and threads in the same code?

rp2/mpthreadport: Fix race with IRQ when entering atomic section.

Navigation Menu

Repository navigation

V1.22 creating new thread leads to hangup #13288

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

V1.22 creating new thread leads to hangup #13288

Description

Activity

peterhinch commented on Dec 28, 2023

mendenm commented on Dec 28, 2023

mendenm commented on Dec 29, 2023

dpgeorge commented on Dec 29, 2023

mendenm commented on Dec 29, 2023

dpgeorge commented on Jan 1, 2024

mendenm commented on Jan 1, 2024

dpgeorge commented on Jan 2, 2024

dpgeorge commented on Jan 3, 2024

TRadigk commented on Jan 3, 2024

robert-hh commented on Jan 3, 2024

nestorld commented on Jun 1, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions