Skip to content

rp2: fix atomic section - #13312

Merged
dpgeorge merged 3 commits into
micropython:masterfrom
dpgeorge:rp2-fix-atomic-section
Jan 3, 2024
Merged

dpgeorge merged 3 commits into
micropython:masterfrom
dpgeorge:rp2-fix-atomic-section

Conversation

@dpgeorge

@dpgeorge dpgeorge commented Jan 1, 2024 •

Copy link
Copy Markdown
Member

This is an attempt to fix #12980 and #13288.

There are two fixes to the MICROPY_BEGIN_ATOMIC_SECTION/ MICROPY_END_ATOMIC_SECTION on the rp2 port:

  • Use a recursive mutex to prevent deadlocks on the same thread, in the case an IRQ races against the thread level (on the same core) when acquiring the atomic section (mutex may be taken but IRQs not yet disabled, then the IRQ waits forever trying to acquire the mutux).
  • Unlock the mutex if it was locked, not if core1_entry is still non-null, to fix the case where the second core finishes (and sets core1_entry to NULL) while the first core is in the middle of an atomic operation.

Edit: the two changes are now:

  1. split out multicore lockout handling so it's only done when doing a flash erase/write
  2. change the ATOMIC_SECTION macros so they use new mutex enter/exit functions that also disable/restore interrupts (atomically).

@mendenm

mendenm commented Jan 1, 2024

Copy link
Copy Markdown

It works! At least the simple test script runs both processors and reports results form them.

Excellent.

Comment thread ports/rp2/mpthreadport.c Outdated
@dpgeorge
dpgeorge force-pushed the rp2-fix-atomic-section branch from d062a6f to 130b4a9 Compare January 2, 2024 03:52
@dpgeorge

dpgeorge commented Jan 2, 2024

Copy link
Copy Markdown
Member Author

I have changed this to use a new set of mutex functions that also disable/restore interrupts when obtaining/releasing the mutex.

@projectgus

Copy link
Copy Markdown
Contributor

New approach looks good to me!

Using the multicore lockout feature in the general atomic section makes it
much more difficult to get correct.

Signed-off-by: Damien George <[email protected]>
These allow entering/exiting a mutex and also disabling/restoring
interrupts, in an atomic way.

Signed-off-by: Damien George <[email protected]>
Prior to this commit there is a potential deadlock in
mp_thread_begin_atomic_section(), when obtaining the atomic_mutex, in the
following situation:
- main thread calls mp_thread_begin_atomic_section() (for whatever reason,
  doesn't matter)
- the second core is running so the main thread grabs the mutex via the
  call mp_thread_mutex_lock(&atomic_mutex, 1), and this succeeds
- before the main thread has a chance to run save_and_disable_interrupts()
  a USB IRQ comes in and the main thread jumps off to process this IRQ
- that USB processing triggers a call to the dcd_event_handler() wrapper
  from commit bcbdee2
- that then calls mp_sched_schedule_node()
- that then attempts to obtain the atomic section, calling
  mp_thread_begin_atomic_section()
- that call then blocks trying to obtain atomic_mutex
- core0 is now deadlocked on itself, because the main thread has the mutex
  but the IRQ handler (which preempted the main thread) is blocked waiting
  for the mutex, which will never be free

The solution in this commit is to use mutex enter/exit functions that also
atomically disable/restore interrupts.

Fixes issues micropython#12980 and micropython#13288.

Signed-off-by: Damien George <[email protected]>
@dpgeorge
dpgeorge force-pushed the rp2-fix-atomic-section branch from 3586586 to dc2a4e3 Compare January 3, 2024 05:00
@dpgeorge
dpgeorge merged commit dc2a4e3 into micropython:master Jan 3, 2024
@dpgeorge
dpgeorge deleted the rp2-fix-atomic-section branch January 3, 2024 05:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

_thread module freezing whole device (RPI_PICO_W)

3 participants