Repository navigation
ESP32C3 and C6: Stack protection fault on IDF 5.2.2 #15667
Description
Port, board and/or hardware
PORT: esp32, BOARD: esp32c6-devkitc-1
MicroPython version
v1.24.0-preview. forked at e9814e987bcc816fb67e38748a5afce466c45606. with #11869 applied for C6 support.
https://github.com/mzakharocsc/micropython/tree/c6
Reproduction
Code:
https://github.com/mzakharocsc/micropython/blob/c6/ports/esp32/modules/main.py
Edit file with your wifi SSID/PASSWORD. Start your MQTT broker ( I used mosquitto). Edit config['server'] = '192.168.50.31' with your broker IP address. Connect the board to UART port, Inside ports/esp32, make BOARD=ESP32_GENERIC_C6 deploy and monitor.
Code has to be run from main.py on board startup for some reason to reproduce this.
Expected behaviour
Expecting the board on startup to connect to a broker and publish continously without crashing. Code does not crash on IDF 5.1.4. On IDF 5.2.2, code crashes after a few minutes of runtime.
Observed behaviour
A fatal error occurred. The crash dump printed below may be used to help
determine what caused it. If you are not already running the most recent
version of MicroPython, consider upgrading. New versions often fix bugs.
To learn more about how to debug and/or report this crash visit the wiki
page at: https://github.com/micropython/micropython/wiki/ESP32-debugging
MPY version : v1.24.0-preview.209.gb5f390386.dirty on 2024-08-16
IDF version : v5.2.2
Machine : ESP32C6 module with ESP32C6
Guru Meditation Error: Core 0 panic'ed (Stack protection fault).
Detected in task "mp_task" at 0x4200e4fc
0x4200e4fc: nlr_jump at /home/mzakharo/station/micropythonc6x/py/nlrrv32.c:55
Stack pointer: 0x4082e8a0
Stack bounds: 0x40826d5c - 0x4082ed50
Core 0 register dump:
Stack dump detected
MEPC : 0x420196f8 RA : 0x40803aac SP : 0x4082e8a0 GP : 0x40818e94
0x420196f8: mp_obj_is_subclass_fast at /home/mzakharo/station/micropythonc6x/py/objtype.c:1364
0x40803aac: mp_execute_bytecode at /home/mzakharo/station/micropythonc6x/py/vm.c:1382 (discriminator 1)
TP : 0x407ecb7c T0 : 0x40022494 T1 : 0x4081321c T2 : 0x0000003f
0x40022494: multi_heap_internal_unlock in ROM
0x4081321c: vTaskSuspend at /home/mzakharo/esp-idf2/components/freertos/FreeRTOS-Kernel/tasks.c:1977
S0/FP : 0x40838518 S1 : 0x40838558 A0 : 0x4219e010 A1 : 0x4219e010
A2 : 0x00000000 A3 : 0x00000100 A4 : 0x0000000e A5 : 0x4082e9a0
A6 : 0x00000001 A7 : 0x00000002 S2 : 0x421a09f4 S3 : 0x408385b0
S4 : 0x0000003c S5 : 0x4219e000 S6 : 0x00000068 S7 : 0x4219e2d0
S8 : 0x4081c000 S9 : 0x4219e000 S10 : 0x421a8b20 S11 : 0x421a8b1b
T3 : 0x00000000 T4 : 0x00002b3a T5 : 0x00000003 T6 : 0x00000001
MSTATUS : 0x00001881 MTVEC : 0x40800001 MCAUSE : 0x0000001b MTVAL : 0x00001101
0x40800001: _vector_table at ??:?
MHARTID : 0x00000000
Backtrace:
mp_obj_is_subclass_fast (object=0x4219e010 <mp_type_StopIteration>, classinfo=0x4219e010 <mp_type_StopIteration>) at /home/mzakharo/station/micropythonc6x/py/objtype.c:1364
1364 bool mp_obj_is_subclass_fast(mp_const_obj_t object, mp_const_obj_t classinfo) {
#0 mp_obj_is_subclass_fast (object=0x4219e010 <mp_type_StopIteration>, classinfo=0x4219e010 <mp_type_StopIteration>) at /home/mzakharo/station/micropythonc6x/py/objtype.c:1364
#1 0x40803aac in mp_execute_bytecode (code_state=code_state@entry=0x40838518, inject_exc=inject_exc@entry=0x0) at /home/mzakharo/station/micropythonc6x/py/vm.c:1382
#2 0x4201257a in mp_obj_gen_resume (self_in=0x40838510, send_value=<optimized out>, throw_value=throw_value@entry=0x0, ret_val=ret_val@entry=0x4082e96c) at /home/mzakharo/station/micropythonc6x/py/objgenerator.c:210
#3 0x420126f4 in gen_resume_and_raise (raise_stop_iteration=true, throw_value=0x0, send_value=<optimized out>, self_in=<optimized out>) at /home/mzakharo/station/micropythonc6x/py/objgenerator.c:259
#4 gen_instance_send (self_in=<optimized out>, send_value=<optimized out>) at /home/mzakharo/station/micropythonc6x/py/objgenerator.c:286
#5 0x408040b8 in mp_execute_bytecode (code_state=code_state@entry=0x408347c0, inject_exc=inject_exc@entry=0x0) at /home/mzakharo/station/micropythonc6x/py/vm.c:1042
#6 0x4201207a in fun_bc_call (self_in=<optimized out>, n_args=1, n_kw=0, args=0x4082eb08) at /home/mzakharo/station/micropythonc6x/py/objfun.c:267
#7 0x00000000 in ?? ()
Backtrace stopped: frame did not save the PC
ELF file SHA256: aa26cad08
Rebooting...
Additional Information
No, I've provided everything above.
Code of Conduct
Yes, I agree
Activity
mzakharocsc commentedon Aug 16, 2024
Digging a bit further. it seems 5.2.x introduced a new feature - Hardware Stack Guard, which explains why 5.1.4 does not crash.
Setting CONFIG_ESP_SYSTEM_HW_STACK_GUARD=n resolves the crashes. Is this the correct fix though?
dpgeorge commentedon Aug 16, 2024
I doubt that's a good fix, it's more of a workaround.
Instead it would be good to understand why the stack is overflowing.
Are you able to run the MicroPython test suite on the C6? That has some recursion tests which may show the problem and make it easier to debug.
mzakharocsc commentedon Aug 16, 2024
Test results:
Also, doubling the stack in mpconfigport.h has no impact on the crash
andrewleech commentedon Aug 23, 2024
Thanks for the report and the reproduction code.
I've just had it running on one of my C6 boards which is running a older build:
I also have a mosquito running so used that, though I had to add user/pass to the config.
Otherwise I copied async mqtt library from your repo and loaded the script as main.
I don't have IDF monitor set up so just watched with mpremote over the USB port (not UART)
After 3 hours it's still running correctly, no crash observed yet.
I'll upgrade the board to a newer version and try again.
andrewleech commentedon Aug 23, 2024
Ok tried again on a somewhat newer version of the PR and still no failure after running for a good 30 minutes.
andrewleech commentedon Aug 25, 2024
Just to round out the testing I rebased again to e9814e9 and re-tested, including getting IDF monitor connected instead of mpremote, still no failure seen....
However I then looked closer at your error message and noted the "rebooting" in the end. Did you monitor stop when the error occurred?
When I'm running the script it's constantly streaming
over and over, so if the error occurs and reboots automatically the error message would be rapidly lost. I'd better try again with the outputs logged to search through.
Separately, could you check and share the revision of your chip?
I've got two C6 modules. One was by 01space back when C6 was first announced. Turns out this one crashes for me immediately as soon as wifi connect is called.
My other unit is a m5stack nanoc6 and it works fine.
When you run esptool on the module, the revision is printed out.
My broken one reports:
Whereas my working one shows:
andrewleech commentedon Aug 25, 2024
Sorry I was wrong, my (good) unit does crash like your report, it was just lost in the mqtt activity logging.
andrewleech commentedon Aug 26, 2024
Oh and @mzakharocsc yes I can confirm your finding about needing to run it as main.py from startup. The exact same script, not installed as
main.pybut instead run viampremote run espc6test.pydoes not crash - mine has been running now for about 2 hours whereas yes it crashes after a few minutes when run at startup.update:
Well actually the mpremote run did eventually crash, after 57MB worth of
We don't have a full stack trace because this was mpremote rather than monitor, but this is what I have:
andrewleech commentedon Aug 26, 2024
On a related note, C3 is susceptible to the same issue
This was the same code, run the same way.
andrewleech commentedon Aug 27, 2024
I then tried running the same script on S2 - got a few hours in without crash there. So perhaps its limited to the risc cores.
projectgus commentedon Aug 27, 2024
@andrewleech What commit of master branch are these branched from? Asking as I don't seem to have that git hash otherwise, and because a bunch of tweaking for stack on esp32c3 went in recently.
andrewleech commentedon Aug 27, 2024
Hi @projectgus yeah those latest test results were from me rebasing my c6 branch onto e9814e9, similar to the original bug report. I had seen your recent stack work and I'm pretty sure it was included in this commit
I've just retested C3 on latest master a8d1c25 (via idf monitor) and got the same crash:
andrewleech commentedon Aug 27, 2024
Retested C3 on v1.23 and still get the same issue:
4 remaining items
projectgus commentedon Aug 28, 2024
I'm pretty sure this is an ESP-IDF or ESP32-C3 issue because the stack pointer stays valid the whole time, from the point of view of our task. (i.e. hardware stack protector is supposed to trigger if the stack pointer is outside the stack bounds for the task, but all these crash dumps show it inside the stack bounds.) Have raised the ESP-IDF issue linked above which has a bunch more details.
If we don't want to wait for an upstream fix, we can workaround:
dpgeorge commentedon Oct 11, 2024
According to #11869 (comment) this also needs fixing for ESP32-C6.
dpgeorge commentedon Oct 11, 2024
I'm able to reproduce the original bug reported here using
ESP32_GENERIC_C6firmware, on current master 82e69df, with the code https://github.com/mzakharocsc/micropython/blob/c6/ports/esp32/modules/main.py . Actually to reproduce it I needed to freeze bothmain.pyandmqtt_async.pyfrom that repo into the C6 firmware (running those files from the filesystem, I couldn't reproduce the issue).esp32: Disable hardware stack protection on ESP32-C6.
dpgeorge commentedon Oct 11, 2024
See #15997 for a fix for C6.
esp32: Disable hardware stack protection on ESP32-C6.
esp32: Disable hardware stack protection on ESP32-C6.
esp32: Disable hardware stack protection on ESP32-C6.
esp32: Disable hardware stack protection on ESP32-C6.
esp32: Disable hardware stack protection on ESP32-C6.