Skip to content

ESP32C3 and C6: Stack protection fault on IDF 5.2.2 #15667

Description

@mzakharocsc

Port, board and/or hardware

PORT: esp32, BOARD: esp32c6-devkitc-1

MicroPython version

v1.24.0-preview. forked at e9814e987bcc816fb67e38748a5afce466c45606. with #11869 applied for C6 support.

https://github.com/mzakharocsc/micropython/tree/c6

Reproduction

Code:
https://github.com/mzakharocsc/micropython/blob/c6/ports/esp32/modules/main.py

Edit file with your wifi SSID/PASSWORD. Start your MQTT broker ( I used mosquitto). Edit config['server'] = '192.168.50.31' with your broker IP address. Connect the board to UART port, Inside ports/esp32, make BOARD=ESP32_GENERIC_C6 deploy and monitor.

Code has to be run from main.py on board startup for some reason to reproduce this.

Expected behaviour

Expecting the board on startup to connect to a broker and publish continously without crashing. Code does not crash on IDF 5.1.4. On IDF 5.2.2, code crashes after a few minutes of runtime.

Observed behaviour

A fatal error occurred. The crash dump printed below may be used to help
determine what caused it. If you are not already running the most recent
version of MicroPython, consider upgrading. New versions often fix bugs.

To learn more about how to debug and/or report this crash visit the wiki
page at: https://github.com/micropython/micropython/wiki/ESP32-debugging

MPY version : v1.24.0-preview.209.gb5f390386.dirty on 2024-08-16
IDF version : v5.2.2
Machine     : ESP32C6 module with ESP32C6

Guru Meditation Error: Core  0 panic'ed (Stack protection fault). 

Detected in task "mp_task" at 0x4200e4fc
0x4200e4fc: nlr_jump at /home/mzakharo/station/micropythonc6x/py/nlrrv32.c:55

Stack pointer: 0x4082e8a0
Stack bounds: 0x40826d5c - 0x4082ed50


Core  0 register dump:
Stack dump detected
MEPC    : 0x420196f8  RA      : 0x40803aac  SP      : 0x4082e8a0  GP      : 0x40818e94  
0x420196f8: mp_obj_is_subclass_fast at /home/mzakharo/station/micropythonc6x/py/objtype.c:1364
0x40803aac: mp_execute_bytecode at /home/mzakharo/station/micropythonc6x/py/vm.c:1382 (discriminator 1)

TP      : 0x407ecb7c  T0      : 0x40022494  T1      : 0x4081321c  T2      : 0x0000003f  
0x40022494: multi_heap_internal_unlock in ROM
0x4081321c: vTaskSuspend at /home/mzakharo/esp-idf2/components/freertos/FreeRTOS-Kernel/tasks.c:1977

S0/FP   : 0x40838518  S1      : 0x40838558  A0      : 0x4219e010  A1      : 0x4219e010  
A2      : 0x00000000  A3      : 0x00000100  A4      : 0x0000000e  A5      : 0x4082e9a0  
A6      : 0x00000001  A7      : 0x00000002  S2      : 0x421a09f4  S3      : 0x408385b0  
S4      : 0x0000003c  S5      : 0x4219e000  S6      : 0x00000068  S7      : 0x4219e2d0  
S8      : 0x4081c000  S9      : 0x4219e000  S10     : 0x421a8b20  S11     : 0x421a8b1b  
T3      : 0x00000000  T4      : 0x00002b3a  T5      : 0x00000003  T6      : 0x00000001  
MSTATUS : 0x00001881  MTVEC   : 0x40800001  MCAUSE  : 0x0000001b  MTVAL   : 0x00001101  
0x40800001: _vector_table at ??:?

MHARTID : 0x00000000  


Backtrace:


mp_obj_is_subclass_fast (object=0x4219e010 <mp_type_StopIteration>, classinfo=0x4219e010 <mp_type_StopIteration>) at /home/mzakharo/station/micropythonc6x/py/objtype.c:1364
1364	bool mp_obj_is_subclass_fast(mp_const_obj_t object, mp_const_obj_t classinfo) {
#0  mp_obj_is_subclass_fast (object=0x4219e010 <mp_type_StopIteration>, classinfo=0x4219e010 <mp_type_StopIteration>) at /home/mzakharo/station/micropythonc6x/py/objtype.c:1364
#1  0x40803aac in mp_execute_bytecode (code_state=code_state@entry=0x40838518, inject_exc=inject_exc@entry=0x0) at /home/mzakharo/station/micropythonc6x/py/vm.c:1382
#2  0x4201257a in mp_obj_gen_resume (self_in=0x40838510, send_value=<optimized out>, throw_value=throw_value@entry=0x0, ret_val=ret_val@entry=0x4082e96c) at /home/mzakharo/station/micropythonc6x/py/objgenerator.c:210
#3  0x420126f4 in gen_resume_and_raise (raise_stop_iteration=true, throw_value=0x0, send_value=<optimized out>, self_in=<optimized out>) at /home/mzakharo/station/micropythonc6x/py/objgenerator.c:259
#4  gen_instance_send (self_in=<optimized out>, send_value=<optimized out>) at /home/mzakharo/station/micropythonc6x/py/objgenerator.c:286
#5  0x408040b8 in mp_execute_bytecode (code_state=code_state@entry=0x408347c0, inject_exc=inject_exc@entry=0x0) at /home/mzakharo/station/micropythonc6x/py/vm.c:1042
#6  0x4201207a in fun_bc_call (self_in=<optimized out>, n_args=1, n_kw=0, args=0x4082eb08) at /home/mzakharo/station/micropythonc6x/py/objfun.c:267
#7  0x00000000 in ?? ()
Backtrace stopped: frame did not save the PC
ELF file SHA256: aa26cad08

Rebooting...

Additional Information

No, I've provided everything above.

Code of Conduct

Yes, I agree

Activity

  1. mzakharocsc commented on Aug 16, 2024

    @mzakharocsc
    ContributorAuthor

    Digging a bit further. it seems 5.2.x introduced a new feature - Hardware Stack Guard, which explains why 5.1.4 does not crash.

    Setting CONFIG_ESP_SYSTEM_HW_STACK_GUARD=n resolves the crashes. Is this the correct fix though?

  2. dpgeorge commented on Aug 16, 2024

    @dpgeorge
    Member

    Setting CONFIG_ESP_SYSTEM_HW_STACK_GUARD=n resolves the crashes. Is this the correct fix though?

    I doubt that's a good fix, it's more of a workaround.

    Instead it would be good to understand why the stack is overflowing.

    Are you able to run the MicroPython test suite on the C6? That has some recursion tests which may show the problem and make it easier to debug.

  3. mzakharocsc commented on Aug 16, 2024

    @mzakharocsc
    ContributorAuthor

    Test results:

    817 tests performed (24544 individual testcases)
    816 tests passed
    97 tests skipped: builtin_next_arg2 builtin_range_binop cexample_class cexample_module cexample_subclass cryptolib_aes128_ctr deflate_compress deflate_stream_error exception_chain float2int_doubleprec_intbig float_divmod float_format_ints_doubleprec float_parse_doubleprec heap_locked heapalloc_bytesio2 import_mpy_native_gc machine_i2s_rate machine_pinbase machine_pulse machine_signal machine_soft_timer machine_uart_tx meminfo memoryview_itemsize memstats namedtuple_asdict nanbox_smallint native_closure native_const native_const_intbig native_for native_fun_attrs native_gen native_misc native_try native_try_deep native_while native_with re_debug re_groups re_span select_poll_fd ssl_ioctl ssl_poll subclass_native_call sys_atexit sys_exc_info sys_getsizeof sys_path sys_settrace_features sys_settrace_generator sys_settrace_loop sys_tracebacklimit vfs_fat_ramdisklarge vfs_lfs vfs_lfs_corrupt vfs_lfs_error vfs_lfs_file vfs_lfs_mount vfs_posix vfs_posix_enoent vfs_posix_ilistdir_del vfs_posix_ilistdir_filter vfs_posix_paths viper_addr viper_args viper_binop_arith viper_binop_arith_uint viper_binop_bitwise_uint viper_binop_comp viper_binop_comp_imm viper_binop_comp_uint viper_binop_divmod viper_binop_multi_comp viper_cond viper_const viper_const_intbig viper_error viper_globals viper_import viper_misc viper_misc2 viper_misc3 viper_misc_intbig viper_ptr16_load viper_ptr16_store viper_ptr32_load viper_ptr32_store viper_ptr8_load viper_ptr8_store viper_storeattr viper_subscr viper_subscr_multi viper_try viper_types viper_unop viper_with
    1 tests failed: machine_spi_rate
    

    Also, doubling the stack in mpconfigport.h has no impact on the crash

  4. andrewleech commented on Aug 23, 2024

    @andrewleech
    SponsorContributor

    Thanks for the report and the reproduction code.
    I've just had it running on one of my C6 boards which is running a older build:

    MicroPython v1.24.0-preview.58.gbe8b352913.dirty
    on 2024-06-26; ESP32C6 module with ESP32C6
    

    I also have a mosquito running so used that, though I had to add user/pass to the config.
    Otherwise I copied async mqtt library from your repo and loaded the script as main.
    I don't have IDF monitor set up so just watched with mpremote over the USB port (not UART)
    After 3 hours it's still running correctly, no crash observed yet.
    I'll upgrade the board to a newer version and try again.

  5. andrewleech commented on Aug 23, 2024

    @andrewleech
    SponsorContributor

    Ok tried again on a somewhat newer version of the PR and still no failure after running for a good 30 minutes.

    drain
    _as_write 109
    drain
    _as_write 109
    drain
    _as_write 109
    drain
    _as_write 109
    drain
    Traceback (most recent call last):
      File "main.py", line 23, in <module>
      File "asyncio/core.py", line 1, in run_until_complete
      File "asyncio/core.py", line 1, in run_until_complete
    KeyboardInterrupt:
    MicroPython v1.24.0-preview.235.g0a8d5b9 on 2024-08-23; ESP32C6 module with ESP32C6
    Type "help()" for more information.
    >>>
    
  6. andrewleech commented on Aug 25, 2024

    @andrewleech
    SponsorContributor

    Just to round out the testing I rebased again to e9814e9 and re-tested, including getting IDF monitor connected instead of mpremote, still no failure seen....

    However I then looked closer at your error message and noted the "rebooting" in the end. Did you monitor stop when the error occurred?

    When I'm running the script it's constantly streaming

    _as_write 109
    drain
    _as_write 109
    drain
    _as_write 109
    

    over and over, so if the error occurs and reboots automatically the error message would be rapidly lost. I'd better try again with the outputs logged to search through.

    Separately, could you check and share the revision of your chip?

    I've got two C6 modules. One was by 01space back when C6 was first announced. Turns out this one crashes for me immediately as soon as wifi connect is called.

    >>> import network
    >>> import asyncio
    >>> sta_if = network.WLAN(network.STA_IF)
    >>> sta_if.active(True)
    True
    >>> sta_if.connect(ssid, pass)
    >>> ESP-ROM:esp32c6-20220919
    Build:Sep 19 2022
    rst:0xc (SW_CPU),boot:0x8 (SPI_FAST_FLASH_BOOT)
    Saved PC:0x4001975a
    SPIWP:0xee
    mode:DIO, clock div:2
    load:0x40875720,len:0xed4
    load:0x4086c410,len:0xc18
    load:0x4086e610,len:0x2b44
    entry 0x4086c410
    MicroPython v1.24.0-preview.216.g8cefcd9432.dirty on 2024-08-24; ESP32C6 module with ESP32C6
    Type "help()" for more information.
    >>>
    

    My other unit is a m5stack nanoc6 and it works fine.

    When you run esptool on the module, the revision is printed out.
    My broken one reports:

    esptool.py v4.7.0
    Serial port /dev/ttyACM1
    Connecting...
    Chip is ESP32-C6 (QFN40) (revision v0.0)
    Features: WiFi 6, BT 5, IEEE802.15.4
    Crystal is 40MHz
    MAC: 40:4c:ca:ff:fe:40:51:e8
    BASE MAC: 40:4c:ca:40:51:e8
    

    Whereas my working one shows:

    esptool.py v4.7.0
    Serial port /dev/ttyACM0
    Connecting...
    Chip is ESP32-C6FH4 (QFN32) (revision v0.1)
    Features: WiFi 6, BT 5, IEEE802.15.4
    Crystal is 40MHz
    MAC: 40:4c:ca:ff:fe:5b:20:94
    BASE MAC: 40:4c:ca:5b:20:94
    MAC_EXT: ff:fe
    
  7. andrewleech commented on Aug 25, 2024

    @andrewleech
    SponsorContributor

    Sorry I was wrong, my (good) unit does crash like your report, it was just lost in the mqtt activity logging.

    A fatal error occurred. The crash dump printed below may be used to help
    determine what caused it. If you are not already running the most recent
    version of MicroPython, consider upgrading. New versions often fix bugs.
    
    To learn more about how to debug and/or report this crash visit the wiki
    page at: https://github.com/micropython/micropython/wiki/ESP32-debugging
    
    MPY version : v1.24.0-preview.216.g8cefcd9432.dirty on 2024-08-25
    IDF version : v5.2.2
    Machine     : ESP32C6 module with ESP32C6
    
    Guru Meditation Error: Core  0 panic'ed (Stack protection fault). 
    
    Detected in task "mp_task" at 0x4200e4fc
    �[0;33m0x4200e4fc: nlr_jump at /home/anl/micropython/py/nlrrv32.c:55
    �[0m
    Stack pointer: 0x4082a9a0
    Stack bounds: 0x40826d5c - 0x4082ad50
    
    Core  0 register dump:
    MEPC    : 0x420196f8  RA      : 0x40803aac  SP      : 0x4082a9a0  GP      : 0x40818e94  
    �[0;33mStack dump detected�[0m
    �[0;33m0x420196f8: mp_obj_is_subclass_fast at /home/anl/micropython/py/objtype.c:1364
    0x40803aac: mp_execute_bytecode at /home/anl/micropython/py/vm.c:1382 (discriminator 1)
    �[0m
    TP      : 0x407eb664  T0      : 0x40030dca  T1      : 0x4081321c  T2      : 0x0000003f  
    
    �[0;33m0x40030dca: memset in ROM
    0x4081321c: vTaskSuspend at /opt/esp/idf/components/freertos/FreeRTOS-Kernel/tasks.c:1977
    �[0m
    S0/FP   : 0x4082e830  S1      : 0x4082e880  A0      : 0x4219d8a4  A1      : 0x4219d8a4  
    A2      : 0x00000000  A3      : 0x4083a210  A4      : 0x4083a210  A5      : 0x4082aa90  
    A6      : 0x00000002  A7      : 0x421a86dc  S2      : 0x421a0288  S3      : 0x4083a200  
    S4      : 0x00000001  S5      : 0x4219e000  S6      : 0x00000068  S7      : 0x4219db64  
    S8      : 0x0000001b  S9      : 0x4219e000  S10     : 0x421a8335  S11     : 0x421a80d2  
    T3      : 0x00000000  T4      : 0x00021ed3  T5      : 0x00000003  T6      : 0x00000001  
    MSTATUS : 0x00001881  MTVEC   : 0x40800001  MCAUSE  : 0x0000001b  MTVAL   : 0x00001101  
    
    �[0;33m0x40800001: _vector_table at ??:?
    �[0m
    MHARTID : 0x00000000  
    
    �[0;33m
    Backtrace:
    �[0m
    mp_obj_is_subclass_fast (object=0x4219d8a4 <mp_type_StopIteration>, classinfo=0x4219d8a4 <mp_type_StopIteration>) at /home/anl/micropython/py/objtype.c:1364
    1364	bool mp_obj_is_subclass_fast(mp_const_obj_t object, mp_const_obj_t classinfo) {
    #0  mp_obj_is_subclass_fast (object=0x4219d8a4 <mp_type_StopIteration>, classinfo=0x4219d8a4 <mp_type_StopIteration>) at /home/anl/micropython/py/objtype.c:1364
    #1  0x40803aac in mp_execute_bytecode (code_state=code_state@entry=0x4082e830, inject_exc=inject_exc@entry=0x0) at /home/anl/micropython/py/vm.c:1382
    #2  0x4201207a in fun_bc_call (self_in=<optimized out>, n_args=1, n_kw=0, args=0x4082ab28) at /home/anl/micropython/py/objfun.c:267
    #3  0x00000000 in ?? ()
    Backtrace stopped: frame did not save the PC
    ELF file SHA256: 84f10135c
    
    Rebooting...
    ESP-ROM:esp32c6-20220919
    Build:Sep 19 2022
    rst:0xc (SW_CPU),boot:0xc (SPI_FAST_FLASH_BOOT)
    Saved PC:0x4001975a
    SPIWP:0xee
    mode:DIO, clock div:2
    load:0x40875720,len:0xed4
    load:0x4086c410,len:0xc18
    load:0x4086e610,len:0x2b44
    entry 0x4086c410
    �[0;33m0x4001975a: software_reset_cpu in ROM
    �[0m
    
  8. andrewleech commented on Aug 26, 2024

    @andrewleech
    SponsorContributor

    Oh and @mzakharocsc yes I can confirm your finding about needing to run it as main.py from startup. The exact same script, not installed as main.py but instead run via mpremote run espc6test.py does not crash - mine has been running now for about 2 hours whereas yes it crashes after a few minutes when run at startup.

    update:
    Well actually the mpremote run did eventually crash, after 57MB worth of

    _as_write 109
    drain
    _as_write 109
    drain
    _as_write 109
    drain
    _as_write 109
    drain
    

    We don't have a full stack trace because this was mpremote rather than monitor, but this is what I have:

    A fatal error occurred. The crash dump printed below may be used to help
    determine what caused it. If you are not already running the most recent
    version of MicroPython, consider upgrading. New versions often fix bugs.
    
    To learn more about how to debug and/or report this crash visit the wiki
    page at: https://github.com/micropython/micropython/wiki/ESP32-debugging
    
    MPY version : v1.24.0-preview.216.g8cefcd9432.dirty on 2024-08-25
    IDF version : v5.2.2
    Machine     : ESP32C6 module with ESP32C6
    
    Guru Meditation Error: Core  0 panic'ed (Stack protection fault).
    
    Detected in task "mp_task" at 0x4200e4fc
    Stack pointer: 0x4082a860
    Stack bounds: 0x40826d5c - 0x4082ad50
    
    
    Core  0 register dump:
    MEPC    : 0x420196f8  RA      : 0x40803aac  SP      : 0x4082a860  GP      : 0x40818e94
    TP      : 0x407eb664  T0      : 0x40022494  T1      : 0x4081321c  T2      : 0x0000003f
    S0/FP   : 0x40853668  S1      : 0x408536a8  A0      : 0x4219d8a4  A1      : 0x4219d8a4
    A2      : 0x00000000  A3      : 0x00000100  A4      : 0x0000000e  A5      : 0x4082a960
    A6      : 0x00000001  A7      : 0x00000002  S2      : 0x421a0288  S3      : 0x40853700
    S4      : 0x0000003c  S5      : 0x4219e000  S6      : 0x00000068  S7      : 0x4219db64
    S8      : 0x4081c000  S9      : 0x4219e000  S10     : 0x421a8398  S11     : 0x421a8393
    T3      : 0x00000000  T4      : 0x00087739  T5      : 0x00000003  T6      : 0x00000001
    MSTATUS : 0x00001881  MTVEC   : 0x40800001  MCAUSE  : 0x0000001b  MTVAL   : 0x00001101
    MHARTID : 0x00000000
    
    Stack memory:
    4082a860: 0x4081c7b4 0x408536a4 0x40853720 0x00000000 0x4081c7b4 0x408536a8 0x00000020 0x00000000
    4082a880: 0x4082a960 0x40853700 0x40803a62 0x40853668 0x408536a8 0x421a0288 0x00000000 0x0000003c
    4082a8a0: 0x421a809c 0x00000068 0x4219db64 0x4081c000 0x4219e000 0x421a8398 0x421a8393 0x4082a860
    4082a8c0: 0x4219e000 0x421a8377 0x421a8371 0x421a8393 0x421a8398 0x4219e000 0x421a8399 0x4219db64
    4082a8e0: 0x00000068 0x421a809c 0x4082a92c 0x00000000 0x00000000 0x4082d3e0 0x40853660 0x4201257a
    4082a900: 0x4219dab8 0x408536c0 0x4082e764 0x00000000 0x421a0288 0x4082e7a0 0x4082e750 0x420126f4
    4082a920: 0x0000008a 0x40853660 0x4082e764 0x42011f0a 0x421a0288 0x4082e7a0 0x4082e750 0x408040b8
    4082a940: 0x4081c7b4 0x4082e79c 0x0000025c 0x00000000 0x00000004 0x4082e7a0 0x0000000f 0x4081c000
    4082a960: 0x4082aa30 0x40853700 0x40803b26 0x4082e750 0x4082e7a0 0x421a0288 0x40853700 0x00000001
    4082a980: 0x00000062 0x00000068 0x4219db64 0x0000001b 0x4219e000 0x421a8335 0x421a80d2 0x4082a940
    4082a9a0: 0x00000001 0x4082c7c0 0x4082aa10 0x421a82b0 0x421a82b8 0x4219e000 0x00000068 0x00000054
    4082a9c0: 0x4082e750 0x0000000f 0x4082aac8 0x00000000 0x00000001 0x4082c7c0 0x4082aa10 0x4201207a
    4082a9e0: 0x00000068 0x421a809c 0x421a82b9 0x4219db64 0x00000068 0x421a809c 0x4082aac4 0x00000000
    4082aa00: 0x421a0288 0x4082aad4 0x4082aab0 0x4080416c 0x4081c7b4 0x4082aad0 0x421997a1 0x00000000
    4082aa20: 0x4082c904 0x4082aac8 0x00000020 0x00000000 0x4082ab30 0x00000006 0x40803a62 0x4082aab0
    4082aa40: 0x4082aad4 0x421a0288 0x00000000 0x00000020 0x00000004 0x00000068 0x4219db64 0x4081c000
    4082aa60: 0x4219e000 0x4082c91c 0x4082c8e2 0x4082aa10 0x4082abc4 0x0000000f 0x4082d030 0x4082c8e2
    4082aa80: 0x4082c91c 0x4219e000 0x00000030 0x00000000 0x4082aab0 0x00000004 0x4082abcc 0x00000000
    4082aaa0: 0x00000001 0x4082c7c0 0x4082ab10 0x4201207a 0x4082cee0 0x421a82b7 0x4082aac0 0x00000004
    4082aac0: 0x4081c810 0x4082ccf0 0x4082e3b0 0x4082d630 0x4082d630 0x4082d410 0x4082ab10 0x42011fce
    4082aae0: 0x00000000 0x00000000 0x4082c91d 0x4219db64 0x00000068 0x4082c950 0x4082abc4 0x00000000
    4082ab00: 0x421a0288 0x4082abd4 0x4082abb0 0x408040b8 0x4081c7b4 0x4082abd0 0x000001d1 0x00000000
    4082ab20: 0x00000041 0x4082abc8 0x0000001e 0x0000000a 0x4082ac30 0x421a9a64 0x40803a62 0x4082abb0
    4082ab40: 0x4082abd4 0x421a0288 0x00000000 0x00000020 0x00000004 0x00000068 0x4219db64 0x4081c000
    4082ab60: 0x4219e000 0x42181000 0x42181000 0x4082ab10 0x421ab3b8 0x7fffffff 0x0000208b 0x42181000
    4082ab80: 0x42181000 0x42181000 0x00000030 0x00000000 0x4082abb0 0x00000004 0x00000000 0x00000000
    4082aba0: 0x00000000 0x4081c810 0x4082ac10 0x4201207a 0x4082c6f0 0x4082c91b 0x4082abc0 0x00000004
    4082abc0: 0x4081c810 0x4082cee0 0x00000000 0x4082d630 0x00002e6a 0x00000001 0x4082ac10 0x42011fce
    4082abe0: 0x4082c880 0x42181000 0x00000080 0x42186000 0x00000041 0x00000005 0x42182000 0x00000004
    4082ac00: 0x00000001 0x4082c6f0 0x00000041 0x4202d248 0x4082ac8c 0x4202d112 0x4202d34c 0x00000001
    4082ac20: 0x00000000 0x00000005 0x4082d0e4 0x00000000 0x00000000 0x00000000 0x4202d1c2 0x00000041
    4082ac40: 0x4082ac94 0x00000001 0x00000004 0x42182000 0x00000005 0x00000041 0x42186000 0x00000080
    
    
    
    ELF file SHA256: 84f10135c
    
    Rebooting...
    ESP-ROM:esp32c6-20220919
    Build:Sep 19 2022
    rst:0xc (SW_CPU),boot:0xc (SPI_FAST_FLASH_BOOT)
    Saved PC:0x4001975a
    SPIWP:0xee
    mode:DIO, clock div:2
    load:0x40875720,len:0xed4
    load:0x4086c410,len:0xc18
    load:0x4086e610,len:0x2b44
    entry 0x4086c410
    MicroPython v1.24.0-preview.216.g8cefcd9432.dirty on 2024-08-25; ESP32C6 module with ESP32C6
    Type "help()" for more information.
    >>> 
    
  9. andrewleech commented on Aug 26, 2024

    @andrewleech
    SponsorContributor

    On a related note, C3 is susceptible to the same issue

    A fatal error occurred. The crash dump printed below may be used to help
    determine what caused it. If you are not already running the most recent
    version of MicroPython, consider upgrading. New versions often fix bugs.
    
    To learn more about how to debug and/or report this crash visit the wiki
    page at: https://github.com/micropython/micropython/wiki/ESP32-debugging
    
    MPY version : v1.24.0-preview.216.g8cefcd9432 on 2024-08-26
    IDF version : v5.2.2
    Machine     : ESP32C3 module with ESP32C3
    
    Guru Meditation Error: Core  0 panic'ed (Stack protection fault). 
    
    Detected in task "mp_task" at 0x4200b1ea
    �[0;33m0x4200b1ea: nlr_jump at /home/anl/micropython/py/nlrrv32.c:55
    �[0m
    Stack pointer: 0x3fca7e60
    Stack bounds: 0x3fca4300 - 0x3fca82f0
    
    Core  0 register dump:
    MEPC    : 0x40382970  RA      : 0x40382936  SP      : 0x3fca7e60  GP      : 0x3fc96e00  
    �[0;33mStack dump detected�[0m
    �[0;33m0x40382970: mp_execute_bytecode at /home/anl/micropython/py/vm.c:1382
    0x40382936: mp_execute_bytecode at /home/anl/micropython/py/vm.c:285
    �[0m
    TP      : 0x3fc6b798  T0      : 0x3fcd4738  T1      : 0x40390f52  T2      : 0x0000003f  
    
    �[0;33m0x40390f52: vTaskSuspend at /opt/esp/idf/components/freertos/FreeRTOS-Kernel/tasks.c:1960 (discriminator 1)
    �[0m
    S0/FP   : 0x3fccd868  S1      : 0x3fccd8a8  A0      : 0x00000001  A1      : 0x00000054  
    A2      : 0x00000000  A3      : 0x00000100  A4      : 0x0000000e  A5      : 0x3fca7f60  
    A6      : 0x00000001  A7      : 0x00000002  S2      : 0x3c170338  S3      : 0x3fccd900  
    S4      : 0x0000003c  S5      : 0x3c16e000  S6      : 0x00000068  S7      : 0x3c16dc08  
    S8      : 0x3fc9a000  S9      : 0x3c16e000  S10     : 0x3c17846c  S11     : 0x3c178467  
    T3      : 0x00000000  T4      : 0x00015cf0  T5      : 0x00000003  T6      : 0x00000001  
    MSTATUS : 0x00001881  MTVEC   : 0x40380001  MCAUSE  : 0x0000001b  MTVAL   : 0x0009a503  
    �[0;33m0x40380001: _vector_table at ??:?
    �[0m
    MHARTID : 0x00000000  
    
    �[0;33m
    Backtrace:
    �[0m
    mp_execute_bytecode (code_state=code_state@entry=0x3fccd868, inject_exc=inject_exc@entry=0x0) at /home/anl/micropython/py/vm.c:1382
    1382	            if (mp_obj_is_subclass_fast(MP_OBJ_FROM_PTR(((mp_obj_base_t*)nlr.ret_val)->type), MP_OBJ_FROM_PTR(&mp_type_StopIteration))) {
    #0  mp_execute_bytecode (code_state=code_state@entry=0x3fccd868, inject_exc=inject_exc@entry=0x0) at /home/anl/micropython/py/vm.c:1382
    #1  0x4200f268 in mp_obj_gen_resume (self_in=0x3fccd860, send_value=<optimized out>, throw_value=throw_value@entry=0x0, ret_val=ret_val@entry=0x3fca7f2c) at /home/anl/micropython/py/objgenerator.c:210
    #2  0x4200f3e2 in gen_resume_and_raise (raise_stop_iteration=true, throw_value=0x0, send_value=<optimized out>, self_in=<optimized out>) at /home/anl/micropython/py/objgenerator.c:259
    #3  gen_instance_send (self_in=<optimized out>, send_value=<optimized out>) at /home/anl/micropython/py/objgenerator.c:286
    #4  0x00000000 in ?? ()
    Backtrace stopped: frame did not save the PC
    ELF file SHA256: 1733dc0f5
    
    Rebooting...
    ESP-ROM:esp32c3-api1-20210207
    

    This was the same code, run the same way.

  10. andrewleech commented on Aug 27, 2024

    @andrewleech
    SponsorContributor

    I then tried running the same script on S2 - got a few hours in without crash there. So perhaps its limited to the risc cores.

  11. projectgus commented on Aug 27, 2024

    @projectgus
    Contributor

    @andrewleech What commit of master branch are these branched from? Asking as I don't seem to have that git hash otherwise, and because a bunch of tweaking for stack on esp32c3 went in recently.

  12. andrewleech commented on Aug 27, 2024

    @andrewleech
    SponsorContributor

    Hi @projectgus yeah those latest test results were from me rebasing my c6 branch onto e9814e9, similar to the original bug report. I had seen your recent stack work and I'm pretty sure it was included in this commit

    I've just retested C3 on latest master a8d1c25 (via idf monitor) and got the same crash:

    A fatal error occurred. The crash dump printed below may be used to help
    determine what caused it. If you are not already running the most recent
    version of MicroPython, consider upgrading. New versions often fix bugs.
    
    To learn more about how to debug and/or report this crash visit the wiki
    page at: https://github.com/micropython/micropython/wiki/ESP32-debugging
    
    MPY version : v1.24.0-preview.230.ga8d1c25a1b.dirty on 2024-08-27
    IDF version : v5.2.2
    Machine     : ESP32C3 module with ESP32C3
    
    Guru Meditation Error: Core  0 panic'ed (Stack protection fault).
    
    Detected in task "mp_task" at 0x4200b1ea
    ESC[0;33m0x4200b1ea: nlr_jump at /home/anl/micropython/py/nlrrv32.c:55
    ESC[0m
    Stack pointer: 0x3fca7f70
    Stack bounds: 0x3fca4408 - 0x3fca8400
    
    
    Core  0 register dump:
    MEPC    : 0x40382970  RA      : 0x40382936  SP      : 0x3fca7f70  GP      : 0x3fc96e00
    ESC[0;33mStack dump detectedESC[0m
    ESC[0;33m0x40382970: mp_execute_bytecode at /home/anl/micropython/py/vm.c:1382
    0x40382936: mp_execute_bytecode at /home/anl/micropython/py/vm.c:285
    ESC[0m
    TP      : 0x3fc6b838  T0      : 0x4005890e  T1      : 0x40390f52  T2      : 0x0000003f
    ESC[0;33m0x4005890e: memset in ROM
    0x40390f52: vTaskSuspend at /opt/esp/idf/components/freertos/FreeRTOS-Kernel/tasks.c:1960 (discriminator 1)
    ESC[0m
    S0/FP   : 0x3fcc7828  S1      : 0x3fcc7860  A0      : 0x00000001  A1      : 0x00000054
    A2      : 0x00000001  A3      : 0x00000019  A4      : 0x00000003  A5      : 0x3fca8070
    A6      : 0x3fca7f48  A7      : 0x3c178820  S2      : 0x3c170390  S3      : 0x3fcaa370
    S4      : 0x00000034  S5      : 0x3c16e000  S6      : 0x00000068  S7      : 0x3c16dc48
    S8      : 0x3fc9a000  S9      : 0x3c16e000  S10     : 0x3c1784dc  S11     : 0x3c1784d7
    T3      : 0x00000000  T4      : 0x0001159b  T5      : 0x00000003  T6      : 0x00000001
    MSTATUS : 0x00001881  MTVEC   : 0x40380001  MCAUSE  : 0x0000001b  MTVAL   : 0x0009a503
    ESC[0;33m0x40380001: _vector_table at ??:?
    ESC[0m
    MHARTID : 0x00000000
    
    ESC[0;33m
    Backtrace:
    
    ESC[0m
    mp_execute_bytecode (code_state=code_state@entry=0x3fcc7828, inject_exc=inject_exc@entry=0x0) at /home/anl/micropython/py/vm.c:1382
    1382                if (mp_obj_is_subclass_fast(MP_OBJ_FROM_PTR(((mp_obj_base_t*)nlr.ret_val)->type), MP_OBJ_FROM_PTR(&mp_type_StopIteration))) {
    #0  mp_execute_bytecode (code_state=code_state@entry=0x3fcc7828, inject_exc=inject_exc@entry=0x0) at /home/anl/micropython/py/vm.c:1382
    #1  0x4200f268 in mp_obj_gen_resume (self_in=0x3fcc7820, send_value=<optimized out>, throw_value=throw_value@entry=0x0, ret_val=ret_val@entry=0x3fca803c) at /home/anl/micropython/py/objgenerator.c:210
    #2  0x4200f3e2 in gen_resume_and_raise (raise_stop_iteration=true, throw_value=0x0, send_value=<optimized out>, self_in=<optimized out>) at /home/anl/micropython/py/objgenerator.c:259
    #3  gen_instance_send (self_in=<optimized out>, send_value=<optimized out>) at /home/anl/micropython/py/objgenerator.c:286
    #4  0x00000000 in ?? ()
    Backtrace stopped: frame did not save the PC
    ELF file SHA256: 23e1a2158
    
    Rebooting...
    ESP-ROM:esp32c3-api1-20210207
    Build:Feb  7 2021
    rst:0xc (RTC_SW_CPU_RST),boot:0xf (SPI_FAST_FLASH_BOOT)
    Saved PC:0x403806b8
    ESC[0;33m0x403806b8: esp_restart_noos at /opt/esp/idf/components/esp_system/port/soc/esp32c3/system_internal.c:111
    ESC[0m
    SPIWP:0xee
    mode:DIO, clock div:1
    load:0x3fcd5820,len:0xf28
    load:0x403cc710,len:0x944
    load:0x403ce710,len:0x2b1c
    entry 0x403cc710
    
  13. andrewleech commented on Aug 27, 2024

    @andrewleech
    SponsorContributor

    Retested C3 on v1.23 and still get the same issue:

    A fatal error occurred. The crash dump printed below may be used to help
    determine what caused it. If you are not already running the most recent
    version of MicroPython, consider upgrading. New versions often fix bugs.
    
    To learn more about how to debug and/or report this crash visit the wiki
    page at: https://github.com/micropython/micropython/wiki/ESP32-debugging
    
    MPY version : v1.23.0-dirty on 2024-08-27
    IDF version : v5.2.2
    Machine     : ESP32C3 module with ESP32C3
    
    Guru Meditation Error: Core  0 panic'ed (Stack protection fault).
    
    Detected in task "mp_task" at 0x400319b4
    ESC[0;33m0x400319b4: longjmp in ROM
    ESC[0m
    ESC[0;33mStack dump detectedESC[0m
    Stack pointer: 0x3fca9180
    Stack bounds: 0x3fca5ad4 - 0x3fca9ad0
    
    
    Core  0 register dump:
    MEPC    : 0x403828e2  RA      : 0x403828a4  SP      : 0x3fca9180  GP      : 0x3fc98200
    ESC[0;33m0x403828e2: mp_execute_bytecode at /home/anl/micropython/py/vm.c:1382
    0x403828a4: mp_execute_bytecode at /home/anl/micropython/py/vm.c:285 (discriminator 2)
    ESC[0m
    TP      : 0x3fc6d104  T0      : 0x3fcc5230  T1      : 0x40392344  T2      : 0x0000003f
    ESC[0;33m0x40392344: vTaskPriorityDisinheritAfterTimeout at /opt/esp/idf/components/freertos/FreeRTOS-Kernel/tasks.c:5261
    ESC[0m
    S0/FP   : 0x3fcb77c0  S1      : 0x3fcab5c0  A0      : 0x00000001  A1      : 0x00000001
    A2      : 0x00000000  A3      : 0x00000100  A4      : 0x0000000e  A5      : 0x3fca9388
    A6      : 0x00000001  A7      : 0x00000002  S2      : 0x00000000  S3      : 0x00000000
    S4      : 0x3fca933c  S5      : 0x0000025a  S6      : 0x3fcad750  S7      : 0x00000054
    S8      : 0x00000068  S9      : 0x3c151000  S10     : 0x3c151000  S11     : 0x3c151000
    T3      : 0x00000000  T4      : 0x000134ff  T5      : 0xc3e88106  T6      : 0x00000001
    MSTATUS : 0x00001881  MTVEC   : 0x40380001  MCAUSE  : 0x0000001b  MTVAL   : 0x00004008
    ESC[0;33m0x40380001: _vector_table at ??:?
    ESC[0m
    MHARTID : 0x00000000
    
    ESC[0;33m
    Backtrace:
    
    ESC[0m
    mp_execute_bytecode (code_state=code_state@entry=0x3fcb7738, inject_exc=inject_exc@entry=0x0) at /home/anl/micropython/py/vm.c:1382
    1382                if (mp_obj_is_subclass_fast(MP_OBJ_FROM_PTR(((mp_obj_base_t*)nlr.ret_val)->type), MP_OBJ_FROM_PTR(&mp_type_StopIteration))) {
    #0  mp_execute_bytecode (code_state=code_state@entry=0x3fcb7738, inject_exc=inject_exc@entry=0x0) at /home/anl/micropython/py/vm.c:1382
    #1  0x420471fc in mp_obj_gen_resume (self_in=0x3fcb7730, send_value=<optimized out>, throw_value=throw_value@entry=0x0, ret_val=ret_val@entry=0x3fca933c) at /home/anl/micropython/py/objgenerator.c:210
    #2  0x4204739a in gen_resume_and_raise (raise_stop_iteration=true, throw_value=0x0, send_value=<optimized out>, self_in=<optimized out>) at /home/anl/micropython/py/objgenerator.c:259
    #3  gen_instance_send (self_in=<optimized out>, send_value=<optimized out>) at /home/anl/micropython/py/objgenerator.c:286
    #4  0x40382f0c in mp_execute_bytecode (code_state=code_state@entry=0x3fcad750, inject_exc=inject_exc@entry=0x0) at /home/anl/micropython/py/vm.c:1042
    #5  0x4200e056 in fun_bc_call (self_in=<optimized out>, n_args=1, n_kw=0, args=0x3fca96b8) at /home/anl/micropython/py/objfun.c:267
    #6  0x40382f94 in mp_execute_bytecode (code_state=0x3fca96a0, inject_exc=<optimized out>) at /home/anl/micropython/py/vm.c:957
    #7  0x00000000 in ?? ()
    Backtrace stopped: frame did not save the PC
    ELF file SHA256: 32d26962d
    
    Rebooting...
    ESP-ROM:esp32c3-api1-20210207
    Build:Feb  7 2021
    rst:0xc (RTC_SW_CPU_RST),boot:0xf (SPI_FAST_FLASH_BOOT)
    Saved PC:0x4038061c
    SPIWP:0xee
    mode:DIO, clock div:1
    load:0x3fcd5820,len:0xf28
    load:0x403cc710,len:0x89c
    load:0x403ce710,len:0x2b14
    entry 0x403cc710
    ESC[0;33m0x4038061c: esp_restart_noos at /opt/esp/idf/components/esp_system/port/soc/esp32c3/system_internal.c:111
    ESC[0m
    
  14. 4 remaining items

  15. projectgus commented on Aug 28, 2024

    @projectgus
    Contributor

    I'm pretty sure this is an ESP-IDF or ESP32-C3 issue because the stack pointer stays valid the whole time, from the point of view of our task. (i.e. hardware stack protector is supposed to trigger if the stack pointer is outside the stack bounds for the task, but all these crash dumps show it inside the stack bounds.) Have raised the ESP-IDF issue linked above which has a bunch more details.

    If we don't want to wait for an upstream fix, we can workaround:

    1. As noted above, disabling the hardware stack protector (which I think was only implemented in v5.1) will fix it. I think we would still have the same stack watchpoint protection that we have on the Xtensa chips.
    2. I strongly suspect implementing a critical section in nlr_jump so we disable interrupts while updating the stack pointer will also work around this, but that is unconfirmed.
  16. added this to the release-1.24.0 milestone on Sep 3, 2024
  17. dpgeorge commented on Oct 11, 2024

    @dpgeorge
    Member

    According to #11869 (comment) this also needs fixing for ESP32-C6.

  18. dpgeorge commented on Oct 11, 2024

    @dpgeorge
    Member

    this also needs fixing for ESP32-C6.

    I'm able to reproduce the original bug reported here using ESP32_GENERIC_C6 firmware, on current master 82e69df, with the code https://github.com/mzakharocsc/micropython/blob/c6/ports/esp32/modules/main.py . Actually to reproduce it I needed to freeze both main.py and mqtt_async.py from that repo into the C6 firmware (running those files from the filesystem, I couldn't reproduce the issue).

  19. dpgeorge commented on Oct 11, 2024

    @dpgeorge
    Member

    See #15997 for a fix for C6.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions