Allocate less micropython heap and make more space for IDF heap in ESP32-S2 port

In the current micropython git master we get following memory in an ordinary ESP32 without SPI RAM right after boot:

>>> esp32.idf_heap_info(esp32.HEAP_DATA)
[(7288, 176, 0, 176), (16648, 5608, 4096, 40), (94064, 5684, 4096, 5664), (15072, 13448, 8192, 13448), (113840, 112216, 65536, 112216)]
>>> micropython.mem_info()
stack: 704 out of 15360
GC: total: 64000, used: 1504, free: 62496
 No. of 1-blocks: 20, 2-blocks: 15, max blk sz: 18, max free sz: 3894

so we have 62496 bytes available for python, 137132 for C malloc()

The changed memory layout of ESP32-S2 changes the ratio strongly in favour of micropython heap:

>>> esp32.idf_heap_info(esp32.HEAP_DATA)
[(8148, 140, 0, 140), (197736, 38728, 32768, 34264), (14864, 13108, 8192, 13108)]
>>> micropython.mem_info()
stack: 704 out of 15360
GC: total: 128000, used: 2640, free: 125360
 No. of 1-blocks: 39, 2-blocks: 18, max blk sz: 32, max free sz: 7829

So on S2 we have 125360 bytes available for python and only 51976 bytes in IDF heaps for C malloc()
This gets exhausted just by connecting WiFi (STA mode with WPA2 PSK) and WebREPL request from network.

IMO we need to adjust mp_task_heap_size at ports/esp32/main.c:124

size_t mp_task_heap_size = heap_caps_get_largest_free_block(MALLOC_CAP_8BIT);

to better reflect the lower total amount of memory.
Something like

size_t mp_task_heap_size = MIN(heap_caps_get_largest_free_block(MALLOC_CAP_8BIT),
                                          heap_caps_get_total_size(MALLOC_CAP_8BIT) / 3);

It does not change allocation on ESP32. On ESP32-S2 it allocates 71872 for python/109464 free for malloc, it's enough for WebREPL and upip.install('logging') works too. idf_heap_info() shows minimal seen size 25868, so we have some margin.
Or use some other percentage of the total size?
What do you think?

This might be fixed by #7163.

#7163 fixes it only for devices with SPIRAM, because it will then use SPIRAM for the heap.

@dpgeorge

Still no progress :-(
Adding reference to #6785, an interesting extension of this.

@dpgeorge @robert-hh I'd like to make a PR, but I like to get some comments on proposed heap size calculation.

@tom-van

This issue specifically mentions ESP32-S2, but I think we need to start with plain ESP32 for which:

on older IDF version (eg v4.0.2, with ESP32), the amount of MicroPython heap is 111168 bytes, and the IDF is left with 88502 bytes
recent IDF versions on ESP32 give only 64000 MicroPython heap (as mentioned above) and already break existing user code, because that code expects around 110k of heap

So there's already something there to fix, just dealing with different IDF versions.

The things that make it difficult are:

different IDF versions have a different layout for the heap, some have more contiguous heap than others
different IDF versions use more or less heap for their own internal operations
different chip variants (S2, S3, C3, etc) have different amounts of total memory
different user applications require different amounts of memory for MicroPython heap vs internal things (like wifi buffers, BLE)

To solve (1) we need #3580, then the MicroPython heap can take as much or as little heap as it likes, it doesn't matter how much it's fragmented within the IDF (to an extent).

To solve (4) we need a way for the user to specify the amount of heap to allocate, eg #6785. An alternative to #6785 would be to create an initial small heap (say 8k) which is used to run boot.py, and within boot.py the user must allocate a second, larger heap segment, eg esp32.allocate_heap(50000). Then both heaps are available from then on.

To solve (2) and (3) we need a sensible default value (or ratio) for MicroPython heap vs IDF free heap vs IDF used heap (already used by the time MicroPython starts). As proposed above by @tom-van, the value of 1/3 for MicroPython heap out of total heap does seem like a good sensible value, although it does give less heap than the original 111168 that ESP32 used to have.

A lot of this was already mentioned in #7963, so I guess this is just a summary of things.

We don't need to solve everything now. I think the idea proposed above of 1/3 is very reasonable and a good step forward.

esp32/main: Allocate at most 1/2 of available IDF heap for MP heap.

In 23b1a4e I made it so that MicroPython allocates at most 1/2 of the total IDF heap. This makes sure 111168 bytes are still available on standard ESP32 with IDF 4.2.

@dpgeorge

@dpgeorge thanks for addressing the issue!

I confirm that 23b1a4e makes webrepl working on GENERIC_S2
Unfortunately IDF heap on S2 is still too small for using TLS, so e.g. upip fails:

>>> import upip
>>> upip.install('logging')
Installing to: /lib/
Error installing 'logging': [Errno 12] ENOMEM, packages may be partially installed

The good point the problem is correctly reported, no cryptic WiFi disconnect happens this time.
Surprisingly upip install goes well on GENERIC_C3

Just for those who want TLS functionality on S2 now I refer to fine tuned heap allocation I was working on
tom-van@07c1ef8
Sorry, due to lack of time I was able to test it only on ESP32 and S2, IDF 4.0.2 and 4.4

I'm looking forward to some dynamic way to partition the heap.

esp32/main: Allocate at most 1/2 of available IDF heap for MP heap.

Navigation Menu

Repository navigation

Allocate less micropython heap and make more space for IDF heap in ESP32-S2 port #7214

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Allocate less micropython heap and make more space for IDF heap in ESP32-S2 port #7214

Description

Activity

dpgeorge commented on May 4, 2021

robert-hh commented on May 4, 2021

tom-van commented on Dec 15, 2021

dpgeorge commented on Dec 15, 2021

dpgeorge commented on Jan 21, 2022

tom-van commented on Feb 17, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions