Repository navigation
mpy-cross: argument heapsize / why two mpy-cross / in case of errors, emit the Python source code line #11664
Description
Activity
BTW... why do exist two tools for the exact same purpose?
Neither of these packages are "official" but rather provided by the community. I'm responsible for
mpy-cross-v6.1. I'm not happy about the duplication of effort and potential confusion that has resulted, but we had a technical requirement to have multiple versions ofmpy-crossin a single Python application and this was simply not technically possible using thempy-crosspackage (you can't install multiple versions of the same package at the same time). To solve it, we decided to create multiple packages, one for each MPY ABI version so they could be installed at the same time.A first step to an "official"
mpy-crossPyPI package has been proposed in #10834 so this thread of the discussion is probably best continued there.If I could start over again, I would propose a single Python package that includes multiple copies of the
mpy-crossexecutable for the various MPY ABI versions. This could be useful for tools like mpremote and rshell that might want to be able to target older and newer versions of MicroPython at the same time.Reacted by massimosalaInvoking mpy-cross with
-X heapsize=65536, both fail withAre you compiling a large program? Why do you need to limit the heap size when compiling?
Invoking mpy-cross with
-X heapsize=65536, both fail withAre you compiling a large program? Why do you need to limit the heap size when compiling?
I thought
-X heap size=...was to check the requirement of the .mpy files against the micro's resources, before loading and running the program.If I've misunderstood, please explain the functionality of this option.
Thanks
The option controls the heap size of the
mpy-crossprocess itself.There isn't a way to know how much a program could possibly allocate since Python is a dynamic language.
There isn't a way to know how much a program could possibly allocate since Python is a dynamic language.
I thought the option was useful for setting an upper limit, not the exact runtime usage.
I thought mpy-cross can compute the value by parsing variable declarations.We're talking about a "cross" build tool, I don't care about resource usage on the OS where I create the mpy files. I don't see the use of it.
I'm interested in being able to check the compiled files against the limits of the micro on which the program will be executed.
If the value is just to limit the heap used by mpy-cross on linux / windows... I don't see the point.
Sorry for the misunderstading.A first step to an "official"
mpy-crossPyPI package has been proposed in #10834 so this thread of the discussion is probably best continued there.If I could start over again, I would propose a single Python package that includes multiple copies of the
mpy-crossexecutable for the various MPY ABI versions. This could be useful for tools like mpremote and rshell that might want to be able to target older and newer versions of MicroPython at the same time.@dlech Yes sorry for the delay on this. I completely agree and this is exactly what I want to do. I've had other priorities in the past few months that has taken me away from MicroPython work but hope to have something ready soon. In particular I think I have a much simpler solution to the whole native dependency thing.
Summary for @massimosala : I have spoken to the owner of the
mpy-crossPyPI package (hi @andrewleech :D ) and our plan is for the micropython project to take that over in the near future, and we intend to make it support multiple bytecode targets. At which point thempy-cross-vXpackages will hopefully become obsolete and can maybe be marked as such (but likely will stick around for backwards compatibility).I'm interested in being able to check the compiled files against the limits of the micro on which the program will be executed.
This is not possible. As David pointed out, the compiler has no idea what your code is doing. A trivial example:
x = int(input("How much should I allocate? ")) y = bytearray(x)
As I explained in that forum post that you linked to above, mpy-cross is just straight-up regular MicroPython, so it has a heap, and therefore it can be configured. There's no point in making this (essentially) infinite because any reasonable Python file is going to be fine with the default.
However, there is one way that this maps to the target device which is that in order for a device to run a piece of code it needs heap for both:
- compiling the code
- executing the code
You're talking about the second, but it's (very) occasionally useful to set the mpy-cross heap limit down so that it matches the ability of the target device to compile code. But on the other hand, if you're using mpy-cross you don't care about the target device's compiler.
If the value is just to limit the heap used by mpy-cross on linux / windows... I don't see the point.
That's OK. Just ignore it. It's there for people who need it. We could improve the
--helpoutput, and as always the documentation could be better, but I don't think anything misrepresents its purpose.Invoking mpy-cross with -X heapsize=65536, both fail with
MemoryError: memory allocation failed, allocating 832 bytes@massimosala This is the real question though. As David asked, what is the .py file you're trying to cross compile. The default is 1MiB, so it's no surprise it also fails at 64kiB. This must be a very large file.
Reacted by massimosalaJim, David, thanks for you patience.
I misunderstood the option ...
Sometimes after 30 years of career in IT I still discover software with magic features !At least my questions started something positive: your discussion on how to improve and unify the tool.
Regarding the compilation, these are the files:
4.104 basic.py 659 boot.py 598 compatibility.py 4.264 crc_generic.py 4.118 crc_speed_32.py 3.025 crc_tests.py 4.332 crc_viper_32.py 1.474 files.py 4.825 fw_update.py 287 main.py 17.451 main2.py 4.920 mdns_announcer.py 539 mimes.py 6.831 nrequests.py 5.059 ntptime2.py 1.745 oled.py 1.478 remote_print.py 7.190 tftp.py 477 version.py 5.981 web_server.py 2.435 wifi.pyThe 4 CRC files are just for stand-alone tests.
The program I am working on is made by the other files.I set now the 64 KB limit and redo the cross-compile, so I can tell you which is the offending file.
And the winner is...Compiling main2.mpy
MemoryError: memory allocation failed, allocating 2032 bytes
Cannot mpy-cross main2.pyBTW, in the CRC files I use viper. Not in the other files.
The execution of the program on esp8266 is:
boot.py --> main.py --> main2.mpy ... main2 will import the other modules, all cross-compiled to mpy.
From main2, it and all the next modules as first instruction do
from compatibility import *While developing I run the program on my pc.
Here is compatibility.pyfrom gc import collect from os import unlink from time import time try : from micropython import const MICRO = True from gc import mem_free except : # other pythons const = lambda x: x MICRO = False ASCII = const("ascii") UTF8 = const("utf-8") IP4_ANY = const("0.0.0.0") BUFFER_OVERRUN = const("!!! INTERNAL ERROR: buffer overrun: req %d bytes") def gc_collect(report = None) : collect() if MICRO and report : print(report, "mem_free", mem_free()) def printf(*args, **kwargs) : print(int(time()), *args, **kwargs) gc_collect()(ops... I see I left a useless unlink)
Feel free to ask me other infos, if you want to troubleshoot this.
CiaoOne last note: often mpy-cross aborts with cryptic error messages.
Program: C:\TOOLS\PYTHON38\lib\site-packages\mpy_cross\mpy-cross.exe
File: ../py/emitnative.c, Line 1470Expression: vtype_base == VTYPE_PYOBJ
Cannot mpy-cross crc_micropython_16.pyI hope the new mpy-cross will report the line of the python source code.
This is essential for troubleshooting.Especially when using decorators (native and viper), where there are already other difficulties.
- changed the title
[-]mpy-cross: possible bug passing the argument heapsize / why two mpy-cross?[/-][+]mpy-cross: argument heapsize / why two mpy-cross / in case of errors, emit the Python source code line[/+]on Jun 5, 2023 The files main2.py is about 17 Kb.
This seems like it's working as intended then!
mpy-cross's heap size has no bearing or relation on the target device's capabilities. 64k is not enough heap for mpy-cross to compile this file. (Important note -- mpy-cross doesn't have the garbage collector enabled, so it uses more heap than it really needs). If it works with the default heap size then I'm not sure what the problem is.
One last note: often mpy-cross aborts with cryptic error messages.
This means it's hitting an unimplemented feature in the native/viper emitter. (In this case I would guess it's viper). For code size there are also a few places where doing certain things are invalid and undefined (you're on your own a bit with viper). My guess in this case is that your code is trying to call a method on something that isn't an object (i.e. it's a viper type). Would need to see the code to debug further.
As suggested, I close this issue.
Pls read the discussion at
#10834
MicroPython v1.20.0 on 2023-04-26; ESP module with ESP8266
I want to enforce the heap size check at compilation time.
I tried with the two Python tools:
Invoking mpy-cross with
-X heapsize=65536, both fail withWithout the argument, they create the .mpy files.
Uploaded the files to the esp8266, the program is running.
From https://forum.micropython.org/viewtopic.php?t=8243
Hmm we have something different between the mpy-cross calculations and the firmware ones.
BTW... why do exist two tools for the exact same purpose?
They have the same options.
They have the same error messages (sometimes a little cryptic).
Isn't it better for the community to have just one mpy-cross?
IHMO if the authors pool their work, they have more resources to improve it.
Sorry if I'm missing some particular reason... or if I'm pissing out of my pot!