Repository navigation
Save RAM: emit code directly to flash/filesystem/readonly-memory #4073
Description
Activity
What is the use case? Is it that of an application where large scripts are passed to it at runtime for compilation and running?
In normal development I've yet to encounter a problem not solved by splitting the code into smaller modules or by cross-compilation - the latter with or without freezing.
While I have little knowledge of the compilation phase I'd be concerned about flash write speed and wearout.
Third party submits a
.pycontaining huge array of coefficients. They cannot be frozen-mpy-d because the firmware is already built.To save RAM, the
.pycan be written to flash as it’s being submitted by third party ie string-frozen-dBut compilation still results in those coefficients being replicated in the compilation artifacts which are in RAM.
Managing the life of the readonly-memory in use is not in the scope of this issue or even micropython. I haven’t seen any micropython module that provides an interface for flash operations.
The interface to be used for this flash writing would be same as used in
persistentcode.c: an object of typemp_print_t. It is in that implementation where things like flash wear-leveling would be taken care of.The only constraint on the non-volatile memory is that it is memory mapped so that reading it is no different than reading RAM. If the system has a separate flash chip that requires a function call to read, that will not work because in C, it’s not possible to overload the dereferencing operator.
You would still have the option of “normal” compilation and saving the compilation artifacts to RAM.
Have you considered submitting the coefficients as a binary file? You could then use Python's file random access to retrieve individual coefficients on demand.
That was just an example. Even if you don’t have a large amount of coefficients but a large amount of code, the bytecode could still consume lots of RAM.
I know you could split up the file but that requires the third party to do it. If you have an application that accepts random
.pyto execute and you say “Error: file too big, heap overflow by 3728 bytes” … what is the user supposed to do with that?Yes you could also run out of flash if you implement this “save artifacts to flash” but at that point you would have done all you possibly could.
Also flash is usually at least 10x the size of RAM and even PCs have a finite amount of storage: you COULD write a program whose source and/or compilation artifacts take up all RAM and HDD.
If you run out of non-RAM storage on PC or micro controller, you either wrote terrible code or need to upgrade your hw or are using wrong hw to solve your problem.
From a usability perspective, getting rid of the need for users to know about and run
mpy-crosson their .py files and that they should manually compile and place .mpy files on device instead of just saving .py files directly to the flash on the device would be a nice win.Effectively this would be micropython automatically mpy-cross'ing .py files on device (assuming space was available). Such a feature opens the same cans of worms that .pyc files have had with CPython: guaranteeing that the .py and .mpy file are in sync. But if you do that with a hash (similar to what we implemented in https://www.python.org/dev/peps/pep-0552/), it could still work. With an impact on startup time to verify the hash when both .py and .mpy are both present.
A downside? Such a feature is more interesting on smaller the devices (the opposite of where microcontrollers are headed: plentiful flash and ram) but would increase micropython code size a bit (it's already a tight squeeze on many M0s)
micropython automatically mpy-cross'ing .py files on device
A major use for mpy-cross is where the target device has insufficient RAM to compile the file (but has enough to run it). In that instance the cross-compilation must be done on the PC.
There is also merit in storing at least the smaller files (main.py and boot.py in particular) as .py files: it is very convenient to use rshell to edit these in-place.
Ah yes, i've run into that limit on M0s as well. It sounds like there may be merit to doing this, but it is more likely to show up as a user friendliness feature on larger class devices.
I see a feature like this as being related to #4124 to natively execute mpy bytecode from flash - but it isn't clear if that'll be a reasonable thing to implement.
Saving RAM is a constant theme for people using micropython.
A super-simple action a user can take to save RAM is to use frozen bytecode but that's only possible if the
.pyis available to be compiled in with the firmware.If that's not an option, the user can still save the
.pyto flash to at least not occupy RAM and simply pass the pointer-to-flash tomp_lexer_t *mp_lexer_new_from_str_len()But this still leaves a huge source of (unnecessary) RAM consumption during the compilation process and the eventually output.
SUGGESTION: the memory used during the compilation process that is O(1), with respect to the size of the input, should remain in RAM but the O(N) stuff should be optionally placed in flash/nonvolatile memory.