Repository navigation
Compile Python Scripts On the Go #2709
Description
Activity
Yes this is entirely possible, and definitely a good idea. A lot of the building blocks are already in place but it would still take some effort to implement properly. Feel free to propose how a user would use this feature, eg what functions they would call to dynamically freeze a script, where it would go in flash, etc.
I believe there is a Python 3 Module for this. I imagine we should implement this following the same as python 3. I have never used the module, but i think it does what ive mentioned above.
py_compile (https://docs.python.org/3/library/py_compile.html)
Following this module you would call something like this:
py_compile.compile(sourceFile, outputFile)
where the prams are string locations to files on the filing system eg '/folder/file.py'On another note, in the documentation in Built-in Functions there is a 'compile()' function that hasn't been implemented on the esp8266 at lest (i only run MP on that)?. Not sure what the story is about that built in function?
It would be nice to see this come to life :)
Thanks for your interest.Or could we incorporate the "MicroPython cross compiler" or something alike that gets the job done?
Reacted by Bishwas MishraI use frozen bytecode and mpy-cross most of the time when the code is too large to get compiled on board. Embedding the compiler into the board would not help in that case. The compile pass would still run out of memory, I guess.
And since most, if not all builds can execute pre-compiled code from the file system, providing mpy-cross variants in the download section of MicroPython would do the same.I do understand that you can use the MicroPython cross compiler to achieve a similar outcome. Buts thats not the vision. The vision is, to achieve compiled python code without having to create a custom firmware. The Vision of only ever using the pre built MicroPython Firmware, but having the functionality of custom firmware.
This means for newbies and basic level programmers , dont have to touch the Toolchain and SDK to achieve advantages of precompiled scripts. Plus the compiler is already there, why not use it.
Also to my understanding, scripts that are interpreted (.py) get compiled when imported, this takes up ram temporarily during the compilation, but also once compiled the compiled version of the script lives in ram, which uses that ram till it is 'unloaded', on top of that the compiled script uses more ram as it executes and starts using resources (normal stuff).
The key idea here is to only compile once, and execute from flash. So the only ram being used the the normal ram used by execution (if any of that makes sense :P). So as you can see, this is a functional and usability feature, that i can imagine many can benefit from who use micropython.
Extending this functionality, it would be cool to see tools like ampy (https://github.com/adafruit/ampy) and webrepl, implent this as an aditional option to instead of only uploading a .py module, take that .py module compile it on the board and save it to the flash. The .py module is never saved to your board. Quick and Easy. Then we all could share and upload new module when ever and never needing to reflash the board just to use one new module.
Hopefully you guys also see my vision of how easy and useful this would be :).
Reacted by Julian Todd and adritium@timpur There is a second use-case which @robert-hh is referring to. That is the case where a module is so large that it cannot be compiled by the compiler on the target hardware because the compiler runs out of RAM. This code size limit is soon reached on RAM challenged targets like the ESP8266. On the Pyboard it typically occurs at about 1K LOC. In this case the only option is to cross-compile it.
This can currently be done in two ways, one being (as you're aware) to incorporate it into a firmware build as frozen bytecode. The second way is to cross compile it to a .mpy file, transfer that to the filesystem of the target, and execute it from there. Cross compiling it simply involves running the command line tool mpy-cross. However in this case the code is loaded into RAM for execution.
The ideal solution would cope with this use-case too, somehow cross-compiling the code before putting it into flash in such a way that it could be executed from there. I can't quite envisage how this would proceed but perhaps you have some ideas?
You suggest to store compiled objects, similar to .mpy files created by mpy-cross, in flash in a format suitable for direct execution. I see one aspect where you suggestion definitely helps: boards without a file system, like the 512k esp8266 or . But these are typically also short of flash memory. About RAM: the question is, where code that is too large to fit in RAM for execution can be compiled on-board. At least that time it must fit into the RAM, even if it is stored in flash after that.
In your suggestion I see however the opportunity to pre-compile native, viper and assembler code too, because for these the absolute storage addresses can already be determined during compile time.
About re-flash: storing a module to flash is a (partial) re-flash. What about storing multiple modules? Freeing space? Would that require a re-flash? Maybe it's possible to find a way for executing modules out of the file system, given that they are stored properly = sequentially.In general, the filesystem can't guarantee that sequential blocks in the file are sequential on the filesystem.
You would also only be able to use this for a fileystem on internal flash. sdcard or spi-flash based filesystems can't be executed from directly.
@robert-hh Arm Thumb machine code is position independent is it not?
Before frozen bytecode was available I did some tests with a super primitive ROM 'filesystem' that allowed to use the idle flash space. However, such a fs image has to be compiled and downloaded. This is easy in the lab but for general usage there has to be an adequate infrastructure (flash erase, download, validation & programming, not to mention access locks etc.).
There is a chance that I have to reanimate this in the near future but I have no clue how this might fit for system architectures other than ARM (eg. where direct execution from flash is possible).So I can imagine two possible use cases for the on-board compile:
- w/o files: compile a script during loading and load it into flash into the space, where frozen bytcode resides. The dictionary would have to be extended, and some kind of space management would be required (startof free ares + length or remaining space). That would be extend-only. Erasing by re-flash of the device, Advantage for the user: Just a raw repl access needed, which woudl be possible for almost any platform/any device.
- with files: Compile on-board from a .py file or raw code coming in though the terminal interface into a .mpy file. The advantage for the user: no need for a cross compiler on the external platform.
In any case, the compile could support also platform specific properties like compile of native/viper/assembler code), which b.t.w. the cross compiler could already support now during creating of aflash image.. The drawback would be the same: large code would still have to be compiled externally and loaded as .mpy file or embeded into the build, and so the biggest advantage of frozen bytecode would still not be achieved, at least for me.
P.S.: Did someone ever think of making the frozen objects at least visible, like as read-only special objects in the file system?I like the idea - it's a nice addition to frozen byte code and makes it updateable in the field (that's in my opinion the target use-case here). It makes sense not only for ressource constraint systems, but also to update mission critical modules (which shouldn't be part of the potential insecure file system).
Regarding space management: One of the advantages of frozen bytecode is its secureness (no data corruption possible). It would be nice if the update process is done in a way that there is little to no chance of data corruption.
A simplified update approach may be to use bank swapping for the frozen bytecode flash segment.
The idea would be to copy the frozen segement from bank A to bank B (or vice versa). While copying, patch/add the bytecode for the module that is being updated. Maybe some pointers in dictionaries must be updated too - i'm not familiar with the internas here. Since read and write address are on different memory banks, there is no need to buffer the content in RAM.
This way the risk of a corrupted state should be minimized since you would enable the target memory bank only if the copying/patching is successful.
The obvious downside is that you'll need a spare flash bank. On the F4 I believe that's in the 64k range.
On the F4 most of the the pages are 128K (except for the 16K/64K ones used by the filesystem).
The way that NOR flash works is that when you erase a page, the entire page becomes 0xff's. You can then change bits to zero by writing the flash. Once a bit has been changed to a zero, the only way to get back to a 1 is to erase the entire page again.
This means that you can erase the page to all 0xff's, and then keep appending modules until you fill the page. Each byte-code segment should have a header. The header should probably be 4 bytes, of which 3 bytes could be a length and one byte is an in-use marker. If the in-use marker is 0x00, then this module should be ignored. If the in-use marker is something else (say 0xEE) then it means that a valid entry is present. There should also be some type of CRC or checksum.
You'd write a new segment by ensuring the region you're going to write to is all 0xFF's. Then write the 0xEE/length, data, and CRC/checksum.
If you want to "delete" a module, you change the 0xEE to 0x00.
You could then use all of the flash from the end of the firmware to the end of flash, by appending modules, until you fill it up, and then you have to start over by erasing the flash and rewriting the firmware. If you had an SDcard (or a spare flash block), you could compact out the not-in-use modules.
Adding in failsafes (to ensure that everything is always valid even if the power is removed halfway during a flash write requires at least a spare flash block and probably some other compromises as well (like not allowing a bytecode segment to cross a 128K boundary).
Here's my 2c, after I saw @ShrimpingIt fighting over this issue for the last week, which resulted in building the entire esp8266 toolchain (in addition to the cross-compiler) to get frozen modules, which he things (though I don't know) consume less RAM than mpy files. Myself, I'm having to use just the cross-compiler, which is bad enough.
These devices we are using are very RAM constrained, but they've got way too much flash memory (it takes days to fill it up even with logging data sensors), so we're fine with solutions that waste it. A whole separate 0.5Mb program that compiled to mpy modules and embedded each one into its own individual page of flash within the main interpreter, so that the reset button alternately ran this "install system" or the real system, which meant you had to press it twice whenever you uploaded new code -- would be preferable than having no choice than to install the toolchain on a PC and insert an awkward command line operation between editing and uploading your python text files.
It would be a whole lot less disruptive for people who had hit the RAM limit and had gotten used to the convenience and debuggability of just running scripting languages on a microcontroller. (An absolute size limit on python modules due to the limitations of this on-board compiler is less of an issue, because you can usually work round it by breaking these files down.)
Maybe later we can think about merging such a development tool seamlessly into the main interpreter so no one even noticed it.
Here is a small test implementation for the ESP8266, a proof of concept if you will:
master...aykevl:mpy-to-flashThe idea is to use the flash space between the end of the .text section (.irom0.text) and the start of the FAT32 filesystem to store imported .mpy files.
TODO + ideas:
- enable via a #define
- limit it's size to the available size
- do not overwrite on every restart
- 'overflow' to RAM if flash is full
- qstrs to flash?
- only store specific modules in flash, not all .mpy files?
- store regular .py files in flash?
- ...
Reacted by Tim Purchas33 remaining items
@slavaza There is already a workflow to freeze
.pyfiles so that they can be executed from flash.Look in the makefiles for a recipe that contains
mpy-tool; the output is.mpywhich is the same structure as the typemp_raw_code_t.Next step is to turn
.mpyinto a c-file; this is done by a.pyutility (I forget the name but it’s in the makefile).A straightforward strategy is to replicate the code that produces this c-file.
The trick here is the linking of all the tables in the frozen-c file to the rest of the code.
What the linker would normally do in terms of address resolution, you’ll have to do.Create a simple
.pyfile - like “a=a+1” - and go through the workflow that produces the frozen-c file.
You’ll see all the tables that are produced and how they’re linked to the rest of the code.I have some code running on an ARM platform that can relocate an
mp_raw_code_tobject to flash, including all references. I have modified the QSTR code to support a second QSTR pool in flash after the static pool embedded in the firmware image.My current process runs outside of the REPL and runs the following steps:
- perform a soft reboot
- creates a QSTR pool in RAM (outside of heap) to match the size allocated in flash
- compile a block of Python code with
mp_compile_to_raw_code() - relocate compiled code from heap to addressable flash
- write QSTR pool to addressable flash
- perform another soft reboot
At this point, I can run that code by passing the entry point to
mp_make_function_from_raw_code()and calling the generated function withmp_call_function_0().I'm about to modify that code to take a list of
.mpyfiles and embed them in addressable flash (along with their QSTRs) in such a way that mimics the use ofmpy-toolto convert a.mpyfile to C code for embedding in the firmware (aka a frozen mpy).And just to summarize the reasons for doing this: on a low-RAM device, it's useful to have
.mpyfiles in addressable flash such that animportof the module requires a limited amount of heap. By implementing it in this fashion, you no longer have to embed the.mpyfile in the firmware image itself -- you can manage it on the device.I've tried to write the current code in a way that splits the HAL layer for flash erase/write from the "bundling" layer that walks the
mp_raw_code_tstructure and relocates it to flash. I'm doing this work on a closed platform, but can share the generalized relocation code upstream.Since I implemented
os.compile()to convert.pyto.mpyon the device (useful when breaking a program up into multiple modules that you can compile separately with a given heap size), I've planned to prototype this asos.bundle()since that's the term I came up with when I originally wrote the code over a year ago. Maybe I should call itmicropython.freeze()instead?@tomlogic I've study a little about the operations performed by the mp_make_function_from_raw_code () function. I think the creation of a version of this function that can place the runtime structures directly into flash will be the best solution. This will be quite sufficient for a remote upgrade. However, this is not easy.
Has there been any progress on this? This would be helpful for over the air updates. I have a project where I allow multiple versions of the code to be installed and the user can switch between them. This is done by keeping each version of the code in a separate directory. In
main.py, the directory of the desired code version is added tosys.pathand the entry point function is executed. The problem is that the code resides in the filesystem and even if it's pre-compiled to.mpy, the modules still get loaded to RAM, which then quickly gets exhausted. Being able to execute the modules directly from flash would be very beneficial in this case.@janpom, I have working code, but I'm afraid I haven't generalized it enough to be of use elsewhere. If someone is genuinely interested in exploring it, they should contact me about getting a copy. Currently, I have a MicroPython method that manages an area of addressable flash on the host processor, and can erase that area and then write a QSTR table and execute-in-place versions of the code from multiple .mpy files. We use that as a way to get more executable code on the device without having it fill the heap.
I'll see if I can create a branch that includes my code, along with instructions on what's missing to get it working on a given platform.
Reacted by Jan PomikálekLinking frozen Python modules by the C language linker using direct addressing is a beautiful idea, but it making impossible to runtime decomposition them. Perhaps we should not break this effective conciseness.
@slavaza I don't think anyone is talking about removing that feature (frozen python modules at compile time) but looking at exploring ways to add Python modules at run time. In my use case, MicroPython runs as a separate task of a closed-source firmware image. We want customers to have the ability to embed their Python modules in a way that reduces RAM usage. Relocating them to addressable flash, along with their QSTRs is a working solution.
Its been a while since this issue has been updated, can anyone provide a updated status on the subject of this issue?
Is it possible to achieve this now? or is this still something that is not implemented?
I would like to use this to be able to build OTA functionality (so freeze in firmware is a no go) I already use .mpy via mpy-cross, but still hitting memory limits. As far as I understand this issue would directly solve that problem by running .mpy from flash.
Thanks :)
As far as I understand this issue would directly solve that problem by running .mpy from flash.
Most of this issue's earlier discussion seems to be about producing a .mpy file on the device. The issue is that the .mpy file is still on the filesystem, and while this is better than loading a .py file from the filesystem, it still loads the bytecode into RAM to execute.
As pointed out in a few comments above, frozen code is special because it can be memory mapped directly from flash. So the direction that MicroPython is headed is to provide a way to memory map from the filesystem. Damien has already done a lot of work on the .mpy file format to support this, however without an MMU, this isn't generally possible on FAT or LittleFS. However, see #8381 for current work in that direction.
Note that this would still require being able to generate a .mpy file on the host (not the device), but these tools are available in pip/pypi now and would not require setting up the toolchain or building firmware.
I was wondering if it was possible to compile python scripts on the go. The idea would be you can upload a script (.py) file to the board and get the board to compile and save the compiled code. No need to build custom firmware. The compiled script don't need to be loaded into ram and run from flash, just like the firmware and core modules.
The reason is, it would be nice for newbie (me a month ago) who doesn't know the complexity of creating a firmware to be able to have the benefits of frozen scripts (Minimum Ram Usage) without going to the trouble of setting up the SDK.
Having feature would also be awesome because, say you had an IOT device located remotely. You could webrepl into the device, upload a new script (.py) compile it on the board, move it to the lib folder (or something), remove the script (.py) and execute the new compiled script from flash. Now doesn't take any time to compile or use any ram to execute (the script doesn't live in ram, unlike .py scripts)
The saved compiled version could be a .mpy or .pyc, not sure about the details, that's why im asking you, the awesome developers. What do you think of a feature like this and is it possible? Tell me your thoughts.
It would be nice to create "Frozen Scripts" with out creating a custom firmware. Lets use that awesome builtin Interpreter to its full potential.
Functionality:
Python Scrips compiled on board and saved as a file in the flash
Compiled Scripts execute from flash (no loading into ram)
Outcome:
Minimum Ram Usage for custom scripts.