tools/make-frozen-py: A small tool to create input files for make-frozen - #3071

Something with a similar purpose is here. In this version the generated Python source instantiates a memoryview into the bytes object. This enables large data objects to be accessed with minimal RAM usage.

Minor points: the generated code is more human-readable and has a version number which may be tested in code.

Is anything like that comes with CPython? If not, why do we need this to come with MicroPython?

The only use case I envision is with MicroPython and arises because currently we can freeze only Python code. If the ability to freeze data files is implemented, the need for such utilities will disappear. I'm agnostic on whether there's any merit in incorporating such a utility in the official repo - when I posted mine, along with a similar one for font files, I assumed they were rather too specialised.

If there is to be an official utility I do think a memoryview should be employed.

If the ability to freeze data files is implemented, the need for such utilities will disappear.

That's exactly what was being worked on during last week, with final commit just landed: 4a4490f . It's worth considering why this wasn't implemented using wrapping binary data in Python code. I of course considered that, but it was yet another step, yet another tool to use. The original idea from 2 years ago (#1346) was fully centered on frozen source, so it was natural to just drop everything to one dir and it was frozen. But since then we acquired frozen bytecode, which is the only viable choice for low-heap applications. And then implementation which was initially conceived for frozen source actually requires extra leg-work anyway: python source needs to go to frozen bytecode dir, resources - to frozen sources dir. Then still there's a question of overhead induced by wrapping binary data in Python objects, and possible limitations on size, etc. Plus, implementing it in C as it is now is a kind of optimization. Well, we'll see if it was a premature optimization and later we may want to switch to automated/transparent wrapping in Python code which will then let everything be used as frozen bytecode.

@peterhinch I actually looked at your font-to-py a while ago, I hadn't yet seen your data-to-py.
The make-frozen-py.py is completely un-optimized and was just quickly adapted from make-frozen.py about a year ago to fit the use case of embedding images in the firmware. I agree that your approach is optimal and should be used in any official tool.

I think the interface to the tool, outputting to stdout, should be followed to keep it consistent with make-frozen.py.

@pfalcon The use case is unique to MicroPython because of the firmware building step.

If the ability to freeze data files is implemented, the need for such utilities will disappear.

I agree. The reason that I proposed this PR is because I was writing a howto and didn't want to add yet another step of cloning a custom tool just to add the resources.

I don't have a preferred method for getting the resources into the firmware. I do strongly think that there should be a standard approach for MicroPython.

@bynds : Ok, as I note in #1346 (comment), I'm a bit disappointed with frozen module-resources and eager to try an alternative along the lines of your script. But it needs to follow the standard interface of pkg_resources.resource_stream() and be as optimal as possible. Do you know how to do all that and interested in doing that? 2 other alternatives:

1. I take this patch and rework it (perhaps, considerably), while still leaving your name on it.
2. I write it all from scratch.

Please suggest.

Superseded by #3228