Added memzip filesystem support for teensy - #147

And of course this all is teensy specific.

Well, it could very easily work for any other controller by adding the _staticfs symbol to the linker script file. I suspect that this code can be put into a generic place, once such a place exists.

I think it will be useful for any microcontroller which doesn't have an sdcard.

import does work, you just need to write mp_import_open_file. See py/lexerunix.c for an example.

And here's a question. Since this is an embedded platform, it makes sense
to me to be able to import byte-codes and/or source code (i.e. look for
.pyc files).

This would involve creating a host-side tool which could generate such byte
code codes, but that seems fairly straight forward. What else would be
needed? From my very brief look at the byte code stuff it seems to me that
string indicies probably need to be "fixed up" - so a kind of linker step.
Anything else?

I don't really need to implement this right now, I just wanted to start
thinking about it.

On Sun, Jan 12, 2014 at 4:55 AM, Damien George [email protected]:

import does work, you just need to write mp_import_open_file. See
py/lexerunix.c for an example.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/147#issuecomment-32121862
.

Dave Hylands
Shuswap, BC, Canada
http://www.davehylands.com

This is not on my to-do list, but would be a nice feature. Of course, it would also require a host-side tool to be made which creates the .mpc files -- note they are not compatibly with .pyc files.

If your byte-code didn't have any opcodes that used qstrs, then you could literally load the byte code into RAM and it would work. So, the thing that needs to be implemented is a simple qstr linker. That's not trivial, because qstrs are variably-length encoded in the byte code. You could either write the .mpc files using the maximum width for the qstr, so the linked version would definitely fit, or you could re-encode the .mpc, adjusting qstr sizes and jump offsets.

Note that qstr encoding will change soon when I change how qstrs are implemented. They will become pointers instead of indexes. This may ease the .mpc translation/linking, because I may have to reserver 4 bytes for all qstr pointers, regardless.

It's really low on my todo list too. I think it will take a while before its needed, but it could be a huge size win. My real goal would be to allow the bytecode to remain in flash and not need to be in RAM at all. This might require the addition of new opcodes to allow the strings to be looked up a little differently, but I think that would be a worthwhile tradefoff fo allowing the bytecodes to be ROMable.

@dhylands

I don't really need to implement this right now, I just wanted to start thinking about it.

See also here (for a buddy): #14 (comment)

Also, please consider that Python deals with zip files in this way: http://docs.python.org/2/library/zipfile . And Python way to keep applications in compressed archives is here: http://www.python.org/dev/peps/pep-0441/. Doing those features for Python implementation in non-Python way is slippy one (typical risk in our case is that teensy will have memzip support, stmf4disovery - zipmem support, stmf3discovery - pizmem support, etc.)

From my very brief look at the byte code stuff it seems to me that string indicies probably need to be "fixed up" - so a kind of linker step.

Why not start with doing it as Python does? It doesn't need any (explicit) "linking" step, so apparently it can deal with multiple bytecode modules, each having its own string pool.

So for the memzip support, my thinking was that I knew python has some
support for zip files. I wanted something that could start off being super
simple, and evolve in the fully generic solution.

There is no reason memzip can't support compressed zip files. It just means
adding a decompressor, and using RAM. I wanted to be able to addd the
optimization of saying if the file is uncompressed then allow it to be
directly accesible from flash and not have to be loaded into RAM.

So I think of memzip as a super simple zip implementation to give the
teensy some read-only storage. Without some persistence, micropython is
more or less useless.

And as I mentioned there is nothing teensy specific about memzip. I just
don't have a pyboard (yet) to test it on, or I would have ported it there
as well.

On Sun, Jan 12, 2014 at 11:32 AM, Paul Sokolovsky
[email protected]:

@dhylands https://github.com/dhylands

I don't really need to implement this right now, I just wanted to start
thinking about it.

See also here (for a buddy): #14 (comment)#14 (comment)

Also, please consider that Python deals with zip files in this way:
http://docs.python.org/2/library/zipfile . And Python way to keep
applications in compressed archives is here:
http://www.python.org/dev/peps/pep-0441/. Doing those features for Python
implementation in non-Python way is slippy one (typical risk in our case is
that teensy will have memzip support, stmf4disovery - zipmem support,
stmf3discovery - pizmem support, etc.)

From my very brief look at the byte code stuff it seems to me that string
indicies probably need to be "fixed up" - so a kind of linker step.

Why not start with doing it as Python does? It doesn't need any (explicit)
"linking" step, so apparently it can deal with multiple bytecode modules,
each having its own string pool.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/147#issuecomment-32131255
.

Dave Hylands
Shuswap, BC, Canada
http://www.davehylands.com

My real goal would be to allow the bytecode to remain in flash and not need to be in RAM at all.

I don't understand. So, is stm port currently keeps bytecode in RAM? I can imagine the only reason for that - corresponding filesystem storage and/or bytecode accessor is not yet implemented, because RAM and NOR flash are read-accessed in the same way.

Of course, next step is supporting NAND flash, and that's why it's important: 1) to support multiple bytecode modules; 2) to implement module loading using proven standard generic Python features - like import hooks, standard modules for accessing archives files (and I/O in general). - Then any module load scenario can be implemented by combining generic features.

I wanted something that could start off being super simple, and evolve in the fully generic solution.

Good plan then, thanks.

I just don't have a pyboard (yet) to test it on, or I would have ported it there as well.

I'd personally suggest prototyping any non hardware dependent features on unix port. If it's done right, the code can go straight into py/ (well, if people will start writing (standards-compliant) zip modules, we can start modules/ to keep them separate from core).

I could emulate memzip under unix, by just allocating enough memory to hold the zip file, and read the entire thing in, and have the variable which points to the memory be called _staticfs.

What memzip is doing is just storing the zipfile in flash, just past the regular image.

Supporting the fully generic solution requires a decompressor, so it really just depends on how much RAM/flash the decompressor needs.

My feeling is that once your python code starts getting big enough to fill up the remaining flash on the teensy, then you're probably running out of RAM, so allowing for larger code doesn't really help.

We won't know how this all plays out until people start coding up real apps and we start looking at RAM/flash usage.

It's really low on my todo list too. I think it will take a while before
its needed, but it could be a huge size win. My real goal would be to allow
the bytecode to remain in flash and not need to be in RAM at all. This
might require the addition of new opcodes to allow the strings to be looked
up a little differently, but I think that would be a worthwhile tradefoff
fo allowing the bytecodes to be ROMable.

That doesn't need new bytecodes, only add support for import hooks and add
one that can be able to read the python modules directly from the flash
memory or the compiled VM binary on demand.

Also, if not using compresion (as CPython support for packages importing),
instead of using zip format maybe it would be better to use tar format.

@pfalcon yes, Micro Python is running byte code from RAM on the stm port. Saving the bytecode to flash would wear out the flash too quickly, and be noticeably slow (erasing flash is not fast). Byte code is small, and there's enough RAM for it.

Of course, it's pretty easy to change this so there is some scratch flash to put scripts, a proper cache for them. But I really don't think it's worth it at this point.

CPython does need to link .cpy files, in a sense. When it loads them in it unpickles the name (actually constant) dictionary for all the referenced names in the byte code. The byte code references these names using a table. Eg, load_attr 1 would lookup name 1 in the table, and then this would point to an actual, live Python object in the current Python process memory space. Unless you want to do strcmp for all bytecodes that reference a name, you inevitably need some RAM and some kind of linker.