Repository navigation
Finalizing frozen modules approach #1346
Description
Frozen modules as implemented in #736, #1074 support only modules (not packages), and implemented in quite adhoc way. Maybe, maybe it could be extended in the same adhoc way to support packages. But as soon you think that to support completely frozen apps we also need to support packaging and accessing data files, it's clear it's not a scalable approach. Alternatives:
One extreme:
Implement complete high-level VFS layer, so it worked up to open("@MY_VFS@/dir/file") on python side.
Another extreme:
Implement all CPython import babylon. (e.g https://docs.python.org/3/library/importlib.html). Sheer number of PEPs required to get it right gives a message that maybe it's still not got right. And wading thru it all to find a suitable subset is a chore on its own.
So, compromise I'm thinking about:
Not implement complete VFS layer for all file system access, but implement kind of abstraction solely on builtinimport.c level, but in such a way so it was reusable to implement pkgutil.get_data(), pkg_resources.resource_stream(). (https://pythonhosted.org/setuptools/pkg_resources.html#resourcemanager-api)
Thoughts?
Activity
pfalcon commentedon Jun 26, 2015
In the meantime I've implemented these (for normal FS case) in micropython-lib, and will think how to adopt their usage in picoweb.
pfalcon commentedon Jun 27, 2015
Next step, I'm working on making picoweb frozen-module friendly. That means that utemplate should acquire ability to load templates from another package. Once again there's thought that just implementing complete VFS layer would solve that almost automagically.
But well, apparently it needs to be done in Pythonic way. Then, after fiddling with it a bit, I came to following conclusion: utemplate's Loader class constructor should take reference to package name from which to import templates. It may be None to work as close to how it currently works (loading templates from an FS path), or if it's not None, then loaders should use package access API.
Then for "compiled" template loaders, I'd like to take further steps to simplify. Currently. it uses
exec()to load templates, if loading from a package, it just makes sense to use__import__(). But then compiled template name may not contain dot in it. That's actually the reason whyexec()is used currently - I started with__import__, but wanted an implementation which doesn't have naming restrictions. Well, utemplate is supposed to be minimal, so if there's new requirement of being able to load from another package, need to give up other features to avoid "bloat". As a final step then, I have an idea to split utemplates distribution package into to - utemplates.compiled and utemplates.source. After all, there's no need to ship utemplates.source with a "production" app packages e.g. as a frozen module.I don't know if somebody follows that ;-) (@dpgeorge? You looked into this stuff, so maybe you have comments, but then I seems like you already have good backlog), but to avoid possible later questions why utemplate/picoweb API was broken without a notice, here's the notice.
pfalcon commentedon Jun 27, 2015
Ok, so utemplate.*.Loader and picoweb.WebApp classes now require extra very first argument - package name from which to load resources (templates in this case). To achieve previous behavior (templates are loaded from the current directory), None should be passed there. Otherwise, the idea is that
__name__should be passed there, but that yet has to be fully debugged to work as expected.pfalcon commentedon Apr 30, 2017
New URL for pkg_resources.resource_stream(): http://setuptools.readthedocs.io/en/latest/pkg_resources.html
The idea is to implement it as uos.resource_stream().
pfalcon commentedon May 1, 2017
Actually, uio.resource_stream()
pfalcon commentedon May 6, 2017
With 4a4490f, this is fully implemented.
pfalcon commentedon May 11, 2017
For reference, I made needed changes to picoweb and confirmed that it's possible to frozen notes_pico sample app completely (with resources) into unix port. But actually trying that need to split package files in different sets, one going into modules/, another into scripts/ it's indeed a bit boring, so I'm more and more leaning towards solution along the lines of: #3071