Repository navigation
Install wheel files from PyPi via upip (requires micropython-zipfile) #7067
Description
Looking into #6461, I came up with this:
def install_pkg(pkg_spec, install_path):
data = get_pkg_metadata(pkg_spec)
latest_ver = data["info"]["version"]
packages = [x for x in data["releases"][latest_ver] if ".tar.gz" in x["filename"]]
upy_platforms = [x for x in data["info"]["classifiers"] if "MicroPython" in x]
assert len(upy_platforms) > 0 and len(packages) == 1, "This doesn't look like MicroPython package"
package_url = packages[0]["url"]
print("Installing %s %s from %s" % (pkg_spec, latest_ver, package_url))
f1 = url_open(package_url)
s = read_lines(f1)
try:
str1 = zlib.decompress(s, gzdict_sz)
with tempfile.TemporaryFile() as temp_file:
temp_file.write(str1)
temp_file.seek(0)
with tarfile.TarFile(fileobj=temp_file) as tar_file: # Expects a file object
meta = install_tar(tar_file, install_path)
finally:
f1.close()
return meta
The problem is the PyPi tar.gz is a source tarball and may not act as expected. zipfile is necessary to extract the wheel file, but requires micropython-zipfile be fleshed out. That in turn depends mainly on zlib, which is fine. CPython's implementation is basically zlib plus a bunch of Python code to read the actual format.
Assuming a zipfile that implements ZipFile.infolist() and ZipFile.open() to read a zip file, the below would conceptually work:
def install_pkg(pkg_spec, install_path):
data = get_pkg_metadata(pkg_spec)
latest_ver = data["info"]["version"]
tgz_packages = [x for x in data["releases"][latest_ver] if ".tar.gz" in x["filename"]]
whl_packages = [x for x in data["releases"][latest_ver] if ".whl" in x["filename"]]
packages = whl_packages if len(whl_packages) > 0 else tgz_packages
upy_platforms = [x for x in data["info"]["classifiers"] if "MicroPython" in x]
assert len(upy_platforms) > 0 and len(packages) == 1, "This doesn't look like MicroPython package"
package_url = packages[0]["url"]
print("Installing %s %s from %s" % (pkg_spec, latest_ver, package_url))
f1 = url_open(package_url)
if len(whl_packages) > 0:
with ZipFile(f1.makefile()) as z:
install_zip(z) # Do magic with z.infolist() and z.open()
else:
s = read_lines(f1)
try:
str1 = zlib.decompress(s, gzdict_sz)
with tempfile.TemporaryFile() as temp_file:
temp_file.write(str1)
temp_file.seek(0)
with tarfile.TarFile(fileobj=temp_file) as tar_file: # Expects a file object
meta = install_tar(tar_file, install_path)
finally:
f1.close()
return meta
I'm not sure anything is gained by using the CPython zipfile (and it seems to have issues e.g. it imports but doesn't really use binascii(), has its own crc32 implementation internally), but it might work nearly as-is, since it doesn't rely on additional C code.
Activity
jrmoserbaltimore commentedon Mar 25, 2021
Alright, on review, there are a few things needed for this:
crc32()fromuzlib(may as well exposeadler32()as well)zlib.decompressobj()compatibleuzlib.decompressobj()zlib.Decompressorcompatibleuzlib.Decompressor, supporting a subsetdecompress()unconsumed_tail()eof()flush()zipfile.infolist()andzipfile.zopen()That's just barely enough to extract files from a Zip file using Deflate compression using the CPython
zipfilemodule. I expect the potential size cost is minimal:crc32()andadler32()are already implementeduzlib.Decompressorwraps existinguzlibfunctionality with a thin interfacezipfileis a small Python modulepeterhinch commentedon Mar 25, 2021
One thing to consider when proposing changes to
upipis that the script was optimised to ensure it can install libraries on an ESP8266. This has restricted RAM: my recollection is that the developer had some difficulty in achieving this.jrmoserbaltimore commentedon Mar 25, 2021
Fair enough.
crc32()andadler32()already exist in the C code foruzlib; I'm not sure the weight ofdecompressobj(), although it's supposed to be for not having a whole object in RAM; andzipfilefrom CPython is 86k of Python code, half of which is comments and white space.zipfileis also thread-safe, so implementing auzipfilethat's much more minimal is probably an easier target and is much less a heavy lift.What about trying to import
uzipfileorzipfileand throwing an exception if attempting to install awhlwhen neither exists, but otherwise running fine? In the very least, this is a small handful of additional bytes, although I worry a suitableuzipfilemight be half a kilo or so and that board is really, really tight on RAM.rohan-ahluwalia commentedon Jun 13, 2022
@jrmoserbaltimore was this able to be solved to create a functional ZipFile module for micropython?
maruno commentedon Dec 16, 2023
In case anyone is curious I did implement zipfile and wheel here:
But I mostly target my own ESP32 hobby boards, I don't trust anything I make to run on the constrained ESP8266 modules.
jonnor commentedon Jul 19, 2024
upip has been replaced by
mip/mpremote mip. Is this issue still relevant?jonnor commentedon Sep 8, 2024
PS: there is now an initial port of the CPython
zipfileto MicroPython here: https://github.com/jonnor/micropython-zipfile