Install wheel files from PyPi via upip (requires micropython-zipfile)

Looking into #6461, I came up with this:

def install_pkg(pkg_spec, install_path):
    data = get_pkg_metadata(pkg_spec)
    latest_ver = data["info"]["version"]
    packages = [x for x in data["releases"][latest_ver] if ".tar.gz" in x["filename"]]
    upy_platforms = [x for x in data["info"]["classifiers"] if "MicroPython" in x]
    assert len(upy_platforms) > 0 and len(packages) == 1, "This doesn't look like MicroPython package"
    package_url = packages[0]["url"]
    print("Installing %s %s from %s" % (pkg_spec, latest_ver, package_url))
    f1 = url_open(package_url)
    s = read_lines(f1)
    try:
        str1 = zlib.decompress(s, gzdict_sz)
        with tempfile.TemporaryFile() as temp_file:
            temp_file.write(str1)
            temp_file.seek(0)
            with tarfile.TarFile(fileobj=temp_file) as tar_file:  # Expects a file object
                meta = install_tar(tar_file, install_path)
    finally:
        f1.close()
    return meta

The problem is the PyPi tar.gz is a source tarball and may not act as expected. zipfile is necessary to extract the wheel file, but requires micropython-zipfile be fleshed out. That in turn depends mainly on zlib, which is fine. CPython's implementation is basically zlib plus a bunch of Python code to read the actual format.

Assuming a zipfile that implements ZipFile.infolist() and ZipFile.open() to read a zip file, the below would conceptually work:

def install_pkg(pkg_spec, install_path):
    data = get_pkg_metadata(pkg_spec)
    latest_ver = data["info"]["version"]
    tgz_packages = [x for x in data["releases"][latest_ver] if ".tar.gz" in x["filename"]]
    whl_packages = [x for x in data["releases"][latest_ver] if ".whl" in x["filename"]]
    packages = whl_packages if len(whl_packages) > 0 else tgz_packages
    upy_platforms = [x for x in data["info"]["classifiers"] if "MicroPython" in x]
    assert len(upy_platforms) > 0 and len(packages) == 1, "This doesn't look like MicroPython package"
    package_url = packages[0]["url"]
    print("Installing %s %s from %s" % (pkg_spec, latest_ver, package_url))
    f1 = url_open(package_url)
    if len(whl_packages) > 0:
        with ZipFile(f1.makefile()) as z:
            install_zip(z) # Do magic with z.infolist() and z.open()
    else:
        s = read_lines(f1)
        try:
            str1 = zlib.decompress(s, gzdict_sz)
            with tempfile.TemporaryFile() as temp_file:
                temp_file.write(str1)
                temp_file.seek(0)
                with tarfile.TarFile(fileobj=temp_file) as tar_file:  # Expects a file object
                    meta = install_tar(tar_file, install_path)
        finally:
            f1.close()
    return meta

I'm not sure anything is gained by using the CPython zipfile (and it seems to have issues e.g. it imports but doesn't really use binascii(), has its own crc32 implementation internally), but it might work nearly as-is, since it doesn't rely on additional C code.

Alright, on review, there are a few things needed for this:

Expose crc32() from uzlib (may as well expose adler32() as well)
Implement zlib.decompressobj() compatible uzlib.decompressobj()
Implement zlib.Decompressor compatible uzlib.Decompressor, supporting a subset
- decompress()
- unconsumed_tail()
- eof()
- flush()
Implement zipfile.infolist() and zipfile.zopen()

That's just barely enough to extract files from a Zip file using Deflate compression using the CPython zipfile module. I expect the potential size cost is minimal:

crc32() and adler32() are already implemented
uzlib.Decompressor wraps existing uzlib functionality with a thin interface
zipfile is a small Python module

One thing to consider when proposing changes to upip is that the script was optimised to ensure it can install libraries on an ESP8266. This has restricted RAM: my recollection is that the developer had some difficulty in achieving this.

Fair enough. crc32() and adler32() already exist in the C code for uzlib; I'm not sure the weight of decompressobj(), although it's supposed to be for not having a whole object in RAM; and zipfile from CPython is 86k of Python code, half of which is comments and white space. zipfile is also thread-safe, so implementing a uzipfile that's much more minimal is probably an easier target and is much less a heavy lift.

What about trying to import uzipfile or zipfile and throwing an exception if attempting to install a whl when neither exists, but otherwise running fine? In the very least, this is a small handful of additional bytes, although I worry a suitable uzipfile might be half a kilo or so and that board is really, really tight on RAM.

@jrmoserbaltimore

@jrmoserbaltimore was this able to be solved to create a functional ZipFile module for micropython?

In case anyone is curious I did implement zipfile and wheel here:

But I mostly target my own ESP32 hobby boards, I don't trust anything I make to run on the constrained ESP8266 modules.

upip has been replaced by mip / mpremote mip. Is this issue still relevant?

PS: there is now an initial port of the CPython zipfile to MicroPython here: https://github.com/jonnor/micropython-zipfile

Navigation Menu

Repository navigation

Install wheel files from PyPi via upip (requires micropython-zipfile) #7067

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Install wheel files from PyPi via upip (requires micropython-zipfile) #7067

Description

Activity

jrmoserbaltimore commented on Mar 25, 2021

peterhinch commented on Mar 25, 2021

jrmoserbaltimore commented on Mar 25, 2021

rohan-ahluwalia commented on Jun 13, 2022

maruno commented on Dec 16, 2023

jonnor commented on Jul 19, 2024

jonnor commented on Sep 8, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions