reset zlib.DecompIO state

This is a common error when creating a zlib.DecompIO(f):

MemoryError: memory allocation failed, allocating 32768 bytes

Obviously the buffer is necessary (for any streams you don't control the creation of, almost always w/ wbits=15), but that's a third of available ram, and sometimes the GC doesn't purge the old one in time to create the new one, or because of fragmentation there's no contiguous space available.

Could we add a function to reset the internal state? IE decomp.reset(f) That way it could reuse the previous allocation, eliminating the risk of an OOM if resetting instead of creating anew.

Would you be willing to sponsor this work? Yes, if the concept is agreed to.

Looks like that 32k buffer is allocated here:

micropython/extmod/moduzlib.c

Line 104 in 38e7b84

uzlib_uncompress_init(&o->decomp, m_new(byte, dict_sz), dict_sz);

When it is passed to uzlib_uncompress_init.

A pointer to it is stored on o->decomp, so the DecompIO object will hold a reference to it, preventing it from being garbage collected until it goes out of scope or is explicitly del'd.

Since DecompIO has no hidden global, internal state then decomp.reset(f) couldn't do anything different to:

del decomp
gc.collect()

I'm guessing the canonical answer would be to manually gc.collect(), though garbage collection should keep up. On the RP2 port with a bit more RAM (about 187k) I can run this indefinitely:

import gc
import time
import uzlib

while True:
    print(f"Mem free: {gc.mem_free()}")
    uzlib.DecompIO(open("witw.z", "rb"))
    time.sleep(0.5)

HOWEVER - for some reason, if I manually run uzlib.DecompIO(open("witw.z", "rb")) on the REPL via Thonny it never lets go of the buffer, even if I explicitly call gc.collect():

>>> import gc
>>> import uzlib
>>> uzlib.DecompIO(open("witw.z", "rb"));print(f"Mem free: {gc.mem_free()}")
Dict Size: 32768
<DecompIO>
Mem free: 94416
>>> uzlib.DecompIO(open("witw.z", "rb"));print(f"Mem free: {gc.mem_free()}")
Dict Size: 32768
<DecompIO>
Mem free: 56304
>>> uzlib.DecompIO(open("witw.z", "rb"));print(f"Mem free: {gc.mem_free()}")
Dict Size: 32768
<DecompIO>
Mem free: 18176
>>> gc.collect()
>>> uzlib.DecompIO(open("witw.z", "rb"));print(f"Mem free: {gc.mem_free()}")
Dict Size: 32768
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError: memory allocation failed, allocating 32768 bytes
>>> uzlib.DecompIO(open("witw.z", "rb"));print(f"Mem free: {gc.mem_free()}")
Dict Size: 32768
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError: memory allocation failed, allocating 32768 bytes

This implies that the REPL is holding a reference to every DecompIO object somehow, and preventing the RAM from being freed.

Is it normal program flow or REPL usage that you're encountering this issue with?

Here's my test file (renamed to .zip so GitHub will accept it):
witw.zip

Normal program flow. I'm writing a git client for my esp32, which involves lots of zlib streams. (Code is already structured to only ever open one at a time, even for DELTA objs.)

del decomp; gc.collect() is basically what I'm doing, but it's non-deterministically failing. (My esp32 has about 100k available ram.) The malloc sometimes will fail even when mem_free reports over 32768. I believe because there's not a single contiguous chunk of 32768. I can do the first example indefinitely too.

I believe the malloc failure you experienced is due to that as well. (Not that REPL is holding a reference.) Try it without the print statements (or any other additional object creation). I can do that second example indefinitely without the prints, as it keeps freeing and reallocating the same chunk. But once you start creating random strings, memory gets fragmented, and occasionally it'll fail even when total available ram is enough.

example:

__enter__ <_ObjReader object at 3ffe69c0> 3
before 68432 after 69936 @ <_ObjReader object at 3ffe69c0>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "mgit.py", line 476, in clone
  File "mgit.py", line 595, in pull
  File "mgit.py", line 495, in checkout
  File "mgit.py", line 526, in _checkout_file
  File "mgit.py", line 211, in __enter__
MemoryError: memory allocation failed, allocating 32768 bytes

code:

    print('__enter__', self, self.kind)
    print('before', gc.mem_free(), gc.collect() or 'after', gc.mem_free(), '@', self)
    self.decompressed_stream = DecompIO(self.f)

You can see it fails even with 69936 free reported.

Whereas other runs (>50% of the time) will happily run until the file-system fills up:

__enter__ <_ObjReader object at 3ffe6650> 3
before 70768 after 72224 @ <_ObjReader object at 3ffe6650>
__exit__ <_ObjReader object at 3ffe6650> 3 True
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "mgit.py", line 470, in clone
  File "mgit.py", line 589, in pull
  File "mgit.py", line 489, in checkout
  File "mgit.py", line 523, in _checkout_file
OSError: 28

I guess self.decompressed_stream will hold a reference to DecompIO() that's not released until you're in C scope creating a new DecompIO object, so there's no opportunity for GC to run and collect that free'd RAM.

Have you tried something like:

del self.decompressed_stream
gc.collect()
self.decompressed_stream = DecompIO(self.f)

That will usually work, but would make for some pretty weird code outside a trivial example.

I don't think the underlying problem (lack of a free 32k block even when total free ram is >32k) is fixable. But a simple reset function would be useful and easy to implement. For example, rather than:

del self.decompressed_stream
self.decompressed_stream = DecompIO(new_f)

just do:

self.decompressed_stream.reset(new_f)

This function should be simple enough:

Save the new pointer.
Zero out the buffer.
Call uzlib_gzip_parse_header/etc.

And doesn't require a risky 32k malloc.

I'll be happy to work on a PR if you're not opposed to it on principal.

I'm definitely not qualified to yay or nay a PR, but I agree with the premise here.

Though I'm not sure MicroPython is necessarily averse to weird code 😆 - for better or worse, writing an interpreted language in an embedded environment is not always going to be graceful.

It feels like the fundamental goal here is to have a fixed region of RAM dedicated to zlib decompression, such that you're never at risk of fragmentation or allocation failure.

I'd guess the approach most likely to be accepted is to make the filename optional in the constructor, like so:

decomp = DecompIO(wbits=15)
decomp.open_stream(new_f)

There's precedent for zlib libraries expecting the user to know about window size, since CPython's zlib has a wbits argument: https://docs.python.org/3/library/zlib.html#zlib.decompress

The above interface would prep the DecompIO object with a suitable buffer size (calculated from wbits) and calls to open_stream would raise a ValueError or otherwise for streams that need a larger window size. open_stream is effectively your reset.

That way your window size isn't a pot-luck from the first compressed stream you decompress, and you don't need to worry about potential re-allocations (or baffling the user) later.

Oh hey I totally missed that DecompIO does accept wbits as a second positional argument as part of dict_opt.

micropython/extmod/moduzlib.c

Lines 84 to 104 in 38e7b84

    
           if (dict_opt >= 16) { 
        
               int st = uzlib_gzip_parse_header(&o->decomp); 
        
               if (st != TINF_OK) { 
        
                   goto header_error; 
        
               } 
        
               dict_sz = 1 << (dict_opt - 16); 
        
           } else if (dict_opt >= 0) { 
        
               dict_opt = uzlib_zlib_parse_header(&o->decomp); 
        
               if (dict_opt < 0) { 
        
               header_error: 
        
                   mp_raise_ValueError(MP_ERROR_TEXT("compression header")); 
        
               } 
        
               // RFC 1950 section 2.2: 
        
               // CINFO is the base-2 logarithm of the LZ77 window size, 
        
               // minus eight (CINFO=7 indicates a 32K window size) 
        
               dict_sz = 1 << (dict_opt + 8); 
        
           } else { 
        
               dict_sz = 1 << -dict_opt; 
        
           } 
        
           uzlib_uncompress_init(&o->decomp, m_new(byte, dict_sz), dict_sz);

Where dict_opt is rather awkwardly used to disambiguate gzip vs zlib or, with a negative number, just assumed to be for a raw DEFLATE stream. Ugh.

The docs say the API is unstable, though, so fill your boots I guess! - https://docs.micropython.org/en/latest/library/zlib.html#zlib.DecompIO

I abandoned my context manager wrapper for this singleton abomination:

from zlib import DecompIO as _DecompIO

#print('mem_free before _master_decompio', gc.mem_free())
_master_decompio = _DecompIO(io.BytesIO(b'x\x9c\x03\x00\x00\x00\x00\x01'))
#print('mem_free after _master_decompio', gc.mem_free())

class DecompIO:

  def __init__(self, f):
    global _master_decompio
    gc.collect() # WTF
    print('mem_free before dealloc', id(_master_decompio), gc.mem_free()) # WTF2
    del _master_decompio
    gc.collect()
    #print('mem_free after dealloc', gc.mem_free())
    _master_decompio = _DecompIO(f)
    self._id = id(_master_decompio)
    #print('mem_free after alloc', self._id, gc.mem_free())

  def read(self, nbytes):
    global _master_decompio
    assert self._id == id(_master_decompio)
    return _master_decompio.read(nbytes)
    
  def readline(self):
    global _master_decompio
    assert self._id == id(_master_decompio)
    return _master_decompio.readline()

At least it seems stable so far! 🤞😅

Curiously, the two # WTF lines I tagged are necessary, otherwise the next alloc will fail. I don't have a working theory as to why this would be...

EDIT 4/2/3023: Sadly not stable. Just less frequent GC misses.

Wow, that's about as elegant as you can get I suppose. 😬

If nothing else your wrapper shows that DecompIO probably needs some work.

Love your solution to allocating (what I guess is) a 32k buffer up front 😆

a minimal example of garbage collection seemingly not working:

MicroPython v1.19.1 on 2022-06-18; ESP32 module with ESP32
Type "help()" for more information.
>>> import gc, micropython, io, zlib
>>> class X: pass
... 
>>> micropython.mem_info(1)
stack: 720 out of 15360
GC: total: 111168, used: 2768, free: 108400
 No. of 1-blocks: 44, 2-blocks: 8, max blk sz: 18, max free sz: 6763
GC memory layout; from 3ffe4db0:
00000: h=hhhhBMShDBhFh=hhh=================hh=======h=======h=hh=======
00400: ========ShShhhhhShT=hh======hh========h==h==h==hBh=hh=hhhBBDhhh=
00800: =h=B.h=======h==========h===h=====hhhhh==........h=...h==.......
       (105 lines all free)
1b000: ....................................
>>> x = X()
>>> x.d = zlib.DecompIO(io.BytesIO(b'x\x9c\x03\x00\x00\x00\x00\x01'))
>>> del x.d
>>> x.d
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'X' object has no attribute 'd'
>>> gc.collect()
>>> micropython.mem_info(1)
stack: 720 out of 15360
GC: total: 111168, used: 36096, free: 75072
 No. of 1-blocks: 31, 2-blocks: 7, max blk sz: 2048, max free sz: 4579
GC memory layout; from 3ffe4db0:
00000: h=hhhh=MhhDB..h=hhh=================hh=======h=======h=h.......h
00400: =...h==.....................................h==..h=.h=.....Dh...
00800: ....hh=======h==========h===h=====...........h.....hhS...h====hh
00c00: ..h====h..h.h.............h=====================================
01000: ===========================================h====================
01400: ================================================================
01800: ================================================================
01c00: ================================================================
02000: ================================================================
02400: ================================================================
02800: ================================================================
02c00: ================================================================
03000: ================================================================
03400: ================================================================
03800: ================================================================
03c00: ================================================================
04000: ================================================================
04400: ================================================================
04800: ================================================================
04c00: ================================================================
05000: ================================================================
05400: ================================================================
05800: ================================================================
05c00: ================================================================
06000: ================================================================
06400: ================================================================
06800: ================================================================
06c00: ================================================================
07000: ================================================================
07400: ================================================================
07800: ================================================================
07c00: ================================================================
08000: ================================================================
08400: ================================================================
08800: ================================================================
08c00: ================================================================
09000: ===========================================.S.....h==..hhh.hh...
09400: B...............................................................
       (70 lines all free)
1b000: ....................................
>>> del x
>>> gc.collect()
>>> micropython.mem_info(1)
stack: 720 out of 15360
GC: total: 111168, used: 1824, free: 109344
 No. of 1-blocks: 24, 2-blocks: 7, max blk sz: 18, max free sz: 4588
GC memory layout; from 3ffe4db0:
00000: h=hhhh=Mh=Dhhhh=hhh=================Bh=======h=======h=h...hh..h
00400: =h.h...B.....h==.....h...........................h=........Dh...
00800: ....hh=======h==========h===h=====.......................h====..
00c00: ..h====h....h...................................................
       (32 lines all free)
09000: .......................................................h........
       (71 lines all free)
1b000: ....................................
>>>

added a reset function to allow buffer reuse. see:

proposed: keredson@7dc82cf

there's some housekeeping needed. arg checks. what to do if the wbits are different between the two streams.
also i can't figure out how to make it return none... 🤦 but it works! (and no more OOMs when parsing a few hundred small streams!)

FYI this is the micropython git client i've been working on: https://github.com/keredson/ygit

return mp_const_none; should do it.

Navigation Menu

Repository navigation

reset zlib.DecompIO state #11146

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if (dict_opt >= 16) {
	int st = uzlib_gzip_parse_header(&o->decomp);
	if (st != TINF_OK) {
	goto header_error;
	}
	dict_sz = 1 << (dict_opt - 16);
	} else if (dict_opt >= 0) {
	dict_opt = uzlib_zlib_parse_header(&o->decomp);
	if (dict_opt < 0) {
	header_error:
	mp_raise_ValueError(MP_ERROR_TEXT("compression header"));
	}
	// RFC 1950 section 2.2:
	// CINFO is the base-2 logarithm of the LZ77 window size,
	// minus eight (CINFO=7 indicates a 32K window size)
	dict_sz = 1 << (dict_opt + 8);
	} else {
	dict_sz = 1 << -dict_opt;
	}

	uzlib_uncompress_init(&o->decomp, m_new(byte, dict_sz), dict_sz);

reset zlib.DecompIO state #11146

Description

Activity

Gadgetoid commented on Mar 28, 2023

keredson commented on Mar 28, 2023

keredson commented on Mar 28, 2023

Gadgetoid commented on Mar 28, 2023

keredson commented on Mar 28, 2023

Gadgetoid commented on Mar 28, 2023

Gadgetoid commented on Mar 28, 2023

keredson commented on Mar 29, 2023

Gadgetoid commented on Mar 29, 2023

keredson commented on Apr 1, 2023

keredson commented on Apr 2, 2023

keredson commented on Apr 2, 2023

Gadgetoid commented on Apr 3, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions