Repository navigation
reset zlib.DecompIO state #11146
Description
Activity
Looks like that 32k buffer is allocated here:
Line 104 in 38e7b84
When it is passed to
uzlib_uncompress_init.A pointer to it is stored on
o->decomp, so theDecompIOobject will hold a reference to it, preventing it from being garbage collected until it goes out of scope or is explicitlydel'd.Since
DecompIOhas no hidden global, internal state thendecomp.reset(f)couldn't do anything different to:del decomp gc.collect()
I'm guessing the canonical answer would be to manually
gc.collect(), though garbage collection should keep up. On the RP2 port with a bit more RAM (about 187k) I can run this indefinitely:import gc import time import uzlib while True: print(f"Mem free: {gc.mem_free()}") uzlib.DecompIO(open("witw.z", "rb")) time.sleep(0.5)
HOWEVER - for some reason, if I manually run
uzlib.DecompIO(open("witw.z", "rb"))on the REPL via Thonny it never lets go of the buffer, even if I explicitly callgc.collect():>>> import gc >>> import uzlib >>> uzlib.DecompIO(open("witw.z", "rb"));print(f"Mem free: {gc.mem_free()}") Dict Size: 32768 <DecompIO> Mem free: 94416 >>> uzlib.DecompIO(open("witw.z", "rb"));print(f"Mem free: {gc.mem_free()}") Dict Size: 32768 <DecompIO> Mem free: 56304 >>> uzlib.DecompIO(open("witw.z", "rb"));print(f"Mem free: {gc.mem_free()}") Dict Size: 32768 <DecompIO> Mem free: 18176 >>> gc.collect() >>> uzlib.DecompIO(open("witw.z", "rb"));print(f"Mem free: {gc.mem_free()}") Dict Size: 32768 Traceback (most recent call last): File "<stdin>", line 1, in <module> MemoryError: memory allocation failed, allocating 32768 bytes >>> uzlib.DecompIO(open("witw.z", "rb"));print(f"Mem free: {gc.mem_free()}") Dict Size: 32768 Traceback (most recent call last): File "<stdin>", line 1, in <module> MemoryError: memory allocation failed, allocating 32768 bytesThis implies that the REPL is holding a reference to every
DecompIOobject somehow, and preventing the RAM from being freed.Is it normal program flow or REPL usage that you're encountering this issue with?
Here's my test file (renamed to .zip so GitHub will accept it):
witw.zipNormal program flow. I'm writing a
gitclient for my esp32, which involves lots of zlib streams. (Code is already structured to only ever open one at a time, even for DELTA objs.)del decomp; gc.collect()is basically what I'm doing, but it's non-deterministically failing. (My esp32 has about 100k available ram.) The malloc sometimes will fail even whenmem_freereports over 32768. I believe because there's not a single contiguous chunk of 32768. I can do the first example indefinitely too.I believe the malloc failure you experienced is due to that as well. (Not that REPL is holding a reference.) Try it without the print statements (or any other additional object creation). I can do that second example indefinitely without the prints, as it keeps freeing and reallocating the same chunk. But once you start creating random strings, memory gets fragmented, and occasionally it'll fail even when total available ram is enough.
example:
__enter__ <_ObjReader object at 3ffe69c0> 3 before 68432 after 69936 @ <_ObjReader object at 3ffe69c0> Traceback (most recent call last): File "<stdin>", line 1, in <module> File "mgit.py", line 476, in clone File "mgit.py", line 595, in pull File "mgit.py", line 495, in checkout File "mgit.py", line 526, in _checkout_file File "mgit.py", line 211, in __enter__ MemoryError: memory allocation failed, allocating 32768 bytescode:
print('__enter__', self, self.kind) print('before', gc.mem_free(), gc.collect() or 'after', gc.mem_free(), '@', self) self.decompressed_stream = DecompIO(self.f)You can see it fails even with 69936 free reported.
Whereas other runs (>50% of the time) will happily run until the file-system fills up:
__enter__ <_ObjReader object at 3ffe6650> 3 before 70768 after 72224 @ <_ObjReader object at 3ffe6650> __exit__ <_ObjReader object at 3ffe6650> 3 True Traceback (most recent call last): File "<stdin>", line 1, in <module> File "mgit.py", line 470, in clone File "mgit.py", line 589, in pull File "mgit.py", line 489, in checkout File "mgit.py", line 523, in _checkout_file OSError: 28I guess
self.decompressed_streamwill hold a reference toDecompIO()that's not released until you're in C scope creating a newDecompIOobject, so there's no opportunity for GC to run and collect that free'd RAM.Have you tried something like:
del self.decompressed_stream gc.collect() self.decompressed_stream = DecompIO(self.f)
That will usually work, but would make for some pretty weird code outside a trivial example.
I don't think the underlying problem (lack of a free 32k block even when total free ram is >32k) is fixable. But a simple reset function would be useful and easy to implement. For example, rather than:
del self.decompressed_stream self.decompressed_stream = DecompIO(new_f)
just do:
self.decompressed_stream.reset(new_f)
This function should be simple enough:
- Save the new pointer.
- Zero out the buffer.
- Call
uzlib_gzip_parse_header/etc.
And doesn't require a risky 32k malloc.
I'll be happy to work on a PR if you're not opposed to it on principal.
I'm definitely not qualified to yay or nay a PR, but I agree with the premise here.
Though I'm not sure MicroPython is necessarily averse to weird code 😆 - for better or worse, writing an interpreted language in an embedded environment is not always going to be graceful.
It feels like the fundamental goal here is to have a fixed region of RAM dedicated to zlib decompression, such that you're never at risk of fragmentation or allocation failure.
I'd guess the approach most likely to be accepted is to make the filename optional in the constructor, like so:
decomp = DecompIO(wbits=15) decomp.open_stream(new_f)
There's precedent for zlib libraries expecting the user to know about window size, since CPython's zlib has a
wbitsargument: https://docs.python.org/3/library/zlib.html#zlib.decompressThe above interface would prep the DecompIO object with a suitable buffer size (calculated from
wbits) and calls toopen_streamwould raise aValueErroror otherwise for streams that need a larger window size.open_streamis effectively yourreset.That way your window size isn't a pot-luck from the first compressed stream you decompress, and you don't need to worry about potential re-allocations (or baffling the user) later.
Reacted by Derek AndersonOh hey I totally missed that
DecompIOdoes acceptwbitsas a second positional argument as part ofdict_opt.Lines 84 to 104 in 38e7b84
Where
dict_optis rather awkwardly used to disambiguate gzip vs zlib or, with a negative number, just assumed to be for a raw DEFLATE stream. Ugh.The docs say the API is unstable, though, so fill your boots I guess! - https://docs.micropython.org/en/latest/library/zlib.html#zlib.DecompIO
Reacted by Derek AndersonI abandoned my context manager wrapper for this singleton abomination:
from zlib import DecompIO as _DecompIO #print('mem_free before _master_decompio', gc.mem_free()) _master_decompio = _DecompIO(io.BytesIO(b'x\x9c\x03\x00\x00\x00\x00\x01')) #print('mem_free after _master_decompio', gc.mem_free()) class DecompIO: def __init__(self, f): global _master_decompio gc.collect() # WTF print('mem_free before dealloc', id(_master_decompio), gc.mem_free()) # WTF2 del _master_decompio gc.collect() #print('mem_free after dealloc', gc.mem_free()) _master_decompio = _DecompIO(f) self._id = id(_master_decompio) #print('mem_free after alloc', self._id, gc.mem_free()) def read(self, nbytes): global _master_decompio assert self._id == id(_master_decompio) return _master_decompio.read(nbytes) def readline(self): global _master_decompio assert self._id == id(_master_decompio) return _master_decompio.readline()
At least it seems stable so far! 🤞😅
Curiously, the two
# WTFlines I tagged are necessary, otherwise the next alloc will fail. I don't have a working theory as to why this would be...EDIT 4/2/3023: Sadly not stable. Just less frequent GC misses.
Wow, that's about as elegant as you can get I suppose. 😬
If nothing else your wrapper shows that
DecompIOprobably needs some work.Love your solution to allocating (what I guess is) a 32k buffer up front 😆
Reacted by Derek Andersona minimal example of garbage collection seemingly not working:
MicroPython v1.19.1 on 2022-06-18; ESP32 module with ESP32 Type "help()" for more information. >>> import gc, micropython, io, zlib >>> class X: pass ... >>> micropython.mem_info(1) stack: 720 out of 15360 GC: total: 111168, used: 2768, free: 108400 No. of 1-blocks: 44, 2-blocks: 8, max blk sz: 18, max free sz: 6763 GC memory layout; from 3ffe4db0: 00000: h=hhhhBMShDBhFh=hhh=================hh=======h=======h=hh======= 00400: ========ShShhhhhShT=hh======hh========h==h==h==hBh=hh=hhhBBDhhh= 00800: =h=B.h=======h==========h===h=====hhhhh==........h=...h==....... (105 lines all free) 1b000: .................................... >>> x = X() >>> x.d = zlib.DecompIO(io.BytesIO(b'x\x9c\x03\x00\x00\x00\x00\x01')) >>> del x.d >>> x.d Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'X' object has no attribute 'd' >>> gc.collect() >>> micropython.mem_info(1) stack: 720 out of 15360 GC: total: 111168, used: 36096, free: 75072 No. of 1-blocks: 31, 2-blocks: 7, max blk sz: 2048, max free sz: 4579 GC memory layout; from 3ffe4db0: 00000: h=hhhh=MhhDB..h=hhh=================hh=======h=======h=h.......h 00400: =...h==.....................................h==..h=.h=.....Dh... 00800: ....hh=======h==========h===h=====...........h.....hhS...h====hh 00c00: ..h====h..h.h.............h===================================== 01000: ===========================================h==================== 01400: ================================================================ 01800: ================================================================ 01c00: ================================================================ 02000: ================================================================ 02400: ================================================================ 02800: ================================================================ 02c00: ================================================================ 03000: ================================================================ 03400: ================================================================ 03800: ================================================================ 03c00: ================================================================ 04000: ================================================================ 04400: ================================================================ 04800: ================================================================ 04c00: ================================================================ 05000: ================================================================ 05400: ================================================================ 05800: ================================================================ 05c00: ================================================================ 06000: ================================================================ 06400: ================================================================ 06800: ================================================================ 06c00: ================================================================ 07000: ================================================================ 07400: ================================================================ 07800: ================================================================ 07c00: ================================================================ 08000: ================================================================ 08400: ================================================================ 08800: ================================================================ 08c00: ================================================================ 09000: ===========================================.S.....h==..hhh.hh... 09400: B............................................................... (70 lines all free) 1b000: .................................... >>> del x >>> gc.collect() >>> micropython.mem_info(1) stack: 720 out of 15360 GC: total: 111168, used: 1824, free: 109344 No. of 1-blocks: 24, 2-blocks: 7, max blk sz: 18, max free sz: 4588 GC memory layout; from 3ffe4db0: 00000: h=hhhh=Mh=Dhhhh=hhh=================Bh=======h=======h=h...hh..h 00400: =h.h...B.....h==.....h...........................h=........Dh... 00800: ....hh=======h==========h===h=====.......................h====.. 00c00: ..h====h....h................................................... (32 lines all free) 09000: .......................................................h........ (71 lines all free) 1b000: .................................... >>>- added a commit that references this issue
on Apr 2, 2023 proposed: keredson@7dc82cf
there's some housekeeping needed. arg checks. what to do if the wbits are different between the two streams.
also i can't figure out how to make it return none... 🤦 but it works! (and no more OOMs when parsing a few hundred small streams!)FYI this is the micropython git client i've been working on: https://github.com/keredson/ygit
return mp_const_none;should do it.Reacted by Derek Anderson
This is a common error when creating a
zlib.DecompIO(f):Obviously the buffer is necessary (for any streams you don't control the creation of, almost always w/
wbits=15), but that's a third of available ram, and sometimes the GC doesn't purge the old one in time to create the new one, or because of fragmentation there's no contiguous space available.Could we add a function to reset the internal state? IE
decomp.reset(f)That way it could reuse the previous allocation, eliminating the risk of an OOM if resetting instead of creating anew.Would you be willing to sponsor this work? Yes, if the concept is agreed to.