Repository navigation
Memory is slowly leaking #2019
Description
Activity
import os import PIL from PIL import Image dir = "/Volumes/Seagate/storage/kaggle/avito/images" onlyfiles = [] dirs = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14] for d in dirs: filedir = os.path.join(dir, str(d)) files = [f for f in os.listdir(filedir) if os.path.isfile(os.path.join(filedir, f))] filepaths = map( lambda x: os.path.join(filedir, x), files ) onlyfiles += list(filepaths) s = 0 for i in range(len(onlyfiles)): fname = onlyfiles[i] image = PIL.Image.open(fname) # Do something s += image.size[0]*image.size[1] if i % 10000 == 0: print('iteration ', i) input("Press Enter to continue...")
Input data: image dataset (10M small images) from Kaggle competition https://www.kaggle.com/c/avito-duplicate-ads-detection
Result: It starts with 221MB for the file names.
500 images - 374MB
1M images - 510MBgc() inside the loop did not help.
Do we know what release this started in? I'd like to pin to a lower release to mitigate any production impact without having to set memory scaling alarm/actions.
@dmpetrov I can't reproduce this on Ubuntu 14.04 with Python 2.7 on Pillow master. Could you tell your Pillow, Python, OS, libjpeg versions?
@damaestro This is not confirmed yet.
- changed the title
[-]Memory is slowly leaking[/-][+]Memory is slowly leaking in Python 3.x[/+]on Jul 14, 2016 So, this is a minimal test case, which I came to:
https://gist.github.com/homm/c6a79ce7b445f47a74a4d296742a5af8
As you can see, there is no Pillow at all! I'm pretty sure this is pure Python 3 memory leak.
The script accepts path to the root folder with tons of files as argument and iterates all files in the folder twice: First time it opens the files and calls
.close()method. At this stage, memory usage doesn't grow. Second time the script doesn't call.close()method and memory usage constanlty increasing. I've tested Python 3.4 and 3.5 on OS X and Python 3.4 on Ubuntu 14.04. It doesn't appear on Python 2.7.This is Python's memory leak because there are no pointers to the file objects from the application space. This is most likely only memory leak. At least
lsofdoesn't show any opened files after script's end. This leak appears only on different files. It doesn't work if we are reopening one file again and again. Average memory consumption is 370 bytes per file.As I understand we can't fix it on Pillow's level because we are assuming that file pointer may be shared with the other code. There are two possibilities: It can be fixed in Python itself. Or it can be fixed on the application level. Something like this:
f = open(filename, 'rb') image = Image.open(f) image.load() f.close()
instead of
Image.open(filename)What happens when using
with open(file.path, 'rb', 0) as f:?@homm thank you for the investigation. It is becoming more and more interesting
Yes, I use python 3.5. As far as I know, Python does not guarantee that file will be closed. So, it might be Python 3+ "feature", not a bug.
It would be great to have opinions of Python experts.
Python does not guarantee that file will be closed.
Of course it guarantees. Moreover, it guarantees that exactly
.close()method will be called. It doesn't guarantee when this will be done, that is all.But as I said we are not speaking about the file descriptors leak. All files are closed. The problem is there is a memory leak.
I'm seeing the issue with Py 2.7 running thumbor, so I don't think this is isolated to Py 3.x.
so I don't think this is isolated to Py 3.x.
Any reasons why you are thinking this is the same leak? Memory leaks are not isolated to Python 3 or Python or any other platform or library. In this thread, we are discussing leak in Python 3 which affects Pillow users. You can report thumbor leaks in appropriate place.
30 remaining items
It's hard to follow this issue can someone explain what is the suggested solution?
@SaschaHeyer I think the problem discussed here is likely resolved. If you have a situation, I'd recommend that you open a new issue with a self-contained example
@radarhere
Thank you for your quick response.I wanted to know how the issue is resolved.
What steps are required to solve?If you are using the latest version of Pillow, then make sure that you close images properly. Either by explicitly calling
close(),im = Image.new("RGB", (100, 100)) im.close()
or using a context manager
with Image.open("hopper.jpg") as im: pass
Reacted by FlyingZebra1, julio5pm and GideonLet's close this, and we can re-open if needed, or open a new issue.
@radarhere
Thank you for your quick response.I wanted to know how the issue is resolved.
What steps are required to solve?To clarify, I meant that I suspect it is resolved in the current version of Pillow, fixed by changes to our code, not to yours.
Is this problem got fixed in python2.7 pillow6.2.2
It's not completely clear in this issue what the problem was, or if/how it was fixed. Here are some notes though
- The memory leak in Python itself wasn't able to be reproduced in Python 2.7.
- Resolve __fp when closing and deleting #3261 may have helped, and that was part of Pillow 5.4.0.
- Improve handling of file resources #3577 only removed support for implicitly closing an image's file in Pillow 7.0.0, but you could still call
close()on images or use a context manager previously.
It's not completely clear in this issue what the problem was, or if/how it was fixed. Here are some notes though在这个问题上，并不完全清楚问题是什么，或者是否/如何解决。不过，这里有一些注意事项
- The memory leak in Python itself wasn't able to be reproduced in Python 2.7.Python 本身的内存泄漏无法在 Python 2.7 中重现。
- Resolve __fp when closing and deleting #3261 may have helped, and that was part of Pillow 5.4.0. Resolve __fp when closing and deleting #3261 可能有所帮助，这是 Pillow 5.4.0 的一部分。
- Improve handling of file resources #3577 only removed support for implicitly closing an image's file in Pillow 7.0.0, but you could still call
close()on images or use a context manager previously. Improve handling of file resources #3577 仅在 Pillow 7.0.0 中删除了对隐式关闭映像文件的支持，但close()您仍然可以调用映像或使用上下文管理器。
Our service encountered some memory leaks and continuous memory growth when using Pillow to extract frames from webp animations. Can you identify any potential issues with the following code? Deeply grateful!
- python: 2.7.9
- pillow: 6.2.2
def extract_gif_use_pillow(image): frames = [] try: with Image.open(StringIO(image)) as im: with BytesIO() as frame: while True: try: frame.seek(0) im.save(frame, format='WEBP', quality=100) frames.append(frame.getvalue()) # move to next frame im.seek(im.tell() + 1) except EOFError: break except Exception as e: logger.error('extract_gif_use_pillow got exception: %s', e) gc.collect() logger.info('extract_gif_use_pillow got frames num: %s', len(frames)) return frames
I see no obvious problems, but I do recommend that you use a version of Python that is still maintained, and a more recent version of Pillow. Pillow has also made significant improvements to how it reads GIF images.
You may like to have a read of #7935 (comment). It is possible that is the explanation for your memory issues.
Memory is slowly leaking when process many image files one by one.
Leak size:
100K images --> ~100MB
300K images --> ~200MB
1M images --> ~1GB
Here you can find reproducing code examples: PyWavelets/pywt#180