Repository navigation
memory allocation segfault in read_asciitable #2877
Description
The following script inconsistently throws segfaults when using the file https://www.dropbox.com/s/546pjh6i6q66iuk/test.csv
from astropy import __version__
print __version__
from astropy.io.ascii.connect import read_asciitable
read_asciitable('test.csv')
(whether or not it actually throws a segfault seems to depend on how much memory I'm using elsewhere on my system).
One crash report reads:
Process: python [58553]
Path: /Users/USER/*/python.app/Contents/MacOS/python
Identifier: com.continuum.python
Version: ???
Code Type: X86-64 (Native)
Parent Process: bash [58409]
User ID: 501
Date/Time: 2014-08-20 13:57:06.928 -0400
OS Version: Mac OS X 10.8.5 (12F45)
Report Version: 10
Interval Since Last Report: 78082 sec
Crashes Since Last Report: 12
Per-App Crashes Since Last Report: 12
Anonymous UUID: 9F0D1600-F5A2-ABFF-0692-767B75E42AA8
Crashed Thread: 0 Dispatch queue: com.apple.main-thread
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000114e89000
VM Regions Near 0x114e89000:
MALLOC_LARGE 0000000114e68000-0000000114e89000 [ 132K] rw-/rwx SM=COW
-->
MALLOC_LARGE 0000000114e8c000-0000000114f0c000 [ 512K] rw-/rwx SM=PRV
For context, the catalog is 2M rows x 7 columns, 100MB in plaintext, and my machine has 16GB of RAM. Pandas loads the data without problem. Because of this, I suspect that read_asciitable is somewhere allocating an array much larger than it needs to.
Activity
embray commentedon Aug 20, 2014
Iiinteresting-- @AmrAS1 , could this be related to any of the work you've done, or so is this something unique?
embray commentedon Aug 20, 2014
Weird--this is not an unusually large CSV file, yet in reading it the Python process's memory usage climbed to over 3 GB (before settling to around 2.7 GB). There's no reason for the converted data to be anywhere near that large, so there must be something fishy going on there...
embray commentedon Aug 20, 2014
A lot of the trouble appears to be occurring in
_guess.astrofrog commentedon Aug 20, 2014
@ChrisBeaumont @embray - ah, do you see the same memory increase if you specify
format='ascii.csv'? My suspicion is that file handles may not be getting closed in the guesser (which basically tries all formats)Maybe @taldcroft can comment here?
taldcroft commentedon Aug 20, 2014
Interesting. When I read this on a mac with 0.4.1 the peak memory usage was about 1.1 Gb, which is roughly consistent with expectations for the pure Python reader. It basically reads the entire ASCII, then parses it into lists of Python string objects. It all adds up.
@ChrisBeaumont - which version are you using? I don't see that in your output. In any case, I could not reproduce the segfault.
This should be a thing of the past, soon... using #2716 the file reads much more quickly and with less memory overhead.
mdmueller commentedon Aug 20, 2014
@embray - This isn't directly related to the work I've done so far, but I am looking now at a better memory allocation scheme for the fast reader (and this issue is certainly good motivation)! You're right, I'm pretty sure it has to do with
_guess...mdmueller commentedon Aug 20, 2014
@taldcroft - Hopefully once I figure out the memory issue! As of right now #2716 won't really help in terms of memory overhead, although obviously the speed will increase significantly.
taldcroft commentedon Aug 20, 2014
It's true that I was using
Table.read('test.csv', format='ascii.csv').@ChrisBeaumont - what led you to directly using
read_asciitable? Was this a secondary step after trying to debug?embray commentedon Aug 20, 2014
Even if using
format='csv'it's still pretty slow but that's what @AmrAS1's work should fix. But it doesn't blow up like this, at least. The problem is definitely in the format guessing...ChrisBeaumont commentedon Aug 20, 2014
@taldcroft yes I'm using read_asciitable as a secondary step -- I first triggered this behavior after registering a couple of extra handlers to the io registry, and I wanted to isolate the lowest-level routine that causes the problem.
I'm on v0.4.1
embray commentedon Aug 20, 2014
Won't it though? The use of mmap still avoids having to load the original file contents all into main memory, at least.
mdmueller commentedon Aug 20, 2014
Yes, that's true. However, the tokenizer still has to store a new, altered copy of the input in memory as part of tokenization (Pandas does this as well--for various reasons, this is probably necessary as alternative solutions could be more costly). One problem is that the io.ascii tokenizer stores output in different column strings, and
realloc()has to double the size of each when necessary, which can be wasteful. That doesn't fully explain the discrepancy between the fast reader and Pandas, though (by the way, Pandas doesn't hold the entire file in memory, it reads chunks into a buffer and loops over reading the buffer until the entire file is read). I'm still looking into it but I think Pandas manages to deallocate memory in the tokenizer at some stage of conversion, so the time in which C and Python objects coexist is kept to a minimum.taldcroft commentedon Aug 20, 2014
In my testing just now the new fast-c-reader seems much better for memory. From what I could see both peak and final resident memory were around a factor of 10 lower using the fast reader. I'm doing this very crudely using top, and maybe I don't understand the details, but at the end the fast reader version is using 139 Mb internal while the Python one has 1178 Mb.
The peak memory I could see with the fast reader was around 400 Mb, but it goes by quickly...
mdmueller commentedon Aug 20, 2014
Interesting. I've been testing with Massif (also crudely--I'm certainly not experienced with memory testing) and have found the peak memory usage to be actually slightly higher for the fast reader, but I'll try using top and see if I can figure out what's going on.
taldcroft commentedon Aug 20, 2014
@embray - I agree that the problem is definitely related to
_guess()inui.py. When I try to read this withguess=Truethen the internal application memory stays below 2 Gb, but the virtual memory keeps climbing with time up through 8 Gb (where I ctrl-c'd out before it crashed my mac which just has 4 Gb).What is that telling us, or what in Python could be causing that? In theory
_guessis just trying to read the file a bunch of times in different ways, and it shouldn't be hanging on to data from one format to the next. Somehow if there was a reference count not going to zero I would have thought the real memory would keep going up.@ChrisBeaumont - is it an option for you to turn off guessing in this context?
21 remaining items
embray commentedon Aug 20, 2014
#2879 and #2880 provide two small patches that should lessen the impact of this issue.
embray commentedon Aug 20, 2014
[Sort of as a side note, thinking about things in the context of Glue, it might be nice if it were possible to register a callback function that is called within the main loop of
_guess, so that applications could provide some feedback to users on what it's doing (or at the very least move a progress bar forward).]hamogu commentedon Aug 25, 2014
Could it be that the same circular referencing issue is going on inside the asciitable Reader object, so that those don't get gc collected either?
Every
Headerobject has a reference to theDataobject of that table and everyDataobject has a reference to theHeader.https://github.com/astropy/astropy/blob/master/astropy/io/ascii/core.py#L827
embray commentedon Aug 26, 2014
@hamogu Maybe but I'd be surprised if that's an issue here. Python's garbage collector can detect and break up reference cycles except in some specific cases. Would have to take a look.
Should fix the reference cycle found in astropy#2877 by storing paren…
taldcroft commentedon Oct 21, 2014
Using the latest master, even after merging #2892 there seems to be some serious bloat associated with guessing. Letting it go through the pure guessing is really bad, using a peak of 5 Gb and taking 80 seconds (!). In contrast:
My by-eye monitoring of memory showed it used a peak of a few hundred Mb (the file is 136 Mb), which is fine. Using the Basic format with
delimiter=','is comparable.One possibility is to add a way of specifying a number of data lines to use in a "pre-guess" phase. I.e. guess the format using the first 100 lines and start with that, then fall back to the normal guess strategy if that fails on the full file. In theory we could overload the
guessparameter to also accept an integer that specifies the number of lines for pre-guessing. I imagine with a default of somewhere between 100 and 1000 it would not break back-compatibility and would much improve big-file reading cases like this.embray commentedon Dec 30, 2014
This issue has been somewhat mitigated by a number of fixes, but as the above comment notes there are still some issues here to address so I will leave this open.
bsipocz commentedon Jun 20, 2019
I'm closing this as #2881 seems to be the perfect follow-up issue to track things not yet addressed here.