Repository navigation
memory allocation segfault in read_asciitable #2877
Description
Activity
Weird--this is not an unusually large CSV file, yet in reading it the Python process's memory usage climbed to over 3 GB (before settling to around 2.7 GB). There's no reason for the converted data to be anywhere near that large, so there must be something fishy going on there...
A lot of the trouble appears to be occurring in
_guess.@ChrisBeaumont @embray - ah, do you see the same memory increase if you specify
format='ascii.csv'? My suspicion is that file handles may not be getting closed in the guesser (which basically tries all formats)Maybe @taldcroft can comment here?
Interesting. When I read this on a mac with 0.4.1 the peak memory usage was about 1.1 Gb, which is roughly consistent with expectations for the pure Python reader. It basically reads the entire ASCII, then parses it into lists of Python string objects. It all adds up.
@ChrisBeaumont - which version are you using? I don't see that in your output. In any case, I could not reproduce the segfault.
This should be a thing of the past, soon... using #2716 the file reads much more quickly and with less memory overhead.
@embray - This isn't directly related to the work I've done so far, but I am looking now at a better memory allocation scheme for the fast reader (and this issue is certainly good motivation)! You're right, I'm pretty sure it has to do with
_guess...@taldcroft - Hopefully once I figure out the memory issue! As of right now #2716 won't really help in terms of memory overhead, although obviously the speed will increase significantly.
It's true that I was using
Table.read('test.csv', format='ascii.csv').@ChrisBeaumont - what led you to directly using
read_asciitable? Was this a secondary step after trying to debug?@taldcroft yes I'm using read_asciitable as a secondary step -- I first triggered this behavior after registering a couple of extra handlers to the io registry, and I wanted to isolate the lowest-level routine that causes the problem.
I'm on v0.4.1
Yes, that's true. However, the tokenizer still has to store a new, altered copy of the input in memory as part of tokenization (Pandas does this as well--for various reasons, this is probably necessary as alternative solutions could be more costly). One problem is that the io.ascii tokenizer stores output in different column strings, and
realloc()has to double the size of each when necessary, which can be wasteful. That doesn't fully explain the discrepancy between the fast reader and Pandas, though (by the way, Pandas doesn't hold the entire file in memory, it reads chunks into a buffer and loops over reading the buffer until the entire file is read). I'm still looking into it but I think Pandas manages to deallocate memory in the tokenizer at some stage of conversion, so the time in which C and Python objects coexist is kept to a minimum.In my testing just now the new fast-c-reader seems much better for memory. From what I could see both peak and final resident memory were around a factor of 10 lower using the fast reader. I'm doing this very crudely using top, and maybe I don't understand the details, but at the end the fast reader version is using 139 Mb internal while the Python one has 1178 Mb.
The peak memory I could see with the fast reader was around 400 Mb, but it goes by quickly...
Interesting. I've been testing with Massif (also crudely--I'm certainly not experienced with memory testing) and have found the peak memory usage to be actually slightly higher for the fast reader, but I'll try using top and see if I can figure out what's going on.
@embray - I agree that the problem is definitely related to
_guess()inui.py. When I try to read this withguess=Truethen the internal application memory stays below 2 Gb, but the virtual memory keeps climbing with time up through 8 Gb (where I ctrl-c'd out before it crashed my mac which just has 4 Gb).What is that telling us, or what in Python could be causing that? In theory
_guessis just trying to read the file a bunch of times in different ways, and it shouldn't be hanging on to data from one format to the next. Somehow if there was a reference count not going to zero I would have thought the real memory would keep going up.@ChrisBeaumont - is it an option for you to turn off guessing in this context?
21 remaining items
[Sort of as a side note, thinking about things in the context of Glue, it might be nice if it were possible to register a callback function that is called within the main loop of
_guess, so that applications could provide some feedback to users on what it's doing (or at the very least move a progress bar forward).]Could it be that the same circular referencing issue is going on inside the asciitable Reader object, so that those don't get gc collected either?
EveryHeaderobject has a reference to theDataobject of that table and everyDataobject has a reference to theHeader.https://github.com/astropy/astropy/blob/master/astropy/io/ascii/core.py#L827
- added a commit that references this issue
on Aug 26, 2014 Using the latest master, even after merging #2892 there seems to be some serious bloat associated with guessing. Letting it go through the pure guessing is really bad, using a peak of 5 Gb and taking 80 seconds (!). In contrast:
In [2]: time ascii.read('test.csv', format='csv', guess=False) CPU times: user 3.92 s, sys: 514 ms, total: 4.43 s Wall time: 5.82 sMy by-eye monitoring of memory showed it used a peak of a few hundred Mb (the file is 136 Mb), which is fine. Using the Basic format with
delimiter=','is comparable.One possibility is to add a way of specifying a number of data lines to use in a "pre-guess" phase. I.e. guess the format using the first 100 lines and start with that, then fall back to the normal guess strategy if that fails on the full file. In theory we could overload the
guessparameter to also accept an integer that specifies the number of lines for pre-guessing. I imagine with a default of somewhere between 100 and 1000 it would not break back-compatibility and would much improve big-file reading cases like this.This issue has been somewhat mitigated by a number of fixes, but as the above comment notes there are still some issues here to address so I will leave this open.
The following script inconsistently throws segfaults when using the file https://www.dropbox.com/s/546pjh6i6q66iuk/test.csv
(whether or not it actually throws a segfault seems to depend on how much memory I'm using elsewhere on my system).
One crash report reads:
For context, the catalog is 2M rows x 7 columns, 100MB in plaintext, and my machine has 16GB of RAM. Pandas loads the data without problem. Because of this, I suspect that read_asciitable is somewhere allocating an array much larger than it needs to.