Repository navigation
Improve io.ascii guessing performance and usability #2881
Description
As noted in #2877, in guessing mode the memory and speed of io.ascii has room for improvement. Some quick ideas:
- Register a callback to give applications a way to monitor progress
- Short-circuiting of failed guesses. The pure Python code can end up reading the entire table for each guess before deciding if it was OK (as long as no exception gets raised first).
Activity
embray commentedon Aug 21, 2014
The short-circuiting sounds like a good idea. I had something like that in mind too but I'm not quite sure how it would work.
Since most tables are going to be line-oriented anyway maybe it could start by reading just a few lines and trying to figure things out. That won't work for everything, such as tables containing string values that contain newlines (ugh). But it would be good for a first-pass (maybe falling back on reading the full table only if nothing else is found).
taldcroft commentedon Dec 20, 2014
This is still an issue, where here
outis a 10000 row table with 5 columns:One easy fix is documentation, e.g. putting in a highlighted Note box in the
Getting started=>Reading tablessection which reminds people to use the format (which is frequentlybasic) for any biggish files.The second is a little more complicated, but perhaps there could be a
max_rowsargument to the readers which would limit the number of data rows that get read in. This is independent ofdata_end. Then in the guessing process there could be a two-pass guess of first trying the table withmax_rows=100and only if that succeeds go on to the full table.cc: @hamogu @mdmueller
astrofrog commentedon Dec 21, 2014
I like the idea of limiting the number of rows read in during guessing.
hamogu commentedon Dec 21, 2014
Since we currently read the table into an array once before the guessing starts, it is easy to just pass a shortened version of the stay to the readers.
@taldcroft are you working on it? Otherwise I'll do that over Christmas so that we can still get it into 1.0 as a bug fix (just pushing the boundaries not to call it a feature here...). If it is as easy as I think it is after looking at the code this does not require more that a dozen lines of code.
I only need to give some more thought to testing...
taldcroft commentedon Dec 21, 2014
I don't think that passing a shortened version of the input lines is robust because you might have an input with 1000 lines of comments. There really needs to be another parameter
max_rowsthat finds its way into the data reader. BTWmax_rowsis a very useful enhancement on its own as an addition to the top-level API. This was recently added tonp.genfromtxt(or at least was discussed on the list).hamogu commentedon Dec 21, 2014
I was envisioning two new configuration parameters with sensible defaults so that most people don't have to care about them, but if you have a table that is special, then you have the option to fiddle with it:
max_rows_guessingwould define how many rows are passed in to the guessing process. I think 1000 is a sensible default value. Most tables will not have that many comment lines. Once is passes guessing with 1000 rows, the guessing would attempt to read the whole table with the same format and if that fails (e.g. the first 1000 rows happen to look like a csv, but then there is no commas at all on line 1001) it would continue guessing.Clearly, trying a shortened version of the table first and than a the full version introduces a new overhead, and only makes sense if the table is significantly longer than
max_rows_guessing.use_max_rows_guessingwould set the the minimum number of rows where themax_rows_guessingis attempted. I'm thinking of 5000 as default start value, but the best value clearly depends on how many columns you have, if one of the fast readers succeeds, etc.@taldcroft : In which situation, that is not addressed by
data_endalready, would you benefit frommax_rows?taldcroft commentedon Dec 21, 2014
The usual reason I have used
data_endis to chop off cruft at the end of the data file, typically with a negative value. So I think ofdata_endas related to specifying the format of the input data file, whilemax_rowsis a generic concept of limiting data rows. Admittedly the use of a non-defaultdata_endto clip cruft is not a terribly common case and otherwise it's possible to usedata_endto have the same effect. So we can leave things at that without addingmax_rows.I'm good with your idea of a config param
max_rows_guessing. I think that in 98% of files you can tell by the first 100 rows, but we don't need to decide on that now. However, it's important that this parameter apply to rows of data actually read in the reader and not just clipping input data lines. It doesn't matter if most files don't have 1000 lines of comments because some will. I just don't like implementations where there is an obvious path to failure. Yes, the user can change their config for that one file, but most won't figure that out. Better to make the code do the right thing.taldcroft commentedon Dec 21, 2014
The reason I was thinking about a smallish number for
max_rows_guessingis that we can just always do that and not worry aboutuse_max_rows_guessing.taldcroft commentedon Apr 17, 2015
Another idea just discussed with @hamogu would be a standard
fast_identifymethod which, if available, will quickly identify content that could be valid for the reader type. It doesn't have to be 100% accurate, i.e. false positives are OK, but false negatives would not be OK. This is related to the discussion in #3693.