Repository navigation
Improve io.ascii guessing performance and usability #2881
Description
Activity
The short-circuiting sounds like a good idea. I had something like that in mind too but I'm not quite sure how it would work.
Since most tables are going to be line-oriented anyway maybe it could start by reading just a few lines and trying to figure things out. That won't work for everything, such as tables containing string values that contain newlines (ugh). But it would be good for a first-pass (maybe falling back on reading the full table only if nothing else is found).
This is still an issue, where here
outis a 10000 row table with 5 columns:In [59]: timeit t2 = ascii.read(out) 10 loops, best of 3: 76.8 ms per loop In [60]: timeit t2 = ascii.read(out, format='basic') 100 loops, best of 3: 5.89 ms per loopOne easy fix is documentation, e.g. putting in a highlighted Note box in the
Getting started=>Reading tablessection which reminds people to use the format (which is frequentlybasic) for any biggish files.The second is a little more complicated, but perhaps there could be a
max_rowsargument to the readers which would limit the number of data rows that get read in. This is independent ofdata_end. Then in the guessing process there could be a two-pass guess of first trying the table withmax_rows=100and only if that succeeds go on to the full table.cc: @hamogu @mdmueller
I like the idea of limiting the number of rows read in during guessing.
Since we currently read the table into an array once before the guessing starts, it is easy to just pass a shortened version of the stay to the readers.
@taldcroft are you working on it? Otherwise I'll do that over Christmas so that we can still get it into 1.0 as a bug fix (just pushing the boundaries not to call it a feature here...). If it is as easy as I think it is after looking at the code this does not require more that a dozen lines of code.
I only need to give some more thought to testing...I don't think that passing a shortened version of the input lines is robust because you might have an input with 1000 lines of comments. There really needs to be another parameter
max_rowsthat finds its way into the data reader. BTWmax_rowsis a very useful enhancement on its own as an addition to the top-level API. This was recently added tonp.genfromtxt(or at least was discussed on the list).I was envisioning two new configuration parameters with sensible defaults so that most people don't have to care about them, but if you have a table that is special, then you have the option to fiddle with it:
max_rows_guessingwould define how many rows are passed in to the guessing process. I think 1000 is a sensible default value. Most tables will not have that many comment lines. Once is passes guessing with 1000 rows, the guessing would attempt to read the whole table with the same format and if that fails (e.g. the first 1000 rows happen to look like a csv, but then there is no commas at all on line 1001) it would continue guessing.
Clearly, trying a shortened version of the table first and than a the full version introduces a new overhead, and only makes sense if the table is significantly longer thanmax_rows_guessing.use_max_rows_guessingwould set the the minimum number of rows where themax_rows_guessingis attempted. I'm thinking of 5000 as default start value, but the best value clearly depends on how many columns you have, if one of the fast readers succeeds, etc.
@taldcroft : In which situation, that is not addressed by
data_endalready, would you benefit frommax_rows?In which situation, that is not addressed by data_end already, would you benefit from max_rows?
The usual reason I have used
data_endis to chop off cruft at the end of the data file, typically with a negative value. So I think ofdata_endas related to specifying the format of the input data file, whilemax_rowsis a generic concept of limiting data rows. Admittedly the use of a non-defaultdata_endto clip cruft is not a terribly common case and otherwise it's possible to usedata_endto have the same effect. So we can leave things at that without addingmax_rows.I'm good with your idea of a config param
max_rows_guessing. I think that in 98% of files you can tell by the first 100 rows, but we don't need to decide on that now. However, it's important that this parameter apply to rows of data actually read in the reader and not just clipping input data lines. It doesn't matter if most files don't have 1000 lines of comments because some will. I just don't like implementations where there is an obvious path to failure. Yes, the user can change their config for that one file, but most won't figure that out. Better to make the code do the right thing.The reason I was thinking about a smallish number for
max_rows_guessingis that we can just always do that and not worry aboutuse_max_rows_guessing.Another idea just discussed with @hamogu would be a standard
fast_identifymethod which, if available, will quickly identify content that could be valid for the reader type. It doesn't have to be 100% accurate, i.e. false positives are OK, but false negatives would not be OK. This is related to the discussion in #3693.
As noted in #2877, in guessing mode the memory and speed of
io.asciihas room for improvement. Some quick ideas: