Skip to content

change the order of file.exists(input) and grep('\\n|\\r', input) in fread.R - #2630

Merged
mattdowle merged 1 commit into
Rdatatable:masterfrom
javrucebo:master
Feb 17, 2018
Merged

mattdowle merged 1 commit into
Rdatatable:masterfrom
javrucebo:master

Conversation

@javrucebo

@javrucebo javrucebo commented Feb 16, 2018 •

Copy link
Copy Markdown
Contributor

change the order of file.exists(input) and grep('\\n|\\r', input)
in fread.R in order to speed up execution time when input is the
actual data. file.exists is very slow, so checking first
for new lines speeds up the process.

For benchmark on file.exists see also this comment in #2531

Not adding additional tests, as no change in functionality, just speed up of execution. (current tests all pass locally).
Not adding NEWS entry, as there is already one for #2531 and this PR is just a trivial addition.

in `fread.R` in order to speed up execution time when `input` is the
actual data. `file.exists` is very slow, so checking first
for new lines speeds up the process.

See also discussion in Rdatatable#2531
@MichaelChirico

MichaelChirico commented Feb 16, 2018 via email •

Copy link
Copy Markdown
Member

@javrucebo

javrucebo commented Feb 16, 2018 •

Copy link
Copy Markdown
Contributor Author

@MichaelChirico
The original reported issue with grepl and the use here are distinct cases.
There is not much difference in testing for newlines between grep and grepl.
What was slow is the anchored search at the beginning of the string - clearly this should run much faster and should be fixed in base R

input <- paste(rep("1,2,3,4.567,some text field", 1e7),collapse="\n")
system.time(length(grep("\n", input)))
##   user  system elapsed 
##  0.000   0.000   0.001 
system.time((grepl("\n", input)))
##   user  system elapsed 
##      0       0       0 
system.time((grepl("^http://", input)))
##   user  system elapsed 
##  2.683   0.000   2.685 

@HughParsonage

Copy link
Copy Markdown
Member

True, though checking file.exists is faster than checking for newlines then checking file.exists.

It would seem grepl("\n", ., fixed = TRUE) || grepl("\r", ., fixed = TRUE) is faster still, noting we have already checked that length(.) == 1 a few lines earlier.

@javrucebo

javrucebo commented Feb 16, 2018 •

Copy link
Copy Markdown
Contributor Author

@HughParsonage
the point here is, if input is the actual raw data - and therefore potentially very large - then file.exists is really slow and eating up most of execution time of fread.

see my comment in #2531 with the benchmark of file.exists

If on the other hand input is a file name, then we can safely assume that it is a relatively short string (not megabytes) and having the length(grep('\\n|\\r', input)) check before file.exists does not add a real penalty for small inputs.

then length(.)==1 check you refer to is checking something else - whether the input is a vector with length > 1, which fread does not accept.

@codecov-io

Copy link
Copy Markdown

Codecov Report

Merging #2630 into master will not change coverage.
The diff coverage is 100%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #2630   +/-   ##
=======================================
  Coverage   93.03%   93.03%           
=======================================
  Files          61       61           
  Lines       12115    12115           
=======================================
  Hits        11271    11271           
  Misses        844      844
Impacted Files Coverage Δ
R/fread.R 96.18% <100%> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8ab3028...1fb6c48. Read the comment docs.

@mattdowle mattdowle left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for investigating and fixing. Yes, I thought file.exists() would be fast to return false on large input. Interesting it isn't. And it's good to avoid too many OS calls too I suppose, say if fread is being calling a lot in a loop with direct character input.. Nice fix.

@mattdowle
mattdowle merged commit 3989ea2 into Rdatatable:master Feb 17, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants