Repository navigation
fread ignores first 2 lines of file with tabs after data in several lines #1416
Description
Activity
It seems the
fill = TRUEargument (since v1.9.8) takes care of this issue. Can you confirm @tdhock?now I get
> fread("iterations.txt") V1 V2 V3 V4 V5 1: 2 -3274352 63539392.790 23.6 NA 2: 3 -2166076 1108276.003 34.5 NA ... 98: 99 -1515355 1.227 1090.5 NA 99: 100 -1515354 1.222 1102.0 NA V1 V2 V3 V4 V5 Warning message: In fread("iterations.txt") : Starting data input on line 3 <<2 -3274351.940 63539392.790 23>> with 5 fields and discarding line 2 <<1 -66813744.730 - 12.3>> before it because it has a different number of fields (4).
above, without
fill=TRUEI get a warning, which is definitely helpful. but it could be more helpful, if you actually add something like "try fill=TRUE to read rather than drop these data." Would you consider adding that to the warning?> fread("iterations.txt", fill=TRUE) Iteration EstimatedLogLikelihood Change TotalTimeSec V5 1: 1 -66813745 - 12.3 NA 2: 2 -3274352 63539392.790 23.6 NA ... 99: 99 -1515355 1.227 1090.5 NA 100: 100 -1515354 1.222 1102.0 NA Iteration EstimatedLogLikelihood Change TotalTimeSec V5 >
above, with
fill=TRUEI get all of the data that I get from read.table, plus an extra column with missing values, which is fine with me.this was with data table 1.10.5
> devtools::session_info() Session info ------------------------------------------------------------------- setting value version R version 3.3.3 (2017-03-06) system i686, linux-gnu ui X11 language en_US collate en_US.UTF-8 tz posixrules date 2017-05-25 Packages ----------------------------------------------------------------------- package * version date source bitops 1.0-6 2013-08-17 CRAN (R 3.2.2) caTools 1.17.1 2014-09-10 CRAN (R 3.2.2) data.table * 1.10.5 2017-04-21 local devtools 1.12.0 2016-12-05 CRAN (R 3.3.3) digest 0.6.12 2017-01-27 cran (@0.6.12) lattice * 0.20-34 2016-09-06 CRAN (R 3.3.3) memoise 1.0.0 2016-01-29 CRAN (R 3.2.2) namedCapture * 2017.01.15 2017-04-29 Github (tdhock/namedCapture@1da425b) RColorBrewer * 1.1-2 2014-12-07 CRAN (R 3.2.2) RCurl * 1.96-0 2016-08-07 local requireGitHub 2017.03.16 2017-04-29 Github (tdhock/requireGitHub@5de2020) RJSONIO * 1.3-0 2014-07-28 CRAN (R 3.2.2) RSelenium * 1.3.6 2016-11-09 Github (ropensci/RSelenium@22f06b9) withr 1.0.1 2016-02-04 CRAN (R 3.2.2) XML * 3.99-0 2016-08-07 local >
- added this to the milestone
on Jan 22, 2018 - added a commit that references this issue
on Jan 22, 2018 this issue is still active, i.e. I am still getting an un-informative message -- it would be much more helpful if you could tell the user to do fill=TRUE.
> fread("iterations.txt") Iteration EstimatedLogLikelihood Change TotalTimeSec 1: 1 -66813745 - 12.3 Warning message: In fread("iterations.txt") : Stopped early on line 3. Expected 4 fields but found 5. Consider fill=TRUE and comment.char=. Fi\ rst discarded non-empty line: <<2 -3274351.940 63539392.790 23.6 >> > fread("iterations.txt", fill=TRUE) Iteration EstimatedLogLikelihood Change TotalTimeSec V5 1: 1 -66813745 - 12.3 NA 2: 2 -3274352 63539392.790 23.6 NA ... 99: 99 -1515355 1.227 1090.5 NA 100: 100 -1515354 1.222 1102.0 NA Iteration EstimatedLogLikelihood Change TotalTimeSec V5 !> sessionInfo() R version 3.5.0 (2018-04-23) Platform: x86_64-pc-linux-gnu (64-bit) Running under: CentOS Linux 7 (Core) Matrix products: default BLAS: /cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/intel2016.4/r/3.5.0/lib64\ /R/lib/libR.so LAPACK: /cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/intel2016.4/r/3.5.0/lib\ 64/R/modules/lapack.so locale: [1] LC_CTYPE=fr_CA.UTF-8 LC_NUMERIC=C LC_TIME=fr_CA.UTF-8 [4] LC_COLLATE=fr_CA.UTF-8 LC_MONETARY=fr_CA.UTF-8 LC_MESSAGES=fr_CA.UTF-8 [7] LC_PAPER=fr_CA.UTF-8 LC_NAME=C LC_ADDRESS=C [10] LC_TELEPHONE=C LC_MEASUREMENT=fr_CA.UTF-8 LC_IDENTIFICATION=C attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] data.table_1.12.3 namedCapture_2019.02.21 loaded via a namespace (and not attached): [1] pak_0.1.2 ps_1.3.0 prettyunits_1.0.2 withr_2.1.2 crayon_1.3.4 [6] rprojroot_1.3-2 assertthat_0.2.1 R6_2.4.0 backports_1.1.4 magrittr_1.5 [11] cli_1.1.0 curl_4.0 remotes_2.1.0 callr_3.3.1 tools_3.5.0 [16] compiler_3.5.0 processx_3.4.1 pkgbuild_1.0.3 >actually never mind, the message does say fill=TRUE now, so i'm closing this
Reacted by Jan Gorecki and Matt Dowle
To see the problem please download https://gist.githubusercontent.com/tdhock/bc1d9c0a76e4df386199/raw/fe436fb41e42c9ba89d8b567abfca587d323aa36/iterations.txt
and then run
The problem seems to be that the first two lines do not have a trailing tab but all the rest of the lines do (screenshot from emacs which highlights all the tab characters):
I expected the output of read.table:
Thanks for looking into this,