Repository navigation
fwrite/fread single column input with NA -vs- empty lines #2106
Description
Activity
library(data.table) temp = data.table(a=c(1,NA,2,3,999,NA)) tmp = tempfile() fwrite(temp, tmp, quote=FALSE) system(paste('cat', tmp)) # a # 1 # # 2 # 3 # 999 #There are no separators in any line, and the last line is blank.
Absent separators,
freadhas no way of knowing whether there are just blank lines or whether they are supposed to have missing data.freadwarns you about this:Warning message:
In
fread("temp.csv", stringsAsFactors = F):
Stopped reading at empty line 3 but text exists afterwards (discarded): 2One potential fix: add a dummy column:
temp[ , b := NA]Now
tmpwill have separators sofreadcan tell which lines have data.@MichaelChirico Side note: to cat it to the console, just use
fwrite, which puts it there by default:fwrite(temp) # a # 1 # # 2 # 3 # 999So it can be reproduced like...
fread(paste(capture.output(fwrite(temp)), collapse="\n")) # a # 1: 1 # Warning message: # In fread(paste(capture.output(fwrite(temp)), collapse = "\n")) : # Found the last consistent line but text exists afterwards (discarded): <<2>>Yeah, I'm inclined towards saying the file should be written better if it wants blank lines read as NA (rather than reconfigure fread to treat this one-column case specially). I mean:
fread(paste(capture.output(fwrite(temp, na="NA")), collapse="\n"))@franknarf1 nice, the default to write to
stdoutis an update I missed, wasn't like that initially. Matcheswrite.tablebehavior 👍And I like your fix better, but not sure if it's the user's responsibility to handle a case like that, or if
na = if (ncol(x) > 1L) '' else 'NA'as the default is a better fixReacted by Matt DowleThe warning message is there. And there are arguments to control it.
Using v1.10.4 on CRAN :> fread("temp.csv") a 1: 1 Warning message: In fread("temp.csv") : Stopped reading at empty line 3 but text exists afterwards (discarded): 2 > fread("temp.csv", fill=TRUE) a 1: 1 2: NA 3: 2 4: 3 5: 999 6: NA > fread("temp.csv", blank.lines.skip=TRUE) a 1: 1 2: 2 3: 3 4: 999 >Perhaps "Consider fill=TRUE and blank.lines.skip=TRUE" should be added to the warning message? (TODO1)
Reacted by Frank, Cruiseee and Ronaldo Alves- changed the title
[-]Serious fread problem.[/-][+]fread single column input with NA or empty lines[/+]on Apr 8, 2017 Also the empty lines can be controlled in
fwritewith thena=argument.> temp a 1: 1 2: NA 3: 2 4: 3 5: 999 6: NA > fwrite(temp, "temp.csv") > system("more temp.csv") a 1 2 3 999 > fwrite(temp, "temp.csv", na="NA") > system("more temp.csv") a 1 NA 2 3 999 NA > fread("temp.csv") a 1: 1 2: NA 3: 2 4: 3 5: 999 6: NA >I just read again and understood @MichaelChirico's comment now :
or if na = if (ncol(x) > 1L) '' else 'NA' as the default is a better fix
Yes - nice idea! Happy to make that change. (TODO2)
- added this to the milestone
on Apr 8, 2017 - changed the title
[-]fread single column input with NA or empty lines[/-][+]fwrite/fread single column input with NA -vs- empty lines[/+]on Apr 8, 2017 - added a commit that references this issue
on Oct 30, 2017 Closed by #2451
Great.
I've checked that it's also working well when a whole row is full of NA.temp = data.table(a=c(1,NA,2,3,999,NA), b=c(1,NA,2,3,999,NA))Now that
freadhandles the blank lines in single-column files, this change in dev can be reverted back to how it was on CRAN which is simpler and cleaner.
CRAN version hasfwrite(..., na="", ...)
dev changed tofwrite(..., na = if (length(x) > 1L) "" else "NA", ...)
I create a toy example.
temp <- data.table(a=c(1,NA,2,3,999,NA))
I save it:
fwrite(temp, "temp.csv", quote=FALSE, sep=",", append=F)
and read it again:
my <- fread("temp.csv", stringsAsFactors=F)
As you can see only the first line is read.
I don't know if it's a problem with fread or with fwrite's output file.