Repository navigation
[R-Forge #4931] Support file connections for fread #561
Description
Activity
This would be a pretty awesome feature that I am after as well!!
Reacted by AnantadinathI agree, this would be great.
👍 I need this!
Reacted by AnantadinathAn interesting use case for this feature would be reading chunks from de CSV file and pass it to a worker to process such information to create additive and/or semi-additive metrics. I have a working example using read.table (read.csv2) to deal with CSV file that doesn’t fit the available memory (supposing the need of all fields in the process). However, it is still not possible to use all workers in the best way given the slow nature of read.table. I have great expectation for fread with file connection as input.
library(doSNOW)
library(data.table)
library(iterators)
library(parallel)#(SOCK - Windows - mem copy-on-call - slower)
#(FORK - Linux - mem copy-on-write - faster)
cl <- makeCluster(detectCores(logical=FALSE), type="SOCK")
registerDoSNOW(cl)chunkSize = 250000
conn = file("FILE_BIGGER_THAN_AVAILABLE_MEMORY.csv","r")
header = scan(conn, what=character(), sep=';', nlines=1)it <- iter(function() {
tryCatch({
#EXCELLENT OPPORTUNITY TO TEST FREAD'S FILE CONNECTION FEATURE
chunk = read.csv2(file=conn, header=FALSE, nrows=chunkSize)
colnames(chunk) = header
setDT(chunk)
return(chunk)
}, error=function(e) {
#READ.TABLE THROWS ERROS WHEN A READ IS MADE AFTER EOF
stop("StopIteration", call. = FALSE)
})
})somefun <- function(dt) {
aggreg = dt[,
list(Obs=.N),
by=list(CATEGORICAL_VARIABLE_A)
]
return(aggreg)
}allaggreg <- foreach(slice=it, .packages='data.table', .combine='rbind', .inorder=FALSE) %dopar% {
somefun(slice)
}setkey(allaggreg, CATEGORICAL_VARIABLE_A)
finalaggreg = allaggreg[,
list(Obs=sum(Obs, na.rm=TRUE)),
by=list(CATEGORICAL_VARIABLE_A)
]close(conn)
stopCluster(cl)Reacted by Greg Werbin, Aline Menezes, Michael Chirico and Vathy M. Kamulete+1. I wish I could fread(bzfile("file.csv")).
Reacted by Maurício Ramos, Pascal Bellerose, Greg Werbin, rfaelens, Michael Chirico and Vathy M. Kamulete+1. I'd like to process
stdinusing data.table with Apache Hive / Hadoop. Here's what I currently do:stream_in = file("stdin") open(stream_in) queue = read.table(stream_in, nrows = rows_per_chunk, colClasses = input_classes , col.names = input_cols, na.strings = "\\N") # Then incrementally refresh and process queueReacted by Michael Chirico@mauriciocramos - does this not work for your case:
fread("bunzip2 file.csv")or
fread(sprintf("bunzip2 %s", "file.csv"))I would like to !
I have a huge CSV file (about 7Gb) and a small RAM (about 8Gb). I do chunk with for loop and skip and nrows parameters for extract some features. If it's very efficient at the beginning it's very slow at the end. I would like to memories were I was in the file and don't look at each time for the beginning of my chunk from the beginning of the file.
I think that use connection could help.
I hope that I was clear,
In advance thank you.
Reacted by Florian Privé@st-pasha made several relevant points to this issue in #1721, for example:
Generally, fread algorithm likes to see the whole file in order to properly detect types, number of columns, etc. Also, sometimes it needs several passes to get the result correctly.
Based on those points I actually don't think data.table needs to support general file connections or chunking. Sure, it would be convenient, but probably not worth the future trouble. There's already several existing solutions:
- Dump to a temporary file
- Basic
read.tableis actually reasonably efficient if we can specify column types and number of rows ahead of time. - iotools package already offers fast stream processing reads.
- removed this from the milestone
on May 10, 2018 2 remaining items
a note to explore after implementing: use
textConnectionto handle input likefread('a,b,c 1,2,3 4,5,6')instead of outputting it to disk as is done now if I'm not mistaken.
Curious here if
freadsimply handling the logic of spilling the connection to disk & then reading would be enough for this FR?At a glance I think this implementation doesn't satisfy the "chunked read" use case, am I missing anything else?
@MichaelChirico This would not satisfy my usecase. The idea of using large bzip'd files is precisely to avoid spilling to (slow) disk.
If
freadindeed needs multiple passes and seeks, then either it should useseek()ea or this FR should be closed as WONTFIX.Reacted by AnantadinathI hope the feature would enable reading a few lines of file at a time and do something to these lines in a much more fast way than using readLines.
reading a few lines of file at a time
If it's just a few lines,
readLinesshould be fine (especially usingn=argument and/or passing a connection rather than a file name to get incremental reads)... could you elaborate your use case / whyreadLineswon't suffice?reading a few lines of file at a time
If it's just a few lines,
readLinesshould be fine (especially usingn=argument and/or passing a connection rather than a file name to get incremental reads)... could you elaborate your use case / whyreadLineswon't suffice?For example, I want to go through a gz file, which is from https://ftp.ncbi.nlm.nih.gov/gene/DATA/gene2accession.gz and is more than 2gb. If I use
readLinesto read it from apipe('gzip -cd gene2accession.gz', 'r'), the bottleneck is R and the gzip process uses only less than 10% CPU of a thread. For comparison, for the same task, Perl uses ~ 80% CPU while the pigz uses about ~170% CPU.Reacted by Michael ChiricoOr, can
freadbe made to be able to process a raw vectorvec1containing file content read byreadBin( con1, 'raw', 1e6)?readBinis super fast, and one can quickly find out line separaters byvec1 == charToRaw( '\n'). The last line is probably incomplete, and this incomplete line can be cut and pasted to the head of the nextreadBinoutput.+1. for
fwritealso, would be great. cheers.
Submitted by: Chris Neff; Assigned to: Nobody; R-Forge link
I use a corporate internal networked file system for much of my data, and so often times i need to call read.csv with a file connection. fread doesn't support this yet.
Namely I would like the following to work:
f = file("~/path/to/file.csv")
dt = fread(f)