Repository navigation
Support for writing to gzip (.gz) with fwrite #2016
Description
write.csv can handle the following:
library(data.table)
DT = data.table(a = 1:10, b = 11:20)
tmp.gz = tempfile(fileext = '.csv.gz')
gzf = gzfile(tmp.gz, 'w')
write.csv(DT, gzf)
close(gzf)
But fwrite cannot:
tmp.gz = tempfile(fileext = '.csv.gz')
gzf = gzfile(tmp.gz, 'w')
fwrite(DT, gzf)
close(gzf)
Error:
is.character(file) && length(file) == 1 && !is.na(file)is notTRUE
(because the gzf object is more complicated than just being a file path).
fwrite(DT, tmp.gz), as expected, does not perform compression -- the output is the same size as fwrite(DT, tempfile()) (which is different from write.csv).
I'm using fwrite to pipe some output to Vowpal Wabbit, trying to use the --compressed argument to save on some disk space. Workaround is to fwrite then gzip, but this seems sub-optimal.
Activity
JhossePaul commentedon Feb 28, 2017
+1
rargelaguet commentedon Mar 13, 2017
+1
msgoussi commentedon May 3, 2017
Dear Mr.
When I export data.table using fwrite, the size of the file is 31.3GB
then when I use gzip in order to compress the file, I get:
Error in writeBin(bfr, con = out, size = 1L) :
'Calloc' could not allocate memory (10000000 of 1 bytes)
How can I solve this problem?
Thanks
Griefberg commentedon May 31, 2018
+1
wookietreiber commentedon May 31, 2018
Please don't post comments with simply "+1". This adds absolutely no value. Either write something of value that adds something to the discussion or use the thumbs up emoticon / reaction on the original post to express interest in the issue and a solution to it.
I'm in favor of deleting such comments, including mine BTW, because it is of no consequence to the technical discussion of this issue as writing "+1". Please, people, please ;)