Skip to content

Support for writing to gzip (.gz) with fwrite #2016

Closed
#3288
Closed
Support for writing to gzip (.gz) with fwrite#2016
#3288
Labels
feature requestfwrite
Milestone
1.12.4

Description

@MichaelChirico
MichaelChirico
opened on Feb 5, 2017

write.csv can handle the following:

library(data.table)
DT = data.table(a = 1:10, b = 11:20)

tmp.gz = tempfile(fileext = '.csv.gz')
gzf = gzfile(tmp.gz, 'w')
write.csv(DT, gzf)
close(gzf)

But fwrite cannot:

tmp.gz = tempfile(fileext = '.csv.gz')
gzf = gzfile(tmp.gz, 'w')
fwrite(DT, gzf)
close(gzf)

Error: is.character(file) && length(file) == 1 && !is.na(file) is not TRUE

(because the gzf object is more complicated than just being a file path).

fwrite(DT, tmp.gz), as expected, does not perform compression -- the output is the same size as fwrite(DT, tempfile()) (which is different from write.csv).

I'm using fwrite to pipe some output to Vowpal Wabbit, trying to use the --compressed argument to save on some disk space. Workaround is to fwrite then gzip, but this seems sub-optimal.

Activity

  1. JhossePaul commented on Feb 28, 2017

    @JhossePaul
    JhossePaul
    on Feb 28, 2017

    +1

  2. rargelaguet commented on Mar 13, 2017

    @rargelaguet
    rargelaguet
    on Mar 13, 2017

    +1

  3. arunsrinivasan
    added
    feature request
    on Mar 30, 2017

  4. msgoussi commented on May 3, 2017

    @msgoussi
    msgoussi
    on May 3, 2017

    Dear Mr.
    When I export data.table using fwrite, the size of the file is 31.3GB
    then when I use gzip in order to compress the file, I get:
    Error in writeBin(bfr, con = out, size = 1L) :
    'Calloc' could not allocate memory (10000000 of 1 bytes)

    How can I solve this problem?
    Thanks

  5. st-pasha
    added
    fwrite
    on Apr 16, 2018

  6. Griefberg commented on May 31, 2018

    @Griefberg
    Griefberg
    on May 31, 2018

    +1

  7. wookietreiber commented on May 31, 2018

    @wookietreiber
    wookietreiber
    on May 31, 2018

    Please don't post comments with simply "+1". This adds absolutely no value. Either write something of value that adds something to the discussion or use the thumbs up emoticon / reaction on the original post to express interest in the issue and a solution to it.

    I'm in favor of deleting such comments, including mine BTW, because it is of no consequence to the technical discussion of this issue as writing "+1". Please, people, please ;)

  8. mattdowle
    added this to the 1.12.4 milestone on Apr 17, 2019
  9. mattdowle
    closed this as completedin #3288on Apr 23, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    feature requestfwrite

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions