fread/fwrite *base data types* directly for efficiency #1656
Description
Activity
- changed the title
[-]fread could read *base data types* directly for efficiency[/-][+]fread/fwrite *base data types* directly for efficiency[/+]on Apr 20, 2016 It would be awesome if
fwriteby default wroteDateandPOSIXctobjects in standard formats (ie ISO 8601, orLubridate) thatfreadwould automatically recognize and read in asDateorPOSIXct, without requiringcolClassesto be specified for each datetime column. A default automatic approach would be really welcome since I spend a lot of programming time converting datetime character strings back to POSIXct when importing time series data, even when I have written the file myself withwrite.csv. It would presumably also require an assumption (default value) fortzon import, unless the default format included the time zone offset on every value. Perhaps aPOSIXct.format="format-string"parameter could be included to specify the format string to use when writing and readingPOSIXctvalues, sincecolClasses = c(a="Date", b="POSIXct")doesn't allow a specific format and/or timezone to be specified.@clarkdk You discuss quite a different issue, as it requires parsing strings that stores date/datetime. This issue is about date/datetime stored as numerics, and those can be effectively optimized for IO speed if fwrite and fread would be able to write and read those types as their numerics transparently (or using just
colClasses). So no parsing is required. If you would like to speedup parsing strings then fasttime can help.@jangorecki OK, now I understand. I will make a separate FR. In implementing datetimes as numerics in
fread/fwrite, please don't rule out a possible future feature involving writing and parsing of datetimes as formatted strings.- added a commit that references this issue
on Nov 11, 2016 - added this to the milestone
on Nov 11, 2016 - modified the milestones: ,
on Mar 17, 2018 2 remaining items
Not sure if this belongs to this issue or to 3391 or to the master task 2247, but I could reproduce an unexpected behavior with
freadandfwritewhenyaml = TRUE:dt = data.table(date = as.POSIXct(c("2006-05-01", "2006-05-02")), b = as.factor(c(1,2)), c = c(3,4)) print(class(dt$date)) fwrite(dt, "dt.csv", yaml = TRUE) dt2 = fread("dt.csv", yaml = TRUE) print(class(dt2$date))See: https://stackoverflow.com/questions/58493926/fread-with-yaml-true-read-type-date-as-character
@PavoDive could you file a new issue for that? please & thank you
There could be an option for
fwriteandfreadto dump/read the unclassed representation of objects. In such a case whole IO process can be much faster, because we don't have to formatPOSIXct,[I]Dateand other classes for writing to file, we justunclassthem, and write numbers (or any other base type they use) to a file. Later when reading withfreadwe providecolClasses(oryaml), and those numbers (or another base types) are properly recognized as their original class.