Repository navigation
Add CSVY support for fread() #1701
Description
Activity
Just documenting how the
.csvyreader ofrioworks:rioencounters a.csvyfile and dispatches to.import.rio_csvy.import.rio_csvycallsread_csvyfrom thecsvypackage.read_csvyusesreadLinesto digest the file, then usesgrepto identify the YAML header.yaml.loadfrom theyamlpackage is called to convert the YAML content string to alistof component parts. This is already implemented in C so is presumably efficient.read_csvthen appliespaste(., collapse = '\n')to the non-YAML portion of the file and can read it withfread.- Content of YAML header is applied to the output.
Major inefficiencies are:
- Using
readLinesto digest the whole file (slow) pasteing the file back into a format forfreadto tackle only after stripping out the YAML part- Some parts of the YAML header are intended to assist
fread, but the YAML data is not fed intofread.
My proposed solution (if we decide to tackle this):
- Stream lines of the file until the end of the YAML header is reached (or until an out-of-format line is reached ->
stop). I'm not sure the most efficient way to pass files line-by-line in R, or if we'll have to implement that ourselves in C. Keep track of the # of lines fed. I see this from StackOverflow: https://stackoverflow.com/q/9871307/3576984 - Add
yamlpackage toSuggestsand rely on that to parse the YAML info - Extract any info relevant to
freaditself (especially/most importantlycolClasses; I'll have to read thecsvystandard to see how open-ended the rest is) freadthe remainder of the file, usingskipto jump past the YAML section, and including relevant info from the YAML.
Remaining API Q for me are: (1) do we try and detect
YAMLautomatically (harder & beyond me to implement currently), or simply add ayaml(or similarly named) argument tofreadand rely on user input? (2) What offwrite?For (1) I lean towards the latter primarily out of laziness (more sophisticated -- doesn't seem to pass the cost/benefit test given limited user requests for this feature). We can revisit in a future issue if this becomes more popular, I suppose. No opinions on (2), I only include it since we seem to be aiming to keep
freadandfwriteas each others' inverse functions.I'm not sure the most efficient way to pass files line-by-line in R
fread(file, sep = NULL)(in dev) ?Also
readLinesis likely to be much faster in R 3.5.0, possibly as fast asfreadorreadr::read_lines.@HughParsonage that still leaves the issue of reading the file twice. The idea of streaming lines is to examine the file line-by-line and only read in, say, 10-20 lines of YAML metadata using
readLines(orfread, or whatever) before parsing that and deployingfreadon (presumably) the bulk of the file which follows the header(anyway good to know they're finally getting around to improving
readLines, it's silly how slow it is considering how minimal its responsibilities are)Progress here:
I would avoid extra dependency, even in suggests, and use some helper function to extract fields from yaml header. Similar way as we would process DESCRIPTION file. Fwrite should be able to write csvy header the way that fread can read it.
as yaml can be arbitrarily nested I didn't see a need to reinvent the wheel.
especially as it seems at the moment to be a rather limited use case -- happy to revisit if this format takes off.
- added a commit that references this issue
on Feb 10, 2019 fwritesupport not done yet... will file separately- changed the title
[-]Add CSVY support for fread() and fwrite()[/-][+]Add CSVY support for fread()[/+]on May 3, 2019
The rio already allows reading of csvy formats by relying on the helper package (same author) csvy. It'd be great to leverage its features to read/write.