Skip to content

Out-of-sample quote rule bump with warning, #2265 - #2643

Merged
mattdowle merged 7 commits into
masterfrom
qrbump
Feb 28, 2018
Merged

mattdowle merged 7 commits into
masterfrom
qrbump

Conversation

@mattdowle

@mattdowle mattdowle commented Feb 24, 2018 •

Copy link
Copy Markdown
Member

Closes #2265
test() no longer relies on option(warn=2). Multiple warnings() are now tested, as needed for the new test for this issue.
\n in output= is now left in rather than stripped out. Or output= can be a vector of lines, too.

@mattdowle mattdowle added this to the v1.10.6 milestone Feb 24, 2018
@codecov-io

codecov-io commented Feb 24, 2018 •

Copy link
Copy Markdown

Codecov Report

Merging #2643 into master will increase coverage by 0.02%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2643      +/-   ##
==========================================
+ Coverage   93.14%   93.16%   +0.02%     
==========================================
  Files          61       61              
  Lines       12130    12166      +36     
==========================================
+ Hits        11298    11335      +37     
+ Misses        832      831       -1
Impacted Files Coverage Δ
src/fwrite.c 91.44% <100%> (ø) ⬆️
R/test.data.table.R 100% <100%> (+0.89%) ⬆️
src/freadR.c 89.43% <100%> (-0.21%) ⬇️
src/fread.c 97.95% <100%> (+0.01%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b392bfe...8b3f338. Read the comment docs.

@mattdowle
mattdowle requested a review from st-pasha February 24, 2018 02:23

@st-pasha st-pasha left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting... So basically on any kind of error you're saying "What if I tried with a different kind of quoting rule instead, would I be able to parse then?" Then usually you won't, but on rare occasions it does work, which means the initial guess of the QR was incorrect and it was fixed. And since we don't care that much about speed for invalid inputs, there's virtually no cost to do this extra check. Great!

I guess the only remaining inconsistency is that half of the input was parsed under one QR, while the other half under a different QR. Say, if first half of the file had QR=1 (doubled quotes), while the second half had QR=2 (escaped quotes), then it won't be flagged as an error. Admittedly, we never saw that happening IRL, so maybe nothing to worry about...

@st-pasha

Copy link
Copy Markdown
Contributor

👍 But I like how you expressed this in terms of just another re-read pass -- I thought it would be much harder than that...

@mattdowle

Copy link
Copy Markdown
Member Author

Yes exactly. If a quote rule type bump occurs there is a warning on that line always, so at least the user knows something might be improper in the file. Some refinement needed for sure but it's erring on the side of caution for now (always warns if any quote rule bump occurred). Hm. That's not true if there's a one-row footer auto removed too, currently. Will improve that ...

@mattdowle
mattdowle merged commit 807f254 into master Feb 28, 2018
@mattdowle
mattdowle deleted the qrbump branch February 28, 2018 09:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants