Control-Z as EOF

Some CSV files generated on MS DOS/Windows, can have ^Z as the end-of-file character as eg at https://www.treasury.gov/ofac/downloads/sdn.csv which results in an error when calling fread:

Expected sep (',') but new line, EOF (or other non printing character) ends field 1 on line 6 when detecting types: ^Z

Removing that character from the end of the file resolves the problem.

Session info:

> sessionInfo()
R version 3.2.2 (2015-08-14)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 15.10

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.9.4

loaded via a namespace (and not attached):
[1] magrittr_1.5   plyr_1.8.3     tools_3.2.2    reshape2_1.4.1 Rcpp_0.12.3   
[6] stringi_1.0-1  stringr_1.0.0  chron_2.3-47

But I can reproduce this problem with the most recent dev version of data.table as well at 6f58f5c.

some awk or sed should be good workaround for now

Yeah, as said, "removing that character from the end of the file resolves the problem" :) But I though it's worth reporting as others might have the very same issue. Not high-priority for sure.

Strip final 0x1A character in the input file, if found. Closes #1612

It would be great if fread could remove it automatically.

Added news item for removed eod-of-file Ctrl-Z character. #1612 #2253

How can we install version 1.10.6?
I think
install.packages("data.table", type = "source", repos = "http://Rdatatable.github.io/data.table")
would install version 1.10.5

Version 1.10.6 hasn't been released yet. There is only 1.10.4 on CRAN, and the "dev" version (1.10.5) -- which is based on the master branch in this repo.

I said it because I read "mattdowle added this to the v1.10.6 milestone". I thought we could try it in the dev version in some way.

@skanskan When last number is odd, that's the dev release. v1.10.5 will be renamed v1.10.6 when it is released to CRAN. Otherwise we all get confused when we grab the dev at different times. It is only possible to obtain an even numbered version number from CRAN and is a guaranteed checkpoint.

Navigation Menu

Repository navigation

Control-Z as EOF #1612

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Control-Z as EOF #1612

Description

Activity

jangorecki commented on Mar 25, 2016

daroczig commented on Mar 25, 2016

juanpide commented on Jul 7, 2017

juanpide commented on Jul 8, 2017

st-pasha commented on Jul 9, 2017

juanpide commented on Jul 9, 2017

mattdowle commented on Jul 13, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions