[fread] broken functionality in 1.9.6

fread from v1.9.4 gives correct results but fread from dev 1.9.5 doesn't.
For eg

create a test.csv file

cat("V1, V2, V3
1,2,3
V4, V5, V6, V7
4,5,6,7
8,9,10,11",
    file = "test.csv"
)

Notice there are different number of columns
fread from 1.9.4

fread("test.csv", nrows = 1, header = TRUE, skip = 0)
V1 V2 V3
1:  1  2  3

This is what I want

But in fread from v1.9.5

     fread("test.csv", nrows = 1, header = TRUE, skip = 0)
    V4 V5 V6 V7
    1:  4  5  6  7
    Warning messages:
    1: In fread("test.csv", nrows = 1, header = TRUE, skip = 0) :
     Starting data input on line 3 and discarded previous non-empty line: 1,2,3
    2: In fread("test.csv", nrows = 1, header = TRUE, skip = 0) :
    Stopped reading at empty line 5 but text exists afterwards (discarded): 8,9,10,11

Not the expected result.

Minimum reproducible code is now added.

This bug has now been released in v1.9.6.

@ziyadsaeed

@ziyadsaeed ~~you can self-close resolved issue~~, OK

it isn't resolved. I'm mentioning that this bug was not fixed in the dev version and now the release version 1.9.6 has this bug.

The default behaviour is right as it is. I'd expect skip=0 explicitly to work as you intend however.

what is the workaround to ask fread to just read the first line.
When 1.9.6 was released it should be mentioned somewhere that it breaks existing functionality from 1.9.4 and a workaround should have been mentioned.

in 1.9.6 (and 1.9.7 to this point of time) nrows no longer forces fread to just read those rows.
this makes me frustrated on small files with only the last line having less columns.

fread("1,2,3\n1,2", nrows=1) wont work and there seems to be no work-around...
i'd like to achieve reading the entire file except for the unbalanced last line.

and this only happens in small files:

for ( i in 100:1) { 
  lines <- paste0(paste(rep("1,2,3", i), collapse='\n'), "\n1,2")
  fread(lines, nrows=i)
}

the above code stopped at i =4, which means the somewhat "undocumented" type checking behavior forces a parsing of at least 5 lines, regardless of the actual number of lines in the given file.

the impact of such feature is that whenever we are freading a bunch of files with a special treatment of last line (as we already know that the last line and only the last line may be unbalanced in column numbers but we still want to read it separately after freading the entire file except the last line), and there are files smaller than 6 lines with the last line unbalanced, we need a workaround. (i.e., to not use fread in these circumstances)

it will be appreciated that if we can optionally disable such type-checking behavior, or the argument nrows is actually not doing what it looks like to do: read only the first nrows lines. :)

Pencilled in tests for skip and nrows in #1267

Navigation Menu

Repository navigation

[fread] broken functionality in 1.9.6 #1267

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[fread] broken functionality in 1.9.6 #1267

Description

Activity

ziyadsaeed commented on Aug 14, 2015

ziyadsaeed commented on Sep 20, 2015

jangorecki commented on Sep 20, 2015

ziyadsaeed commented on Sep 20, 2015

arunsrinivasan commented on Sep 22, 2015

ziyadsaeed commented on Jan 9, 2016

everdark commented on Jun 5, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions