in fread integer64 argument is ignored if int64 type-bump occurs out-of-sample

dt0 = data.table(a=seq(10000), b="100")
dt0[111, b := "1000000000000"]
f = tempfile()
fwrite(dt0, f)

fread(f, integer64="numeric")
           a     b
       <int> <i64>
    1:     1   100
    2:     2   100
    3:     3   100
    4:     4   100
   ---            
 9996:  9996   100
 9997:  9997   100
 9998:  9998   100
 9999:  9999   100
10000: 10000   100

fread(f, integer64="character")
           a     b
       <int> <i64>
    1:     1   100
    2:     2   100
    3:     3   100
    4:     4   100
   ---            
 9996:  9996   100
 9997:  9997   100
 9998:  9998   100
 9999:  9999   100
10000: 10000   100

Just came across this in fixing #3683. See test 899.1 with integer64 = 'character'

The workaround #2607 (comment) could be implemented quite easily in https://github.com/Rdatatable/data.table/blob/master/R/fread.R, as long as the bug isn't fixed (possible fix described here: #2607 (comment)). If not, at least the documentation of fread https://github.com/Rdatatable/data.table/blob/master/man/fread.Rd should be adjusted and mention it. Should I send a pull request?

@petres

@petres yes please, also please link this issue. Eventually if issue gets resolved and someone will forget to update documentation then status of the issue can be obtained after following link.

hi. sorry, but does "type-bump occurs out-of-sample" mean, effectively, that the column looked, in the first however-many-rows, like "int", but then, later, oops, (2**32)+1000 showed up? (i'm getting integer64's on a specific column in a file, in spite of integer64 = "numeric" in my fread call.)

almost, not quite. the sample is not the first n rows, but rather 100 equally-spaced blocks of 100 rows. so the non-int row happens to fall outside those regions.

PS you may want to specify colClasses to avoid the type bump warning.

@MichaelChirico

thanks, @MichaelChirico. yes, i already specify colClasses. (my code was failing later, i guess tripping over the integer64 column, not having bit64.)

This issue comes from the use of the typebump label instead of userOverride function.

The object that holds the replacement type for i64 is readInt64As (declared as static) is inaccessible from within the type bump code path.

To clarify, the integer64 type remapping occurs in the userOverride function, which only gets called once, and doesn't get recalled during a type bump.

In addition, the readInt64As variable isn't accessible from fread.c, which is where the type bump system is implemented.

fixed #2749

Respect integer64= selection when out-of-sample type bumps happen in …

Navigation Menu

Repository navigation

in fread integer64 argument is ignored if int64 type-bump occurs out-of-sample #2749

11 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

in fread integer64 argument is ignored if int64 type-bump occurs out-of-sample #2749

Description

Activity

MichaelChirico commented on Jul 7, 2019

11 remaining items

petres commented on Dec 18, 2020

jangorecki commented on Dec 19, 2020

greg-minshall commented on Jul 31, 2021

MichaelChirico commented on Jul 31, 2021

greg-minshall commented on Aug 1, 2021

badasahog commented on Apr 10, 2025

badasahog commented on Apr 28, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions