Skip to content

Crash in fread in dev when under-estimating the number of rows #2246

Closed
Closed
Crash in fread in dev when under-estimating the number of rows#2246
Labels
devfreadsegfault
Milestone
v1.11.0

Description

@st-pasha
st-pasha
opened on Jul 5, 2017
require(data.table)
DT1 = data.table(A=rep(123, 100), B=456)
DT2 = data.table(A=rep(1, 200), B=2)
DT = rbind(DT1, DT2, DT1, DT2, DT1, DT2, DT1, DT2, DT1, DT2, DT1, DT2, DT1, DT2, DT1, DT2, DT1, DT2, DT1, DT1)
fwrite(DT, f<-tempfile())
ff = fread(f)

Normally this produces

Error in fread(f) : 
  Internal error: DTi(2900) > allocnrow(2208) but nrows=9223372036854775807 (not limited)

but occasionally also

R(22039,0x7fff7504e000) malloc: *** error for object 0x7f957ec29608: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

Activity

  1. st-pasha
    added
    fread
    segfault
    on Jul 5, 2017
  2. mattdowle
    changed the title [-]Possible crash in fread when mis-estimating the number of rows in a file[/-] [+]Crash in fread in dev when under-estimating the number of rows[/+] on Jul 5, 2017

  3. mattdowle commented on Jul 5, 2017

    @mattdowle
    mattdowle
    on Jul 5, 2017
    Member

    Great find Pasha!

  4. mattdowle
    added this to the milestone on Jul 5, 2017
  5. st-pasha
    added
    dev
    on Jul 21, 2017

  6. st-pasha commented on Jul 21, 2017

    @st-pasha
    st-pasha
    on Jul 21, 2017
    ContributorAuthor

    Solution outline:

    1. Introduce function DTrealloc(int64_t nrows) (similar to DTalloc()). The new function's job will be to re-alloc every column in the datatable to the new length, however preserving all the data read so far.
    2. Set the maxNrows parameter in freadMain to be the minimum of args.maxNrows and allocnrows.
    3. If after the main loop we find that allocnrows rows were read, but more exist afterwards -- then guess how many rows are remaining and call the DTrealloc() function. After that, go back to the top of the data reading loop and continue reading the data from where we ended.
  7. mattdowle
    closed this as completedin #2355on Sep 14, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    devfreadsegfault

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions