Skip to content

[Request] With fread, do not throw 32 bit large file support error if nrows is specified #949

Closed
Closed
[Request] With fread, do not throw 32 bit large file support error if nrows is specified#949
Labels
feature requestfread

Description

@xiaodaigh
xiaodaigh
opened on Nov 13, 2014

For the fread function I wonder if it's possible to not show the 32-bit large file support error if I actually specified the nrows? I am only interested in the first 5000 rows, say, and I need not read in the whole file on a 32bit Windows machine.

The error message:

#  "Error in data.table::fread("c:/testing/data.csv",  : 
#   Opened file ok, obtained its size on disk (1631.0MB), but couldn't memory map it. This is a 32bit 
# machine. You don't need more RAM per se but this fread function is tuned for 64bit addressability,
# at the expense of large file support on 32bit machines. You probably need more RAM to store the 
# resulting data.table, anyway. And most speed benefits of data.table are on 64bit with large RAM, 
# too. Please either upgrade to 64bit (e.g. a 64bit netbook with 4GB RAM can cost just £300), or 
# make a case for 32bit large file support to datatable-help."

Activity

  1. xiaodaigh
    changed the title [-]Do not though 32 bit large file support error if nrows is specified[/-] [+][Request] With fread, do not throw 32 bit large file support error if nrows is specified[/+] on Nov 13, 2014
  2. arunsrinivasan
    added
    feature request
    on Dec 4, 2014
  3. arunsrinivasan
    added
    fread
    on Sep 4, 2015

  4. AtroXWorf commented on Dec 15, 2016

    @AtroXWorf
    AtroXWorf
    on Dec 15, 2016

    I'd also like this to be implemented. It should be handy for 'nrows=0' situations or when one ins only interested to read in small portions via nrows.

  5. mattdowle commented on Mar 3, 2018

    @mattdowle
    mattdowle
    on Mar 3, 2018
    Member

    Please retry with latest dev 1.10.5. From NEWS :

    Memory maps lazily; e.g. reading just the first 10 rows with nrow=10 is 12s down to 0.01s from cold for the 9GB file. Large files close to your RAM limit may work more reliably too. The progress meter will commence sooner and more consistently.

    There's a chance it will work now. If not, please file a new issue.

    For Windows you can download a binary .zip for dev 1.10.5 by following instructions here.

  6. mattdowle
    closed this as completedon Mar 3, 2018

  7. st-pasha commented on Mar 4, 2018

    @st-pasha
    st-pasha
    on Mar 4, 2018
    Contributor

    On a 32-bit system it's not possible to memory-map a file bigger than 4GB in its entirety. It's possible to memory-map only part of the file though. Trying to map the whole file will raise an error

    EOVERFLOW
    On 32-bit architecture together with the large file extension
    (i.e., using 64-bit off_t): the number of pages used for
    length plus number of pages used for offset would overflow
    unsigned long (32 bits).

    Of course we can map only the first 4GB of the file, and then hope that that would be sufficient to read the amount of rows requested by the user. But what if not? It is possible to work around this by carefully managing the window of the file which is currently mapped; however it would increase the code complexity significantly. In addition, it is unclear how to test this functionality because finding a 32-bit machine is challenging nowadays.

  8. AtroXWorf commented on Mar 4, 2018

    @AtroXWorf
    AtroXWorf
    on Mar 4, 2018

    I no longer have access to a 32-bit machine, so I can not test it anymore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    feature requestfread

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions