Skip to content

Numbers become char in fread/fwrite roundtrip #4802

Open
Open
Numbers become char in fread/fwrite roundtrip#4802
Labels
fread

Description

@renkun-ken
renkun-ken
opened on Nov 6, 2020

Consider the following tmp.csv:

writeLines("q01
-0.207623873704532
-1.4852856113776
-2432100838991580000
-2.33192914235436
-44.8581139833089", tmp <- tempfile())

There's a big number in it. We'll get a character column from this csv:

library(data.table)

dt <- fread(tmp)
dt
                    q01
                 <char>
1:   -0.207623873704532
2:     -1.4852856113776
3: -2432100838991580000
4:    -2.33192914235436
5:    -44.8581139833089

Coerce the column into numeric:

dt[, q01 := as.numeric(q01)]
dt
             q01
           <num>
1: -2.076239e-01
2: -1.485286e+00
3: -2.432101e+18
4: -2.331929e+00
5: -4.485811e+01

Then fwrite it to another csv and re-fread it:

fwrite(dt, tmp2 <- tempfile())

dt2 <- fread(tmp2)
dt2

We still get character column.

                    q01
                 <char>
1:   -0.207623873704532
2:     -1.4852856113776
3: -2432100838991580000
4:    -2.33192914235436
5:    -44.8581139833089

I'm not sure if it is intended behavior. It makes sense if we want to preserve all information but such behavior could be surprising if user assumes that numbers written by fwrite() could be read into numbers by fread(). In this example, it is not the case. Even if we specify colClasses = "numeric", it won't work either:

> dt2 <- fread(tmp2, colClasses = "numeric")
Warning message:
In fread(tmp2, colClasses = "numeric") :
  Attempt to override column 1 <<q01>> of inherent type 'string' down to 'float64' ignored. Only overrides to a higher type are currently supported. If this was intended, please coerce to the lower type afterwards.

> dt2
                    q01
                 <char>
1:   -0.207623873704532
2:     -1.4852856113776
3: -2432100838991580000
4:    -2.33192914235436
5:    -44.8581139833089

Activity

  1. jangorecki
    added
    fread
    on Nov 6, 2020

  2. MichaelChirico commented on Nov 9, 2020

    @MichaelChirico
    MichaelChirico
    on Nov 9, 2020
    Member

    Note that we don't have enough precision in numeric for your big number:

    sprintf("%.0f", -2432100838991580000)
# [1] "-2432100838991580160"

  3. jangorecki commented on Nov 9, 2020

    @jangorecki
    jangorecki
    on Nov 9, 2020
    Member

    I think it make sense to close then. We see that fread takes care more about retaining original value which is good, and following advice from warning seems pretty good way to address what user needs (loss of precision).

  4. renkun-ken commented on Nov 9, 2020

    @renkun-ken
    renkun-ken
    on Nov 9, 2020
    MemberAuthor

    I'm wondering if there's a way to tell fread that I don't care about retaining original value in this case? The ideal way I can imagine is that when I fread the file, it shows up some warning why the column is character and what I can do to force it into numeric with some loss of precision.

  5. renkun-ken commented on Nov 9, 2020

    @renkun-ken
    renkun-ken
    on Nov 9, 2020
    MemberAuthor

    Sorry, just a warning is enough. I could always as.numeric() to coerce it to numeric with loss of precision.

  6. renkun-ken commented on Nov 9, 2020

    @renkun-ken
    renkun-ken
    on Nov 9, 2020
    MemberAuthor

    But if fread() supports this, we could avoid unnecessary allocation of character vectors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    fread

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions