Repository navigation
:= changes address of a data table #1729
Description
Activity
Are you aware of FAQ: 5.3 Reading
data.tablefrom RDS or RData file? I think it answers your question.Reacted by Arun SrinivasanThanks, I was not aware of
alloc.col, but I don't see how that answers my question. The FAQ item states that the DT is re-allocated on next by reference operation with a warning. I don't see the warning.That FAQ item indeed explains why the copy happens and why I don't see the change outside of a function call. I guess this issue then boils down to the missing warning.
Seems like the warning message was removed a long time ago.. but it needs to be there.
verbosehelps clear things up a bit:address_change <- function(df){ cat("df address before:", address(df), "\n") df[, c("new_var") := 1, verbose = TRUE] cat("df address after:", address(df), "\n") } tt <- readRDS("tt.txt") cat('tt address before:', address(tt), '\n') address_change(tt) cat('tt address after:', address(tt), '\n') "new_var" %in% names(tt)Has output:
tt address before: 0x7f61c6b5fc70 df address before: 0x7f61c6b5fc70 Detected that j uses these columns: c .internal.selfref ptr is NULL. This is expected and normal for a data.table loaded from disk. If not, please report to data.table issue tracker. Growing vector of column pointers from truelength 0 to 1025 . A shallow copy has been taken, see ?alloc.col. Only a potential issue if two variables point to the same data (we can't yet detect that well) and if not you can safely ignore this. To avoid this message you could alloc.col() first, deep copy first using copy(), wrap with suppressWarnings() or increase the 'datatable.alloccol' option. .internal.selfref ptr is NULL. This is expected and normal for a data.table loaded from disk. If not, please report to data.table issue tracker. Assigning to all 1000 rows RHS_list_of_columns == false df address after: 0x7f61cf6b5200 tt address after: 0x7f61c6b5fc70 [1] FALSE- A bit strange that ".internal.selfref ptr is NULL." is repeated
- We could include advice to
setDTafterreadRDSin this verbose message as well, as that would have helped:
tt <- readRDS("tt.txt") setDT(tt) cat('tt address before:', address(tt), '\n') address_change(tt) cat('tt address after:', address(tt), '\n') "new_var" %in% names(tt)has output
tt address before: 0x7f61cc964f40 df address before: 0x7f61cc964f40 Detected that j uses these columns: c Assigning to all 1000 rows RHS_list_of_columns == false df address after: 0x7f61cc964f40 tt address after: 0x7f61cc964f40 [1] TRUE- added a commit that references this issue
on May 5, 2019
I have narrowed down this to a specific dataset which I attach. It happens only when loaded from rds. If I read same data with
freadthe problem does not occur.I am disguising rds file with txt extension in order to be able to upload it to github.
I see consistent change of address on every trial:
Any ideas of how to get around this without a copy?
tt.txt