Repository navigation
Bug interacting with data.table embedded in list created by another package #1310
Description
As mentioned and ellaborated in this SO post, I've run into what appears to be a bug trying to get maptools/sp to cooperate with data.table.
Copy-pasting the example from the post (data here):
library(maptools)
us.states<-readShapePoly("cb_2014_us_state_5m.shp")
setDT(us.states@data) #works fine
> class(us.states@data)
[1] "data.table" "data.frame"
us.states@data[,test:=1L]
Warning produced from this line:
Warning message: In
[.data.table(us.states@data, ,:=(test, 1L)) : Invalid .internal.selfref detected and fixed by taking a (shallow) copy of the data.table so that := can add this new column by reference. At an earlier point, this data.table has been copied by R (or been created manually using structure() or similar). Avoid key<-, names<- and attr<- which in R currently (and oddly) may copy the whole data.table. Use set* syntax instead to avoid copying: ?set, ?setnames and ?setattr. Also, in R<=v3.0.2, list(DT1,DT2) copied the entire DT1 and DT2 (R's list() used to copy named objects); please upgrade to R>v3.0.2 if that is biting. If this message doesn't help, please report to datatable-help so the root cause can be fixed.
Further, no update was made:
> names(us.states@data)
[1] "STATEFP" "STATENS" "AFFGEOID" "GEOID" "STUSPS" "NAME"
"LSAD" "ALAND" "AWATER"
The problem appears to be similar to the general .internal.selfref problem alluded to in several SO answers--I'll choose this one as canon--namely, that the data for a SPDF is stored as an element of a list.
Not clear how to get around this without creating a copy.
Activity
franknarf1 commentedon Aug 22, 2016
I'm also running into this, without the involvement of other packages. I'm using
split.data.table; working with each subset; and then rbinding back again. Or at least that was the initial plan.mbacou commentedon Feb 21, 2017
Indeed, there's currently no way to work with sp object attributes without creating copies. My usual (wasteful) workflow is along the lines of:
sfimplementation seems to have solved the copy problem, but as I understandsfclass is not, as of yet, fully supported by data.table.MichaelChirico commentedon Feb 21, 2017
@mbacou yea that's basically how my workflow has settled to this point as well (I usually try and be explicit and use
DT = copy(sp@data)).I don't know what
sfis?mbacou commentedon Feb 21, 2017
Here is Pebesma's
sfreplacement library forsphttps://edzer.github.io/sfr/index.html.