Repository navigation
retain attributes while chaining #995
Description
Activity
Have marked as FR for now.
Reacted by (Ant|T)h?on[iey] Hen{1,2}es{1,2}e?ynew SO about that: https://stackoverflow.com/questions/33441632/retaining-metadata-when-subsetting-in-data-table
this SO highlights it is a gap comparing to data.frameedit: another new one: https://stackoverflow.com/questions/34318299/user-specified-attributes-of-data-table-get-removed
Guy, is there are any problems with a solution like the following?
First, adding a function that will copy required attributes:
retainAttributes <- function(x, ans) { if ("retain.attributes" %in% names(attributes(x))) { for (a in c(attr(x, "retain.attributes"), "retain.attributes")) { attr(ans, a) <- attr(x, a) } } ans }
And then using it is as a wrapper when neccessary in
[.data.table:diff --git a/R/data.table.R b/R/data.table.R index d18dd3f..dda4985 100644 --- a/R/data.table.R +++ b/R/data.table.R @@ -1343,7 +1343,7 @@ chmatch2 <- function(x, table, nomatch=NA_integer_) { setattr(ans, "class", class(x)) # fix for #5296 setattr(ans, "row.names", .set_row_names(nrow(ans))) - if (!with || missing(j)) return(alloc.col(ans)) + if (!with || missing(j)) return(alloc.col(retainAttributes(x, ans))) SDenv$.SDall = ans SDenv$.SD = if (!length(othervars)) SDenv$.SDall else shallow(SDenv$.SDall, setdiff(ansvars, othervars))With these, the following scenario works:
> t <- data.table(a=1:2, b=3:4) > attr(t, "asd") <- "qwe" > attr(t, "retain.attributes") <- "asd" > attributes(t) $names [1] "a" "b" $row.names [1] 1 2 $class [1] "data.table" "data.frame" $.internal.selfref <pointer: 0x2cb5018> $asd [1] "qwe" $retain.attributes [1] "asd" > attributes(t[1, ]) $names [1] "a" "b" $class [1] "data.table" "data.frame" $row.names [1] 1 $asd [1] "qwe" $retain.attributes [1] "asd" $.internal.selfref <pointer: 0x2cb5018>I can't provide a full patch, as
[.data.tableis long and full of returns... I'm not 100% sure which ones should be modified.Interesting idea, but
attr<-isn't best idea. There is in-memory copy there, we usesetattrin such cases.d1=data.table::data.table(a=1) data.table::address(d1) attr(d1, "asd") <- "qwe" data.table::address(d1)
@jangorecki Could you explain a little bit more? I have the same address printed there:
> d1=data.table::data.table(a=1) > data.table::address(d1) [1] "0x6b4f910" > attr(d1, "asd") <- "qwe" > data.table::address(d1) [1] "0x6b4f910"What's get copied there?
But it's just for my education, I have no problems with changing
attr<-tosetattr, or whatever else works best.That is strange, when I was calling that before I get different address and now I'm getting the same. I must have made some wrong then.
want this feature very much, it's not available yet?
+1 for this features. Any concerns about solution above?
+1 for this feature also from my side.
I am not an expert with
data.tableor even with programming, therefore, I am currently using a clumsy function that stores custom attributes in a separatedata.tablecalledDT.attributesin the.GlobalEnvusingeval(parse())expressions, that makes accessing/updating these attributes similar to usingattr. The code does not interfere with source code ofdata.table, hence, the object created is stable during chaining and can always be referenced/accessed by its name. The clear drawback is clearly that the functions messes around in.GlobalEnv, which is not a very good idea. Knowing that my solution is not clean and clumsy, I thought this might still be helpful to somebody and wanted to share this here. Please do not hang me for this attempt...library(data.table) attr_DT <- function(DT, j, value = NULL) { if (is.null(value)) { call_get_attr <- paste0(substitute(DT), ".attributes$", j, "[[1]]") return(eval(parse(text = call_get_attr))) } else { DT.attributes <- paste0(substitute(DT), ".attributes") call_make_attr <- paste0(DT.attributes, "[,", j,":= list(list(value))]") if (!exists(DT.attributes)) { #init assign(DT.attributes, data.table(init = integer(1L)),envir = .GlobalEnv) eval(parse(text = call_make_attr)) #delete init eval(parse(text = paste0(DT.attributes, "[,init:= NULL]"))) } else { eval(parse(text = call_make_attr)) } } } myDT <- data.table(x = c(1:3), y = (letters[1:3])) my_attr1 <- data.table(A1 = c(9:10), A2 = c(letters[9:10])) my_attr2 <- "attribute2" attr_DT(myDT, j = "name_myattr1", value = my_attr1) attr_DT(myDT, j = "name_myattr2", value = my_attr2) myDT.attributes # name_myattr1 name_myattr2 # 1: <data.table> attribute2 attr_DT(myDT, j = "name_myattr1") # A1 A2 # 1: 9 i # 2: 10 j
Question / Feature request.
Any tricky way to retain attributes when chaining? of course not all should be retained ("index","sorted","names", etc.).
So the Feature request would be about retaining all user defined (non-DT related: "sorted","index", etc.) attributes.
Or maybe retain attrs defined by name, something like
This would allow to store custom metadata together with data.table, possibility reuse them while processing, or manipulate when using
DT[,f(.SD)].