retain attributes while chaining

Question / Feature request.
Any tricky way to retain attributes when chaining? of course not all should be retained ("index","sorted","names", etc.).

library(data.table)
DT <- data.table(a = rep(1:5,2), b = rnorm(10))
setattr(DT,"my_custom_attr","attr_value")
str(DT)

# I would like to reuse attribute while chaining and also after chaining
DT[,attr(.SD,"my_custom_attr")] # NULL
attr(DT[,lapply(.SD,sum),by="a"],"my_custom_attr") # NULL

So the Feature request would be about retaining all user defined (non-DT related: "sorted","index", etc.) attributes.
Or maybe retain attrs defined by name, something like

options("datatable.retain.attr"=c("my_custom_attr","my_custom_attr2"))

This would allow to store custom metadata together with data.table, possibility reuse them while processing, or manipulate when using DT[,f(.SD)].

Have marked as FR for now.

new SO about that: https://stackoverflow.com/questions/33441632/retaining-metadata-when-subsetting-in-data-table
this SO highlights it is a gap comparing to data.frame

edit: another new one: https://stackoverflow.com/questions/34318299/user-specified-attributes-of-data-table-get-removed

Guy, is there are any problems with a solution like the following?

First, adding a function that will copy required attributes:

retainAttributes <- function(x, ans) {
    if ("retain.attributes" %in% names(attributes(x))) {
        for (a in c(attr(x, "retain.attributes"), "retain.attributes")) {
            attr(ans, a) <- attr(x, a)
        }
    }
    ans
}

And then using it is as a wrapper when neccessary in [.data.table:

diff --git a/R/data.table.R b/R/data.table.R
index d18dd3f..dda4985 100644
--- a/R/data.table.R
+++ b/R/data.table.R
@@ -1343,7 +1343,7 @@ chmatch2 <- function(x, table, nomatch=NA_integer_) {
             setattr(ans, "class", class(x)) # fix for #5296
             setattr(ans, "row.names", .set_row_names(nrow(ans)))

-            if (!with || missing(j)) return(alloc.col(ans))
+            if (!with || missing(j)) return(alloc.col(retainAttributes(x, ans)))

             SDenv$.SDall = ans
             SDenv$.SD = if (!length(othervars)) SDenv$.SDall else shallow(SDenv$.SDall, setdiff(ansvars, othervars))

With these, the following scenario works:

> t <- data.table(a=1:2, b=3:4)
> attr(t, "asd") <- "qwe"
> attr(t, "retain.attributes") <- "asd"
> attributes(t)
$names
[1] "a" "b"

$row.names
[1] 1 2

$class
[1] "data.table" "data.frame"

$.internal.selfref
<pointer: 0x2cb5018>

$asd
[1] "qwe"

$retain.attributes
[1] "asd"

> attributes(t[1, ])
$names
[1] "a" "b"

$class
[1] "data.table" "data.frame"

$row.names
[1] 1

$asd
[1] "qwe"

$retain.attributes
[1] "asd"

$.internal.selfref
<pointer: 0x2cb5018>

I can't provide a full patch, as [.data.table is long and full of returns... I'm not 100% sure which ones should be modified.

Interesting idea, but attr<- isn't best idea. There is in-memory copy there, we use setattr in such cases.

d1=data.table::data.table(a=1)
data.table::address(d1)
attr(d1, "asd") <- "qwe"
data.table::address(d1)

@jangorecki

@jangorecki Could you explain a little bit more? I have the same address printed there:

> d1=data.table::data.table(a=1)
> data.table::address(d1)
[1] "0x6b4f910"
> attr(d1, "asd") <- "qwe"
> data.table::address(d1)
[1] "0x6b4f910"

What's get copied there?

But it's just for my education, I have no problems with changing attr<- to setattr, or whatever else works best.

That is strange, when I was calling that before I get different address and now I'm getting the same. I must have made some wrong then.

want this feature very much, it's not available yet?

+1 for this features. Any concerns about solution above?

+1 for this feature also from my side.

I am not an expert with data.table or even with programming, therefore, I am currently using a clumsy function that stores custom attributes in a separate data.table called DT.attributes in the .GlobalEnv using eval(parse()) expressions, that makes accessing/updating these attributes similar to using attr. The code does not interfere with source code of data.table, hence, the object created is stable during chaining and can always be referenced/accessed by its name. The clear drawback is clearly that the functions messes around in .GlobalEnv, which is not a very good idea. Knowing that my solution is not clean and clumsy, I thought this might still be helpful to somebody and wanted to share this here. Please do not hang me for this attempt...

library(data.table)

attr_DT <- function(DT, j, value = NULL) {
  if (is.null(value)) {
    call_get_attr <- paste0(substitute(DT), ".attributes$", j, "[[1]]")
    return(eval(parse(text = call_get_attr)))
  } else {
    DT.attributes <- paste0(substitute(DT), ".attributes")
    call_make_attr <- paste0(DT.attributes, "[,", j,":= list(list(value))]")
    if (!exists(DT.attributes)) {
      #init
      assign(DT.attributes, data.table(init = integer(1L)),envir = .GlobalEnv)
      eval(parse(text =  call_make_attr))
      #delete init
      eval(parse(text = paste0(DT.attributes, "[,init:= NULL]")))
    } else {
      eval(parse(text =  call_make_attr))
    }
  }
}

myDT <- data.table(x = c(1:3), y = (letters[1:3]))
my_attr1 <- data.table(A1 = c(9:10), A2 = c(letters[9:10]))
my_attr2 <- "attribute2"

attr_DT(myDT, j = "name_myattr1", value = my_attr1)
attr_DT(myDT, j = "name_myattr2", value = my_attr2)
myDT.attributes
#    name_myattr1 name_myattr2
# 1: <data.table>   attribute2
attr_DT(myDT, j = "name_myattr1")
#    A1 A2
# 1:  9  i
# 2: 10  j

Navigation Menu

Repository navigation

retain attributes while chaining #995

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

retain attributes while chaining #995

Description

Activity

arunsrinivasan commented on Jan 3, 2015

jangorecki commented on Oct 30, 2015

assaron commented on Aug 13, 2016

jangorecki commented on Aug 13, 2016

assaron commented on Aug 13, 2016

assaron commented on Aug 13, 2016

jangorecki commented on Aug 13, 2016

changcw83 commented on Feb 21, 2017

dselivanov commented on May 17, 2017

manuelbickel commented on Mar 6, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions