Setting data.table column outputs whole table to screen

If I run the code below, the second line will cause the entire dat object to be output, whereas at an R console it wouldn't return anything. The syntax used is the special syntax for setting columns with the popular data.table package. I'm using the 1.9.5 devel version of data.table.

library(data.table)
dat <- data.table(x1=1:10)
dat[, x2 := 1:10]

withVisible(d[, x2:=1:10]) tells me that the result should be visible. I thought that was how R determined whether or not to print it. Clearly there's something else going on.

Just a note if testing this: the same behavior you see in the notebook is also cropping up at the console in R 3.2.0, as described at Rdatatable/data.table#1122. So if trying to fix you will likely want to use R < 3.2.0 until they get this bug fixed.

Fixing this on the IRkernel side is probably not reasonable. Under the hood, when you run a statement with the data.table := syntax, it sets a global variable equal to address(x), which is then checked inside of the print.data.table command to see if the output should be suppressed. In other words, when you run the command

dat[, x2 := 1:10]

It is triggering [.data.table followed by print.data.table, where print.data.table sees that it should not print this time, after which it resets the state of the global variable so that other print statements will run. However, when IRkernel is running evaluate(), the print.data.table statement never gets called, so all you see is the regular data.table object.

Ah, R ;-)

Unfortunately because [ is a R primitive function, it cannot return an invisible object, see here. This is why the developers of data.table have resorted to the workaround of using an internal global variable to alter the behavior of print.data.table.

A fix on the IRkernel side is to check of the length of data[['text/plain']] inside the execution.R/handle_value() function. If the length is zero, nchar(data[['text/plain']]==0, then don't send back a response because if you were running this at the console then you would be getting nothing, and in general we would expect the printing of text objects to mimic the console in the notebook. This still allows the notebook to work properly with functions that return a blank string or NULL, which will print properly.

On a related note, it would be desirable to have a repr option that controls the maximum number of rows to print for a generic data.frame or matrix, similar to what RStudio does. By default this should be set to something not too large. Otherwise the user who accidentally prints their 10 million element matrix to screen ends up waiting a long time for the HTML to be built and displayed.

@flying-sheep

@flying-sheep : the idea of checking whether there's any text output, and suppressing all output if there isn't, sounds basically reasonable to me. Do you see any problems with that?

Generally yes, but the question is what that means.

Will print per default do something unless you override it?

Because if it is overridden to not output anything, then it will usually do something unrelated, right? Like plotting or something.

In general I would say an overridden print statement would not do anything else, though to be frank this use in data.table is the only place I've seen it. If the intention was to print graphics, the more canonical form would be to have a plot function, so that the user is typing plot(myobj).

But regardless, if a graphics command is called inside the print statement, the graphics callback used with the output handler in evaluate will still pick it up, allowing you to send back the plot even if there is no console output. For example,

library(evaluate)

mat <- function(x) {
  class(x) <- "mat"
  x
}

print.mat <- function(x, ...) {
  plot(rnorm(10))
  return(invisible())
}

oh <- new_output_handler(
  value = function(obj) {
    print("VALUE")
    val <- capture.output(print(obj))
    # Check the length of val to see if we should send text output back
  },

  graphics = function(plotobj) {
    print("GRAPHICS") # This will always get run
  }
)

m1 <- mat(matrix(1:4, 2, 2))
evaluate("m1", output_handler = oh)

true. as said: i think we should do it. this was just a side thought :)

@abielr

OK, @abielr, do you want to make a pull request?

Very much appreciate the kind language in this thread. Yes all correct.

Have just fixed Rdatatable/data.table#1122. About to release v1.9.6 to CRAN.

Note new wording of bug fixes in https://github.com/Rdatatable/data.table/blob/master/README.md :

if (TRUE) DT[,LHS:=RHS] no longer prints, #869 and #1122. Tests added. To get this to work we've had to live with one downside: if a := is used inside a function with no DT[] before the end of the function, then the next time DT or print(DT) is typed at the prompt, nothing will be printed. A repeated DT or print(DT) will print. To avoid this: include a DT[] after the last := in your function. If that is not possible (e.g., it's not a function you can change) then DT[] at the prompt is guaranteed to print. As before, adding an extra [] on the end of a := query is a recommended idiom to update and then print; e.g. > DT[,foo:=3L][]. Thanks to Jureiss and Jan Gorecki for reporting.

DT[FALSE,LHS:=RHS] no longer prints either, #887. Thanks to Jureiss for reporting.

:= no longer prints in knitr for consistency with behaviour at the prompt, #505. Output of a test knit("knitr.Rmd") is now in data.table's unit tests. Thanks to Corone for the illustrated report.

We had to add a workaround in data.table for knitr. Obviously ugly and not ideal. But := by reference is so fundamental in the DT[where, select|update|do, by] general form, that it was worth this hassle, so far. Could add a similar workaround for IRkernel too if that helps - let me know.

This seems to be the same or a related issue I am seeing. I found a simple reproducible example to show it.

The following data.table creation and := works correctly (though it does output to the screen).

DT1 = data.table(x=rep(c("a","b","c", "d"),each=15), 
                 y=c(1,3,NA,9), 
                 v=c(1:6,NA,NA,NA,NA,NA,NA), 
                 z=1:12)
DT1[,min:=pmin(y, v, na.rm=TRUE)]

When I make the data.table a bit larger by increasing each, I get a warning/error

DT2 = data.table(x=rep(c("a","b","c", "d"),each=18), 
                 y=c(1,3,NA,9), 
                 v=c(1:6,NA,NA,NA,NA,NA,NA), 
                 z=1:12)
DT2[,min:=pmin(y, v, na.rm=TRUE)]

Error in rbindlist(l, use.names, fill, idcol): Item 2 of list input is not a data.frame, data.table or list

If I separate the commands, the warning is from the line DT2[,min:=pmin(y, v, na.rm=TRUE)] However, the resulting DT2 prints to screen, and it is modified correctly with column min added. But even this is a bit strange.

print(DT2)

Gives no error, and outputs all 72 rows and 4 columns, just like in RStudio.

DT2

Gives the same error

Error in rbindlist(l, use.names, fill, idcol): Item 2 of list input is not a data.frame, data.table or list

but then outputs the same 72x4 data.table.

In my actual script, which has a much larger data.table, where I am doing several new columns with :=, this causes multiple of these errors to be reported, and the data.table to be output, but the end result seems to be the same as what I get in RStudio with no errors. The assignment seems to work. It does seem to run MUCH slower than in RStudio. I'm not sure if this is because it's outputing the table at each step, or if it's not assigning by reference in place.

Session info with versions is below:

sessionInfo()

R version 3.2.2 (2015-08-14)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux Workstation release 6.7 (Santiago)

locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] data.table_1.9.6

loaded via a namespace (and not attached):
[1] magrittr_1.5 IRdisplay_0.3 tools_3.2.2 base64enc_0.1-3
[5] uuid_0.1-2 stringi_1.0-1 rzmq_0.7.7 IRkernel_0.5
[9] jsonlite_0.9.17 stringr_1.0.0 digest_0.6.8 chron_2.3-47
[13] repr_0.4 evaluate_0.8

jupyter --version
4.0.6

python --version
Python 2.7.10 :: Anaconda 2.4.0 (64-bit)

I would guess that when you see the error followed by the table, the error comes from the code in repr that attempts to generate an HTML version of that table. That's failing, so it falls back to showing the plain text table.

You can check this by doing:

repr::repr_html(DT2)

You're right, repr::repr_html(DT2) gives the same error, without showing the table. Perhaps the error is because repr_html() has a size limit for the resulting table?

I also get the same error with a much taller table, perhaps in this case it's the --- causing the issue:

      x  y  v  z min
   1: a  1  1  1   1
   2: a  3  2  2   2
   3: a NA  3  3   3
   4: a  9  4  4   4
   5: a  1  5  5   1
  ---               
7196: d  9 NA  8   9
7197: d  1 NA  9   1
7198: d  3 NA 10   3
7199: d NA NA 11  NA
7200: d  9 NA 12   9

I just added mimicsAutoPrint to the callstack hack:
Rdatatable/data.table@689b624
If you could fetch that, add your function name using assignInNamespace() and test please. Then if it works let me know the function name I should add: repr_print.default ?

I also added and exported shouldPrint() to expose the flag (Rdatatable/data.table@3ec2d61). It resets the flag within it so it is a read-once function. If you need it twice in your logic, store the value from the first call.

Does this resolve everything? Should we continue to keep IRkernel on the whitelist or remove it? Maybe we can remove knitr from the whitelist too since the comment there is "knitr's eval is passed envir=globalenv() so doesn't need to be listed here currently, but we include it in case it decides to change that."

I updated to current dev versions of data.table and IRkernel and ran the following described in #343:

assignInNamespace("cedta.pkgEvalsUserCode", NULL, "data.table")
repr_html.data.table <- function(obj, ...){
    if (data.table:::.global$print != "" && address(obj) == data.table:::.global$print) {
        NULL
    } else {
        NextMethod()
    }
}
repr_latex.data.table <- function(obj, ...){
    if (data.table:::.global$print != "" && address(obj) == data.table:::.global$print) {
        NULL
    } else {
        NextMethod()
    }
}
repr_text.data.table <- function(obj, ...){
    if (data.table:::.global$print != "" && address(obj) == data.table:::.global$print) {
        NULL
    } else {
        NextMethod()
    }
}

Now assignment by reference no longer displays output:

dat <- data.table(x1=1:10)
dat[, x2 := 1:10]

but calling the object no longer prints (a summary of) the data.table--i.e., this does nothing:

dat

Personally, I'm fine with that, though I suspect others will dislike this behavior.

sessionInfo():

R version 3.2.2 (2015-08-14)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 15.10

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.9.7

loaded via a namespace (and not attached):
 [1] R6_2.1.2        magrittr_1.5    IRdisplay_0.3   pbdZMQ_0.2-3   
 [5] tools_3.2.2     base64enc_0.1-3 uuid_0.1-2      stringi_1.0-1  
 [9] IRkernel_0.6    jsonlite_0.9.20 stringr_1.0.0   digest_0.6.9   
[13] repr_0.7        evaluate_0.9

@flying-sheep

Ok, I tried this:

library(data.table)
old = data.table:::mimicsAutoPrint
old
reprs = c("repr_text.data.table", "repr_latex.data.table", "repr_markdown.data.table", "repr_html.data.table", "repr_text.default")
assignInNamespace("mimicsAutoPrint", c(old, reprs), "data.table")
dat <- data.table(x1=1:10)
dat[, x2 := 1:10]

and it still prints because we do have a repr_text.data.frame which is used and prints the dat because it is also a data.frame.

I also tried to remove the repr_text.data.frame method, but then repr ran into this callstack:

[...]
[[20]]
withCallingHandlers(withVisible(value_fun(ev$value, ev$visible)), 
    warning = wHandler, error = eHandler, message = mHandler)

[[21]]
withVisible(value_fun(ev$value, ev$visible))

[[22]]
value_fun(ev$value, ev$visible)

[[23]]
value_handler(x)

[[24]]
prepare_mimebundle(obj, .self$handle_display_error)

[[25]]
repr_text(obj)

[[26]]
repr_text.data.frame(obj)

[[27]]
NextMethod()

[[28]]
repr_text.default(obj)

[[29]]
paste(utils::capture.output(print(obj)), collapse = "\n")

[[30]]
utils::capture.output(print(obj))

[[31]]
evalVis(expr)

[[32]]
withVisible(eval(expr, pf))

[[33]]
eval(expr, pf)

[[34]]
eval(expr, envir, enclos)

[[35]]
print(obj)

[[36]]
print.data.table(obj)

which seems not to match ( length(SYS) > 3L && as.character(SYS[[length(SYS)-3L]][[1L]]) %chin% mimicsAutoPrint )

So Rdatatable/data.table@689b624 seems to be not working here because the callstack is just too different than the one from knit_print :-(

So we do have to go the repr_text.data.table with shouldPrint() way.

repr_text.data.table <- function(obj, ...){
    if (!data.table::shouldPrint(obj)) {
        invisible(NULL) # in IRkernel, will prevent any other repr_xx methods from being called
    } else {
        NextMethod() # fallsback to `repr_text.default`, which uses print(obj)
    }
}
# No need for repr_html/... as the only reason we have the above method 
# is to return null, which indicates to the IRkernel that nothing else should be printed.
# the shouldPrint() actually resets the flag, so it can't be used twice anyway...

This works as intended:

dat <- data.table(x1=1:10)
dat[, x2 := 1:10] # does not print
dat # prints

But now we have a different problem:

Currently we implicitly assume that each repr_* is independent of the other, so by calling shouldPrint() once in repr_text.data.table() we do not prevent printing with the other methods (e.g. a new repr_html.data.table) because the flag is reset. The actual situation is a bit different, because when repr_text returns an empty string, the rest of the mimetypes are not called in irkernel.

So maybe we should make this explicit in the documentation of repr?

The alternative is adding a repr_should_represent method which irkernel could then use and which has a special repr_should_represent.data.table which uses the flag.

I'm currently would prefere the former because it's basically what we do and it will only confuse users if they test in irkernel and it works and then (in the future) in another lib it works differently. On the other hand, for performance reasons, weh should probably add a repr_get_shorter_version() so that we don't do subseting 4 times and do not convert such big data.tables to data.frames... @flying-sheep @takluyver ?

@mattdowle: would you be able to include the repr_text.data.table function in the data.table? That way we do not need to guard against data.table being loaded and against older versions of the data.table package.

(In general, we would like to have packages exporting repr_xxx implementations for their data structures instead of having these implementations in the repr packages for this reasons...)

As a bonus, you could probably replace the knit_print specific callstack lookup (|| ( length(SYS) > 3L && as.character(SYS[[length(SYS)-3L]][[1L]]) %chin% mimicsAutoPrint ) with

knit_print.data.table <- function(x, ...) {
    if (!data.table::shouldPrint(x)) {
        invisible(NULL)
    } else {
        NextMethod() # which will fall back to your normal print, which will now see `shouldPrint() == T`
    }

This would prevent a problem if knitr would ever change it's knit_print.default implementation so that the above callstack would be different.

Thanks all! This is great info. Ok yes I see what you mean that adding the print methods to data.table might be best. Happy to give that a go. Will do.

Any update on this issue? Same behavior with data.table_1.9.6. If the data.table is very large, I find this causes intolerable lags in performance (hanging while trying to print).

The following prints the data.table during assignment by reference:

library(data.table)
dat <- data.table(x1=1:10)
dat[, x2 := 1:10]

but when I run this first:

assignInNamespace("cedta.pkgEvalsUserCode", NULL, "data.table")
repr_html.data.table <- function(obj, ...){
    if (data.table:::.global$print != "" && address(obj) == data.table:::.global$print) {
        NULL
    } else {
        NextMethod()
    }
}
repr_latex.data.table <- function(obj, ...){
    if (data.table:::.global$print != "" && address(obj) == data.table:::.global$print) {
        NULL
    } else {
        NextMethod()
    }
}
repr_text.data.table <- function(obj, ...){
    if (data.table:::.global$print != "" && address(obj) == data.table:::.global$print) {
        NULL
    } else {
        NextMethod()
    }
}

printing is suppressed during assignment by reference, but:

dat

prints no output.

R version 3.2.2 (2015-08-14)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 15.10

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.9.6

loaded via a namespace (and not attached):
 [1] R6_2.1.2           magrittr_1.5       IRdisplay_0.4.9000 pbdZMQ_0.2-3      
 [5] tools_3.2.2        crayon_1.3.2       uuid_0.1-2         stringi_1.0-1     
 [9] IRkernel_0.7       jsonlite_1.1       stringr_1.0.0      digest_0.6.10     
[13] chron_2.3-47       repr_0.9.9000      evaluate_0.9

Hi, looks like it works fine, but it needs the as-of-now unreleased shouldPrint

@mattdowle

Hi, this still seems to be an issue with the new notebook feature in the recent Rstudio 1.0 release. Any time a data table is modified with := it will inline the output in the notebook. I was reading this thread and saw that @mattdowle did something for knitr to avoid this behavior, can something be done with IRkernel too? The issue is very visible now that notebooks are so mainstream inside Rstudio 1.0

Is the Rstudio notebook using IRkernel? I know nothing of this.

i don’t think so. i think it has nothing to do with us.

Are there any plans to integrate the repr_XXX.data.table functions described above into the repr package so that this issue is fixed by default for anyone running a recent version of data.table? They work, but at the moment I end up copying and pasting them into the top of every notebook.

PRs welcome!

Fixed according to Rdatatable/data.table#933

Navigation Menu

Repository navigation

Setting data.table column outputs whole table to screen #109

25 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Setting data.table column outputs whole table to screen #109

Description

Activity

takluyver commented on May 8, 2015

abielr commented on May 9, 2015

abielr commented on May 12, 2015

takluyver commented on May 12, 2015

abielr commented on May 15, 2015

takluyver commented on May 15, 2015

flying-sheep commented on May 16, 2015

abielr commented on May 16, 2015

flying-sheep commented on May 16, 2015

takluyver commented on May 16, 2015

mattdowle commented on Jun 19, 2015

ericwatt commented on Dec 1, 2015

takluyver commented on Dec 1, 2015

takluyver commented on Dec 1, 2015

ericwatt commented on Dec 1, 2015

25 remaining items

mattdowle commented on May 19, 2016

breschke commented on May 25, 2016

jankatins commented on May 26, 2016

mattdowle commented on May 26, 2016

breschke commented on Sep 21, 2016

flying-sheep commented on Sep 21, 2016

jeffwong-nflx commented on Nov 22, 2016

takluyver commented on Nov 22, 2016

flying-sheep commented on Nov 22, 2016

abielr commented on Feb 19, 2017

flying-sheep commented on Feb 19, 2017

flying-sheep commented on Jun 8, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions