Repository navigation
groupingsets() wrong scoping logic #5560
Description
Activity
I suspect this is coming from:
jj = if (!missing(jj)) jj else substitute(j)
from
data.table:::groupingsets.data.table. According tosubstitute's documentation:Substitution takes place by examining each component of the parse tree as follows: If it is not a bound symbol in
env, it is unchanged. If it is a promise object, i.e., a formal argument to a function or explicitly created usingdelayedAssign(), the expression slot of the promise replaces the symbol. If it is an ordinary variable, its value is substituted, unlessenvis.GlobalEnvin which case the symbol is left unchanged.By which I think since
wdoes not exist indata.table:::groupingsets.data.table, it is left unchanged. You seem to be able to solve it by usingsubstitutein the call todata.table:::groupingsets.data.table, but as ajjargument.library(data.table) irisDT <- data.table(iris) foo = function(w) { bv = "Species" groupingsets( irisDT, jj = substitute(lapply(.SD, \(y) sum(y > w))), by = bv, sets = as.list(bv), .SDcols = c("Sepal.Length", "Petal.Length") ) |> print() irisDT[, lapply(.SD, \(y) sum(y > w)), .SDcols = c("Sepal.Length", "Petal.Length"), by=bv] } foo(w=5)
Reacted by Sindri and Kyle HaynesI would call it misuse of j arg.
Isn't jj argument description good enough to explain this use cases? Or maybe adding examples could be useful? PR welcomeThanks @avimallu for working example.
I'm not sure how that can be called a misuse of the
jarg, since the documentation mentions that whatever is provided to the argument will be sent to thejofdata.table. A typical call todata.tablecan use another variable defined outside of the set of columns in thatdata.tableobject, yeah?The
jjargument description is probably good enough from a developer perspective, but I haven't seen quoting being encouraged often online (unless you meant literal quotes",'), so it wasn't obvious to me how to use it until now (but that's also because I've looked it up before). One resource I came across is Hadley's Advanced R, specifically the section on metaprogramming.I'd love to file a PR with more examples on the use of the
jjargument - could you provide a better source for quoting that is specific to base R?R language manual, chapter 6.
Great about the PR. Ideally if it also convey that the use case described is a misuse of j and jj should be used instead.
Reacted by avimalluHi @jangorecki,
I came across this issue regarding the use of the jj argument in groupingsets() to resolve scoping issues when using external variables. I suggest enhancing the documentation by adding more examples to demonstrate its proper use.
Since this issue is still open, I wanted to check if there is a need for this update. If so, I would be happy to contribute by filing a PR.Reacted by Jan GoreckiThe underlying problem is that it's hard to properly forward NSE arguments because the substituted expressions become detached from their environments. For example, a
groupingsetscaller gives ajexpression that references a local variable (perhaps even using..), but whengroupingsetsitself callsx[, eval(jj), ...], thatevalhappens in an environment inheriting from the calling frame ofgroupingsets, not its caller. With base R metaprogramming, there's no nice way to handle this problem.We might try to construct an environment inheriting from the
parent.frame()ofgroupingsets(), mark it asdata.table-aware, populate it with references tox,jj, other arguments, and then evaluatequote(x[, eval(jj), other arguments])in it, but then the named variables in the environment might clash with the variables referenced byjj.Another way is evaluating
substitute(x[, jj, other arguments], list(x = x, jj = jj, other arguments))directly in theparent.frame()ofgroupingsets(), but that riskscedta()problems and creates a giant call object with the values of all arguments inlined.A more compatible solution might involve adding another argument to
[.data.table,enclos = parent.frame(), and setting that argument when callingx[, eval(jj), ..., enclos = parent.frame()]fromgroupingsets(), but would we want the extra complexity?Reacted by Jan GoreckiI’m not sure with any of the implementation , but I can help with an immediate documentation fix. This would involve updating the last example to use direct value substitution that is by avoiding symbols from outer environments.
Also ,I will add a Note section to highlight important limitations, including:
->When using jj with variables from outer environments, substitute values directly using substitute(expr, list(var = value)) to prevent environment detachment issues.
->For dynamic column names, use as.name(), as demonstrated in the cube() and rollup() examples.I don't think this is a responsible solution, but I could confirm that
jj = if (!missing(jj)) jj else substitute(j)is the issue. replacingsubstitute(j)withjcauses the script to run, albeit with warnings.> foo(4) Empty data.table (0 rows and 1 cols): Species Species Sepal.Length Petal.Length <fctr> <int> <int> 1: setosa 50 0 2: versicolor 50 34 3: virginica 50 50 Warning messages: 1: In `[.data.table`(x, 0L, eval(jj), by, .SDcols = .SDcols) : This j doesn't use .SD but .SDcols has been supplied. Ignoring .SDcols. See ?data.table. 2: In `[.data.table`(x, , eval(jj), by.set, .SDcols = .SDcols) : This j doesn't use .SD but .SDcols has been supplied. Ignoring .SDcols. See ?data.table.Thanks for clarifying . I see that replacing substitute(j) with j may bypass some issues but then leads to warnings with .SD handling.
From what I find out, preserving substitute(j) is important for proper evaluation, and a temporary fix might be to update the docs to use direct value substitution (e.g., substitute(expr, list(var=value))).- added a commit that references this issue
on May 22, 2026
I'm using data.table ‘1.14.7’ (just updated with
data.table::update_dev_pkg()) with R 4.2.1 on Windows 10.