Repository navigation
.SD in grouped operations causes update() to refit on wrong data subset #7831
Description
Activity
- Will try to give a better answer later, but when you're struggling with data mutation during by-group operations, use copy().Reacted by Jan Gorecki
copy(.SD)on it's own does not solve this, likely because the two model calls still live in the same environment and are executed lazily.mods_copy <- dat[, .(model = .({d <-copy(.SD) ; lmer(y ~ Time + (1 | ID), data=d)} )) , by = Grp] mods_copy[,.(env=.(environment(model[[1]]@call$formula))),by=Grp][,env]## same env in both groups all.equal(update(mods_copy[Grp=="A", model][[1]]), update(mods_copy[Grp=="B", model][[1]]) ) ## but the updated models on diff grps are the same
my current workaround is to put the
lmerinsidelocal()or inside a custom function This forces the formula calls into separate environments for differentgrp.fit_group <- function(d) lmer(y ~ Time + (1 | ID), data = d) datDT_fix1 <- dat[, .(model = .(fit_group(copy(.SD)))), by = Grp] mods_func[,update(model[[1]]) |> AIC() , by = Grp] # correct AICs in both `Grp`s
I understand that the shared env in grouping operations is relevant to the architecture and a considerable efficiency gain.
However it would be very helpful, if you could consider- adding a warning when this behaviour might cause unexpected results or even
- introducing a mechanism to force environment isolation if call objects are present.
It does seem to be a duplicate of #507.
In my particular case,
update()was quite well hidden (emmeans::contrastscallspbkrtest::vcovAdj.lmerModwhich then under some conditions callsupdate). I'd very much appreciate to see a warning in such a use caseReacted by aitapOh, I see. Normally people get bitten because something remembers a pointer to the by-group object while
data.tablemutates its contents between the groups:> dat[, .(model = .(lmer(y ~ Time + (1 | ID))), `address(y)` = address(y)), by = Grp] Key: <Grp> Grp model address(y) <char> <list> <char> 1: A <lmerMod[13]> 0x55a1a5b38e30 2: B <lmerMod[13]> 0x55a1a5b38e30...but here you're bitten because
lme4:::update.merModevaluatesgetCall(model)insideenvironment(formula(model)), and that environment is also the same between the by-group calls tolmer(). You can force the creation of a different environment usinglocal()or theenvir=...argument of theeval()family of functions:dat[, .(model = .( evalq(lmer(y ~ Time + (1 | ID)), copy(.SD)) )), by = Grp ][, model[[1]] |> update() |> all.equal(model[[1]]) # still fine after update() ] # [1] TRUE
It might be possible to use the R ≥ 4.0 reference counts to prevent this kind of bugs, but it won't be easy.
I often fit models to subsets within data.tables, similar to Section 3.3 in the .SD vignette. However, is the following behavior intended?
A model fit m1 on the subset Grp == "A" is identical to the respective grouped model fit m2, as expected. Yet, a trivial update() call yields vastly different results, which did surprise me a little.
As far as I understand, this issue stems from how environments are evaluated:
I would greatly appreciate it if there were (or someone could point me to) an elegant way to ensure that .SD retains the correct group-specific subset when used with a grouping variable :)