Repository navigation
nested ':=' reference assignment fails #6768
Description
Activity
There is a workaround of returning a DT containing the results of the inner and outer function and assigning them all at once, however this structure was more desirable.
could you confirm (1) whether the issue is present on CRAN (2) whether updating to the current version of data.table fixes the issue? There are some recent related commits that might fix it
Thank you, the latest CRAN does indeed resolve this issue!
Great! Please also check on the current devel version, this is probably due to #6551
I see the same behavior on 1.16.0, 1.16.4 and current
master:Detected that j uses these columns: <none> Detected that j uses these columns: <none> Assigning to all 10 rows RHS_list_of_columns == false RHS for item 1 has been duplicated because MAYBE_REFERENCED==1 MAYBE_SHARED==1, but then is being plonked. length(values)==10; length(cols)==1) Assigning to all 10 rows RHS_list_of_columns == false RHS for item 1 has been duplicated because MAYBE_REFERENCED==1 MAYBE_SHARED==1, but then is being plonked. length(values)==10; length(cols)==1) a se new <int> <int> <NULL> 1: 1 11 NULL 2: 2 12 NULL 3: 3 13 NULL 4: 4 14 NULL 5: 5 15 NULL 6: 6 16 NULL 7: 7 17 NULL 8: 8 18 NULL 9: 9 19 NULL 10: 10 20 NULLIt's definitely odd that you've managed to create a column that's just
NULL-- the resultingdtis not a proper data.table.Could you double check & report which version produces the correct result?
Reacted by Anton N. MladenovI went back as far as 1.13.0 and was unable to find a version that produces the correct result, however I believe I found the root of the confusion:
library(data.table) options(datatable.verbose = T) dt = data.table(a=1:10) inner <- function(dt){ dt[,se:=1:10] } outer <- function(dt){ inner(dt); return (11:20) } dt[,new:=outer(dt)] print(dt) dt[,new:=outer(dt)] print(dt)The second occurrence of the assignment statement produces the correct result. I believe this is what occurred when I first tested the latest CRAN version.
- added a commit that references this issue
on Feb 2, 2025 I'm surprised you would expect this to work at all. Shouldn't it be
outer(dt)instead ofdt[,new:=outer(dt)]?I would expect dt[, new:=outer(dt)] to create 2 new columns one called 'new' with the return value of outer, (11:20) and another "side-effect" column, called 'se', with the values of 1:10 as created in the inner function.
Calling only outer(dt) would creates the column 'se' correctly, however the return value of outer() is lost.
library(data.table) options(datatable.verbose = F) dt = data.table(a=1:10) inner <- function(dt){ dt[,se:=1:10] } outer <- function(dt){ inner(dt); return (11:20) } outer(dt) print(dt)[1] 11 12 13 14 15 16 17 18 19 20 #lost return value of outer a se <int> <int> 1: 1 1 2: 2 2 3: 3 3 4: 4 4 5: 5 5 6: 6 6 7: 7 7 8: 8 8 9: 9 9 10: 10 10The result I would expect would be the following:
a se new <int> <int> <int> 1: 1 1 11 2: 2 2 12 3: 3 3 13 4: 4 4 14 5: 5 5 15 6: 6 6 16 7: 7 7 17 8: 8 8 18 9: 9 9 19 10: 10 10 20Thanks for more detail. If just running
outer(dt)where would the returned vector get placed into the data.table? The output makes sense to me. It is runninginner(dt)and then returning the vector you asked for. I don't see anywhere that it should be added as a column.The original example included an assignment to the column 'se' and 'new'
1.) column se should be created with values 1-10 here
2.) the outer function returns the values 11-20 which should be assigned to new
3.) the return value of outer is assigned to new
The original example:
library(data.table) options(datatable.verbose = TRUE) dt = data.table(a=1:10) inner <- function(dt){ dt[,se:=1:10] #(1) } outer <- function(dt){ inner(dt); return (11:20) #(2) } dt[,new:=outer(dt)] # (3) print(dt)The results of the above snippet were:
Detected that j uses these columns: <none> Detected that j uses these columns: <none> Assigning to all 10 rows RHS_list_of_columns == false RHS for item 1 has been duplicated because MAYBE_REFERENCED==1 MAYBE_SHARED==1, but then is being plonked. length(values)==10; length(cols)==1) Assigning to all 10 rows RHS_list_of_columns == false RHS for item 1 has been duplicated because MAYBE_REFERENCED==1 MAYBE_SHARED==1, but then is being plonked. length(values)==10; length(cols)==1) a se new <int> <int> <NULL> 1: 1 11 NULL 2: 2 12 NULL 3: 3 13 NULL 4: 4 14 NULL 5: 5 15 NULL 6: 6 16 NULL 7: 7 17 NULL 8: 8 18 NULL 9: 9 19 NULL 10: 10 20 NULLThe results I expected were:
a se new <int> <int> <int> 1: 1 1 11 2: 2 2 12 3: 3 3 13 4: 4 4 14 5: 5 5 15 6: 6 6 16 7: 7 7 17 8: 8 8 18 9: 9 9 19 10: 10 10 20I found that this occurs because the evaluation of the left-hand side at points (#1, & #3) both occur at the point when dt only contains the column 'a'. This leads to both new columns being assign the following available integer index, i.e '2'
Hopefully this was a more complete explanation of the bug. Please let me know is there are any other questions, confusion or concerns with the PR!
- added a commit that references this issue
on May 22, 2026
Issues arise when a function is called with the reference assignment operator ':=', which in-turn modifies the dt by reference as well. Below is the example running on the latest dev version. The result of the outer function overwrites the inner function in the incorrect column named 'se', the column named new is left empty.