Repository navigation
unique gives segfault from C stack overflow #4300
Description
Activity
Would you mind try to call setDT(data) before the unique call and report the results back here?
Just check if it’s due to the null self pointer.
I tried calling
setDT(data)first, but the result was the same.Oops, having your data would be necessary then.
Please find the data set here
OK, I create a smaller data set (23MB) that can reproduce this. The cause, I believe, is the recursive function
radix_r()is called for too many times (too deep), leading to an overflow on the calling stack.I don't know why this happens for this specific dataset as I can't reproduce it with randomly generated data.
Code
library(data.table) dt <- readRDS('~/Downloads/data_debug_tan.rds') dt2 <- data.table:::duplicated.data.table(dt)
Data
Debugged message
Error: C stack usage 8094212 is too close to the limitReacted by Jan GoreckiIts reproducible with the following setting:
- A wide data.table here with at least 500 columns (at least for my stack size)
- Duplicated rows
As @shrektan pointed out it's
fordervand therevoid radix_r()what actually errors.Example
x = matrix(rnorm(5000), nrow=10) idx = sample(10, 20, TRUE) DT = as.data.table(x[idx,]) forderv(DT, by=names(DT), sort=FALSE, retGrp=TRUE) # Error: segfault from C stack overflow
Calling it a second time also changes the error to
x = matrix(rnorm(5000), nrow=10) idx = sample(10, 20, TRUE) DT = as.data.table(x[idx,]) forderv(DT, by=names(DT), sort=FALSE, retGrp=TRUE) # Error in colnamesInt(x, by, check_dups = FALSE) : # Internal error: savetl_init checks failed (0 100 0x556d9d49d540 0x556d9e05f430). please report to data.table issue tracker.
Error seems to have been introduced by 05c0d45, no segfault at 092fec3
edit:
What seems to be the problem here is thatnradixgets so big that the recursionradix += 1untilradix+1 == nradixleads to the stack overflow.Reacted by Xianying Tan, Václav Hausenblas and Michael ChiricoI think I'm running into this with a data.table that is 18k columns, but I'm not getting any errors in R, it just crashes. If a fix is on the roadmap, is there a recommended way to de-duplicate wide data.tables in the meantime?
I think I'm running into this with a data.table that is 18k columns, but I'm not getting any errors in R, it just crashes. If a fix is on the roadmap, is there a recommended way to de-duplicate wide data.tables in the meantime?
You should at least get the
# Error: segfault from C stack overflowwhen running your code from terminal.AFAIA nobody is working on this fix yet. If the method which is failing is
uniquethen you can fall back to callunique.data.frame(DT)on yourdata.tableusing the method from base.edit: Another way to avoid this is to enlarge your stack size. Under linux you can do this with
ulimit -s unlimitedto set the stack size to unlimited. You can check that this worked viaCstack_info()["size"] #> size #> NA
Reacted by Michael ChiricoThis issue is no longer reproducible on recent master. This is likely due to recent rewrites of internal functions.
- added a commit that references this issue
on May 22, 2026
#Minimal reproducible examplecrashes RStudio, or in the R console:
I can provide
data_debug.rds(~300MB) if desired.#Output of sessionInfo()Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.8.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.8.0
locale:
[1] LC_CTYPE=en_CA.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_CA.UTF-8 LC_COLLATE=en_CA.UTF-8
[5] LC_MONETARY=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8
[7] LC_PAPER=en_CA.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_CA.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] data.table_1.12.8
loaded via a namespace (and not attached):
[1] compiler_3.6.2