gcc-UBSAN on CRAN errors on forder.c:137:3: runtime error: null pointer passed as argument 2

I got a message from CRAN regarding seqHMM package that there is a test failuri with gcc-UBSAN stemming from data.table: https://www.stats.ox.ac.uk/pub/bdr/memtests/gcc-UBSAN/seqHMM/tests/testthat.Rout, namely forder.c.

I'll paste the trimmed output here in case the log files disappear:

> library(testthat)
> library(seqHMM)
Please cite seqHMM appropriately, see `citation('seqHMM')` for details.
> 
> test_check("seqHMM")
forder.c:137:3: runtime error: null pointer passed as argument 2, which is declared to never be null
    #0 0x7b6289002f38 in flush /tmp/Rtmp8svXjO/R.INSTALL29ad3d450313e2/data.table/src/forder.c:137
    #1 0x7b628907a303 in flush /tmp/Rtmp8svXjO/R.INSTALL29ad3d450313e2/data.table/src/forder.c:128
    #2 0x7b628907a303 in radix_r /tmp/Rtmp8svXjO/R.INSTALL29ad3d450313e2/data.table/src/forder.c:1327
    #3 0x7b6289085235 in forder /tmp/Rtmp8svXjO/R.INSTALL29ad3d450313e2/data.table/src/forder.c:804
    #4 0x7b628908a830 in forderReuseSorting /tmp/Rtmp8svXjO/R.INSTALL29ad3d450313e2/data.table/src/forder.c:1764
    #5 0x000000742222 in R_doDotCall /data/gannet/ripley/R/svn/R-devel/src/main/dotcode.c:780
    #6 0x0000008d6f6a in bcEval_loop /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:8668
    #7 0x0000008ad7ab in bcEval /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:7501
    #8 0x00000084e0f2 in Rf_eval /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:1167
    #9 0x000000865b0a in R_execClosure /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2393
    #10 0x000000869d1a in applyClosure_core /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2306
    #11 0x000000a85ab3 in Rf_applyClosure /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2328
    #12 0x000000a85ab3 in applyMethod /data/gannet/ripley/R/svn/R-devel/src/main/objects.c:120
    #13 0x000000a89de0 in dispatchMethod /data/gannet/ripley/R/svn/R-devel/src/main/objects.c:473
    #14 0x000000a8bd65 in Rf_usemethod.isra.0 /data/gannet/ripley/R/svn/R-devel/src/main/objects.c:513
    #15 0x00000088b1d3 in tryDispatch /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:6069
    #16 0x0000008d8784 in bcEval_loop /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:8264
    #17 0x0000008ad7ab in bcEval /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:7501
    #18 0x00000084e0f2 in Rf_eval /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:1167
    #19 0x000000865b0a in R_execClosure /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2393
    #20 0x000000869d1a in applyClosure_core /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2306
    #21 0x000000a85ab3 in Rf_applyClosure /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2328
    #22 0x000000a85ab3 in applyMethod /data/gannet/ripley/R/svn/R-devel/src/main/objects.c:120
    #23 0x000000a89de0 in dispatchMethod /data/gannet/ripley/R/svn/R-devel/src/main/objects.c:473
    #24 0x000000a8bd65 in Rf_usemethod.isra.0 /data/gannet/ripley/R/svn/R-devel/src/main/objects.c:513
    #25 0x000000a8c986 in do_usemethod /data/gannet/ripley/R/svn/R-devel/src/main/objects.c:579
    #26 0x0000008cd771 in bcEval_loop /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:8138
    #27 0x0000008ad7ab in bcEval /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:7501
    #28 0x00000084e0f2 in Rf_eval /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:1167
    #29 0x000000865b0a in R_execClosure /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2393
    #30 0x000000869d1a in applyClosure_core /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2306
    #31 0x00000084e793 in Rf_applyClosure /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2328
    #32 0x00000084e793 in Rf_eval /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:1280
    #33 0x00000087ea2e in do_set /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:3567
    #34 0x00000084ebb6 in Rf_eval /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:1232
    #35 0x00000072d88a in do_External /data/gannet/ripley/R/svn/R-devel/src/main/dotcode.c:573
    #36 0x0000008c0562 in bcEval_loop /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:8118
    #37 0x0000008ad7ab in bcEval /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:7501
    #38 0x00000084e0f2 in Rf_eval /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:1167
    #39 0x000000865b0a in R_execClosure /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2393
    #40 0x000000869d1a in applyClosure_core /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2306
    #41 0x00000084e793 in Rf_applyClosure /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2328
    #42 0x00000084e793 in Rf_eval /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:1280
    #43 0x00000086426b in do_begin /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2996
    #44 0x00000084ebb6 in Rf_eval /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:1232
    #45 0x000000884541 in do_eval /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:3941
    #46 0x0000008c0562 in bcEval_loop /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:8118
    #47 0x0000008ad7ab in bcEval /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:7501
    #48 0x00000084e0f2 in Rf_eval /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:1167
    #49 0x000000865b0a in R_execClosure /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2393
    #50 0x000000869d1a in applyClosure_core /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2306
    #51 0x00000084e793 in Rf_applyClosure /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2328
    #52 0x00000084e793 in Rf_eval /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:1280
    #53 0x000000885633 in do_eval /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:3959
    #54 0x0000008c0562 in bcEval_loop /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:8118
    #55 0x0000008ad7ab in bcEval /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:7501
    #56 0x00000084e0f2 in Rf_eval /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:1167
    #57 0x000000865b0a in R_execClosure /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2393
    #58 0x000000869d1a in applyClosure_core /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2306
    #59 0x00000086b637 in Rf_applyClosure /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2328
    #60 0x00000086b637 in R_forceAndCall /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2460
    #61 0x00000049a0b6 in do_lapply /data/gannet/ripley/R/svn/R-devel/src/main/apply.c:75
    #62 0x000000a848a6 in do_internal /data/gannet/ripley/R/svn/R-devel/src/main/names.c:1411
    #63 0x0000008cd771 in bcEval_loop /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:8138
    #64 0x0000008ad7ab in bcEval /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:7501
    #65 0x00000084e0f2 in Rf_eval /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:1167
    #66 0x000000865b0a in R_execClosure /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2393
    #67 0x000000869d1a in applyClosure_core /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2306
    #68 0x00000084e793 in Rf_applyClosure /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:2328
    #69 0x00000084e793 in Rf_eval /data/gannet/ripley/R/svn/R-devel/src/main/eval.c:1280
    #70 0x0000009f2fa3 in Rf_ReplIteration /data/gannet/ripley/R/svn/R-devel/src/main/main.c:265
    #71 0x0000009f2fa3 in R_ReplConsole /data/gannet/ripley/R/svn/R-devel/src/main/main.c:317
    #72 0x0000009f82ba in run_Rmainloop /data/gannet/ripley/R/svn/R-devel/src/main/main.c:1234
    #73 0x000000a024d2 in Rf_mainloop /data/gannet/ripley/R/svn/R-devel/src/main/main.c:1241
    #74 0x0000004130af in main /data/gannet/ripley/R/svn/R-devel/src/main/Rmain.c:29
    #75 0x7f629d8115f4 in __libc_start_call_main (/lib64/libc.so.6+0x35f4) (BuildId: 2b3c02fe7e4d3811767175b6f323692a10a4e116)
    #76 0x7f629d8116a7 in __libc_start_main@@GLIBC_2.34 (/lib64/libc.so.6+0x36a7) (BuildId: 2b3c02fe7e4d3811767175b6f323692a10a4e116)
    #77 0x000000413a34 in _start (/data/gannet/ripley/R/gcc-SAN3/bin/exec/R+0x413a34) (BuildId: 219e2e1b6ad65e4b9894b1f68e9454130a96e247)

Check output on CRAN:

`* using log directory ‘/data/gannet/ripley/R/packages/tests-gcc-SAN/seqHMM.Rcheck’
* using R Under development (unstable) (2025-06-03 r88260)
* using platform: x86_64-pc-linux-gnu
* R was compiled by
    gcc (GCC) 15.1.1 20250425 (Red Hat 15.1.1-1)
    GNU Fortran (GCC) 15.1.1 20250425 (Red Hat 15.1.1-1)
* running under: Fedora Linux 42 (Workstation Edition)
* using session charset: UTF-8
* using option ‘--no-stop-on-test-error’
* checking for file ‘seqHMM/DESCRIPTION’ ... OK
* this is package ‘seqHMM’ version ‘2.0.0’
* package encoding: UTF-8
* checking package dependencies ... OK
* checking if this is a source package ... OK
* checking if there is a namespace ... OK
* checking for hidden files and directories ... OK
* checking for portable file names ... OK
* checking whether package ‘seqHMM’ can be installed ... [26m/46m] OK
* used C++ compiler: ‘g++ (GCC) 15.1.1 20250521 (Red Hat 15.1.1-2)’
* checking package directory ... OK
* checking whether the package can be loaded ... [24s/109s] OK
* checking whether the package can be loaded with stated dependencies ... [20s/89s] OK
* checking whether the package can be unloaded cleanly ... [20s/74s] OK
* checking whether the namespace can be loaded with stated dependencies ... [20s/71s] OK
* checking whether the namespace can be unloaded cleanly ... [23s/74s] OK
* checking loading without being on the library search path ... [23s/80s] OK
* checking compiled code ... OK
* checking installed files from ‘inst/doc’ ... OK
* checking files in ‘vignettes’ ... OK
* checking examples ... [211s/584s] OK
* checking tests ... [27m/58m] OK
  Running ‘testthat.R’ [27m/58m]
* checking package vignettes ... OK
* checking re-building of vignette outputs ... [15m/20m] OK
* DONE
Status: OK

I'm not sure how to debug this due to lack of access to gcc-UBSAN?

Couldn't easily reproduce, but it's most likely related to #6911 and probably needs a similar fix (don't try to copy 0 bytes from a potentially invalid pointer). Will try with a different compiler version. Don't see an easy workaround, we may need another patch release to fix this problem.

@MichaelChirico

Cc: @MichaelChirico & @TysonStanley: when some other package gets a CRAN deadline due to our code and there's no obvious workaround, we have to publish a patch release, right?

Long story short, it wasn't enough to find the right version of the compiler and sanitizer runtime. (Although seqHMM does find a couple of 0-length data pointer accesses in vctrs and rlang.)

The memcpy() call in flush() can only get a NULL second argument and not crash if n*sizeof(int) is also 0, that is, the current thread has never performed any work, or at least never called push(). What if the problem only occurs under heavy load?

Let's catch this manually:

--- data.table.orig/src/forder.c        2025-05-25 20:52:47.000000000 +0300
+++ data.table/src/forder.c     2025-06-08 12:22:59.803429161 +0300
@@ -1,3 +1,4 @@
+#include <signal.h>
 #include "data.table.h"
 /*
   Inspired by :
@@ -134,6 +135,7 @@
     gs = realloc(gs, gs_alloc*sizeof(int));
     if (gs==NULL) STOP(_("Failed to realloc group size result to %d*4bytes"), (int)gs_alloc);
   }
+  if (!gs_thread[me]) raise(SIGTRAP);
   memcpy(gs+gs_n, gs_thread[me], n*sizeof(int));
   gs_n += n;
   gs_thread_n[me] = 0;

And simulate heavy load by pinning all OpenMP threads to one CPU core:

# artificial thread contention:
library(data.table)
setDTthreads(0) # create many threads...
system(paste("taskset -p 1", Sys.getpid())) # ...all on one core
stopifnot(getDTthreads() > 1)

# reproducer from seqHMM tests:
library(seqHMM)
data("hmm_biofam")
y <- hmm_biofam$channel_names
age <- "age"
id_var <- "individual"
d <- stslist_to_data(hmm_biofam$observations, id_var, age, y)
set.seed(1)
fit <- estimate_nhmm(
 n_states = 3,
 emission_formula = c(Marriage, Parenthood) ~ Residence,
 data = d, time = age, id = id_var, maxeval = 3, method = "DNM"
)
fit <- bootstrap_coefs(fit, nsim = 5)
d$Marriage[d$age > 25] <- NA
d$Parenthood[d$age > 28] <- NA
out <- predict(
 fit, newdata = d, condition = "Residence", probs = c(0.1, 0.9)
)

And there you go:

Thread 1 "R" received signal SIGTRAP, Trace/breakpoint trap.
(gdb) frame 3
#3  0x00007b5024d3c072 in flush () at forder.c:138
138       if (!gs_thread[me]) raise(SIGTRAP);
(gdb) p me
$2 = 0
(gdb) p gs_thread[me]
$4 = (int *) 0x0
(gdb) p n
$5 = 0
(gdb) call Rf_PrintValue(R_GlobalContext->nextcontext->call)
`[.data.table`(newdata, , list(probability = mean(estimate)), 
    by = cond_all)

Here's a non-seqHMM reproducer:

# need to induce contention first (as above)
n <- 288000
repeat data.table(
 id = sample(2000, n, TRUE),
 a = sample(2, n, TRUE), b = sample(3, n, TRUE), c = sample(3, n, TRUE),
 x = rnorm(n)
)[, .(x = mean(x)), by = c('a', 'b', 'c')]
# forder.c:137:3: runtime error: null pointer passed as argument 2, which is declared to never be null

It's not 100% reliable, but it does reach the gs_thread[me] == NULL code path after a few tries. I'll try to patch this and similar places in the code.

This is a pretty unique situation as CRAN did not reach out to us directly but did prompt them to work with us. Based on a quick review, does look like we'll need to release a patch for this to keep seqHMM up and running.

Does the PR fix it? If so, can cherry-pick this weekend and get the patch release put together.

Had a sec to read the PR, does look like it fixes it. Will plan to get a patch ready this weekend.

@helske

Thank you for the prompt reply! Yes, cherry-picking #7055 should solve @helske's problem.

Thank you for the quick reaction! Indeed, it was interesting that I got the message about potential removal from CRAN while being clealy data.table issue as acknowledged in the email as well...

(textproc/R-data.table) Updated 1.16.4 to 1.18.4, Fix build against 4…

Navigation Menu

Repository navigation

gcc-UBSAN on CRAN errors on `forder.c:137:3: runtime error: null pointer passed as argument 2` #7051

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

gcc-UBSAN on CRAN errors on forder.c:137:3: runtime error: null pointer passed as argument 2 #7051

Description

Activity

aitap commented on Jun 6, 2025

aitap commented on Jun 8, 2025

TysonStanley commented on Jun 9, 2025

TysonStanley commented on Jun 9, 2025

aitap commented on Jun 10, 2025

helske commented on Jun 10, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

gcc-UBSAN on CRAN errors on `forder.c:137:3: runtime error: null pointer passed as argument 2` #7051