WISH/BUG: Respect CPU resource limitations set by Linux CGroups to avoid CPU overuse and slowdown

Issue

data.table::getDTthreads() is not agile to Linux CGroups settings. If CGroups limits the number of CPU cores, then data.table will overuse the CPU resources is available to the R process.

For example, the 'Free' Posit Cloud plan gives you a single CPU core to play with. They use CGroups v1 to limit the CPU resource. Running the following from within their RStudio server reveals this:

> total <- as.integer(readLines("/sys/fs/cgroup/cpu/cpu.cfs_period_us"))
> total
[1] 100000
> quota <- as.integer(readLines("/sys/fs/cgroup/cpu/cpu.cfs_quota_us"))
> quota
[1] 100000
> cores <- quota / total
> cores
[1] 1

A user on the 'Premium' plan has 4 CPUs to play with, so they would get quota = 400000 and cores = 4 above.

The defaults of data.table does not pick this up:

> data.table::getDTthreads(verbose = TRUE)
  OpenMP version (_OPENMP)       201511
  omp_get_num_procs()            16
  R_DATATABLE_NUM_PROCS_PERCENT  unset (default 50)
  R_DATATABLE_NUM_THREADS        unset
  R_DATATABLE_THROTTLE           unset (default 1024)
  omp_get_thread_limit()         2147483647
  omp_get_max_threads()          16
  OMP_THREAD_LIMIT               unset
  OMP_NUM_THREADS                unset
  RestoreAfterFork               true
  data.table is using 8 threads with throttle==1024. See ?setDTthreads.
[1] 8

This means multi-threaded data.table tasks will overuse the CPU resources by 800%, which results in lots of overhead from context switching (unless there are other low-level mechanisms in data.table detecting this). CPU overuse will slow down the performance.

The overuse problem becomes worse the more CPU cores the host has. For example, the Posit Cloud instances currently runs with 16 vCPUs, but if they upgrade to say 64 vCPUs, the overuse will be 3200%. On research HPC environments, it's now common to see 192 CPUs, and I'd expect this number to grow over time.

FWIW, parallelly::availableCores() queries also CGroups/CGroups v2, e.g.

> parallelly:::availableCores()
cgroups.cpuquota 
               1 

> parallelly:::availableCores(which = "all")
          system   cgroups.cpuset cgroups.cpuquota            nproc 
              16               16                1               16

Session info

> sessionInfo()
R version 4.2.3 (2023-03-15)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.5 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3.10.3
LAPACK: /usr/lib/x86_64-linux-gnu/atlas/liblapack.so.3.10.3

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
 [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
 [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
[10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.14.8

loaded via a namespace (and not attached):
[1] compiler_4.2.3 tools_4.2.3

similar to #5573 about using data.table on slurm cluster.
currently we assume this kind of configuration should be handled by the user. For example, the user can set R_DATATABLE_NUM_THREADS environment variable.
in terms of dev/maintenance time, how many types of environment variables like this should we support? (SLURM, CGroups, ...?) how would we test each of them? given constraints on dev time, I would argue that it would be better to keep asking users to handle this.

it would be better to keep asking users to handle this

Given that data.table is such a central infrastructure package used internally by many packages and pipelines, I wonder how many users even know they are using data.table yet know they need to configure the number threads it should use.

For the problem reported here, CGroups throttling, I believe there are lots of data.table instances out there running slower than a single-thread version would do, and this without anyone even noticing the problem. It's only the savvy user who would know that this could be a problem and that it should be fixed.

in terms of dev/maintenance time, how many types of environment variables like this should we support? (SLURM, CGroups, ...?) how would we test each of them? given constraints on dev time, I would argue that it would be better to keep asking users to handle this.

These days, parallelly::availableCores() does a pretty good job at querying such settings + I'm happy to query additional settings if I get the specs.

Have there been any proposals to get parallel::detectCores() to match parallelly::availableCores()?

I don't think so - I'm not aware of any such attempts (except that I mentioned it in the below bugzilla report). I don't have the bandwidth to drive it myself, but I'd support any efforts in getting some or all of this into base R, especially since I think detectCores() does a disfavor to the community as it works right now - we would have been better off if it didn't exist in base R. I attempted a much smaller addition:

PR#17641 - WISH: Make parallel::detectCores() agile to new env var R_DEFAULT_CORES (https://bugs.r-project.org/show_bug.cgi?id=17641)

but it didn't fly.

Navigation Menu

Repository navigation

WISH/BUG: Respect CPU resource limitations set by Linux CGroups to avoid CPU overuse and slowdown #5620

Issue

Session info

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

WISH/BUG: Respect CPU resource limitations set by Linux CGroups to avoid CPU overuse and slowdown #5620

Description

Issue

Session info

Activity

tdhock commented on Mar 27, 2023

HenrikBengtsson commented on Mar 28, 2023

HenrikBengtsson commented on Jan 21, 2026

MichaelChirico commented on Jan 21, 2026

HenrikBengtsson commented on Jan 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions