Repository navigation
Best practice for using max 2 cores on CRAN? #5658
Description
Activity
they have recently implemented a mechanism for checking compliance
I don't think it is that recent. Package
tiledbgot dinged three or so years ago, and I then implemented a scheme that works.Here is what the package does now. In essence, in every example file or unit test file (i.e. files that CRAN runs) I explicitly call a function that throttles cores down to
- the given function value, if there is one, in this throttler function
- or the value of option
Ncpuwith a fallback value of 2 (and that is what CRAN gets; as I recall they setNcpu) - or the value of environment variable
OMP_THREAD_LIMITcommon to set OpenMP and MKL thread counts.
So this gets us the max of 2 in everything that CRAN runs yet it can be overriden (if, they, your CI has more cores). I already had
Ncpuset locally to e.g. install multiple packages in parallel viainstall.packages()/update.package()).Importantly, it leaves general startup (i.e.
.onLoad(),.onAttach()) alone: it still gets full core count and performance.So normal users are not affected and get full speed, but CRAN gets its limit. I can detail that some more you if you care; it is in
tiledbfileR/config.R(but a little tied to how the config object progagates for us).Reacted by Avraham Adler, Sam Abbott, Chitu Okoli, Fonti Kar and Gregory JohnsonAlso reading DT news file can point to commits that set that up for DT
Reacted by Dirk Eddelbuettel@eddelbuettel thanks for the quick feedback, that approach seems similar to what I did, but more flexible. You wrote that CRAN sets option Ncpu -- do you know where is that documented? On https://cran.r-project.org/web/packages/policies.html I see "If running a package uses multiple threads/cores it must never use more than two simultaneously: the check farm is a shared resource and will typically be running many checks simultaneously. " but no mention of option Ncpu.
Also I was thinking that if CRAN indeed sets option Ncpu then the data.table default number of threads should respect that (avoids having to change example/test code in 1000+ packages with hard dependency on data.table).
@jangorecki I checked https://github.com/Rdatatable/data.table/blob/master/NEWS.md but I did not see mention of commits, can you please clarify?
An alternative to option Ncpu would be to just detect if on CRAN, for example using code below, and then throttle to two cores for data.table default.
> testthat:::on_cran function () { !env_var_is_true("NOT_CRAN") }
I do not see mention of NOT_CRAN env var on CRAN repository policy either, do you know where that is documented?
Relying on absence of
NOT_CRANhas always been and still is a hack I do not recommended, or use. YYMV.I do not recall where they document but it is documented somewhere that they allow two threads and AFAICR
Ncpusis the one vessel for that payload.There is a related R-devel thread, https://stat.ethz.ch/pipermail/r-devel/2021-November/081289.html that explains the env var
_R_CHECK_LIMIT_CORES_=TRUEis set when running R CMD check --as-cran.Reacted by Mauricio "Pachá" Vargas Sepúlveda?tools::check_packages_in_dir says Ncpus option is used to specify how many packages are checked in parallel,
Ncpus: the number of parallel processes to use for parallel installation and checking.But note that what you quoted does not limit it to package checks as your statement implies but to general 'checking'. Which is where we started: how to play nice at CRAN and not exceed limits.
This check is implemented via this code https://github.com/wch/r-source/blob/1c0545ba7c6c07e8c358eda552b875b1e4d6826d/src/library/tools/R/check.R#L4123 which gets the 2.5 ratio from the environment variable
_R_CHECK_EXAMPLE_TIMING_CPU_TO_ELAPSED_THRESHOLD_so we could check to see if that is set and then round down as a default for number of threads.Also:
Line 95 in 8803918
Recall that my approach was about taking the lower limit from Ncpu amd OMP_NUM_THREADS (and I no longer recall why I do not / did not also set OMP_THREAD_LIMIT).
Also
data.table/man/openmp-utils.Rd
Line 27 in 8803918
I also ran in to this on a submission this week and so did others recently (see r-devel thread where Dirk gives similar guidance).
I was confused as I assumed this would always be handled on the data.table side. I wonder if the CRAN test machine is no longer setting the
OMP_THREAD_LIMITenvironment variable which data.table is assuming it does?I haven't found in NEWS file anything useful as well.
#3300 seems to be related45 remaining items
- added a commit that references this issue
on Dec 7, 2023 - added a commit that references this issue
on Dec 8, 2023 Not sure this a proper solution, but this is how I handle the issue within the webseq package. The package is not yet on CRAN but it does pass
devtools::check()with the defaultcran = TRUE.In each of my functions that use parallelisation, I set the default value for the number of cores to
NULLand validate the number of cores with an internalget_mc_cores()function which 1. returns the number of cores if it has been set, 2. otherwise looks atgetOption("Ncpu")and 3. if that returnsNULL, falls back toparallel::detectCores().Within a regular R session I do not set "Ncpu" so R will not find it and will fall back to using many cores by default. However, I start each of my test files with
options("Ncpu" = 2L)so R CMD check will always work with 2 cores.Do you folks think this will work on CRAN? Thanks.
DT uses srtDTthreads rather than global options. Please read manual.
- R CMD check also runs example() for all your help pages, so any examples that use parallel processing must pass mc_cores = at most 2. Do not set options(Ncpu=2L) in examples because the user may have set it to a different value they are more comfortable with, and running such an example would change user's observable global state. What will your code do if parallel::detectCores() returns NA?Reacted by Tamás Stirling
Thanks @aitap! Oh, some of my examples were wrapped in
dontrun{}so they were not evaluated by R CMD check.. I unwrapped them to see what happens and R CMD check threw an error, as expected. I'm okay with wrapping, and AFAIK it is not against CRAN policies, but if that's not acceptable, then unfortunately settingoptions("Ncpu" = 2L)within tests is not enough to pass the checks. Thanks for flagging theNA, currently nothing. I am unsure exactly whenparallel::detectCores()returnsNA, so I'll probably just addstop()for now.You may want to look into the parallelly package's
availableCoresfunction which is written to always return an integer. Increases your dependency count but may be worth it.Reacted by Tamás Stirling and Chitu OkoliSorry to dredge up an old thread, but I've just been bitten by this again with a new CRAN submission.
Reading through the thread, I'm not sure whether there is a truly bullet proof solution. But I think @jangorecki's suggestions in #5658 (comment) seem the most sensible port of first call, alongside the
Sys.setenv("OMP_THREAD_LIMIT" = 2)option mentioned by @lrberge, @eddelbuettel and others.Should these suggestions not be added to the Importing data.table vignette?
It amazes me that this is still being discussed. Some of us fixed it years ago. It can be done zero-dependency too. YMMV.
- Unless some downstream code deliberately takes a look at the environment variables and limits its own thread usage (as suggested in <#5658 (comment)>), Sys.setenv("OMP_THREAD_LIMIT" = 2) may only work for the child processes, not OpenMP threads spawned by the current process. The OpenMP runtime is required to ignore Sys.setenv() done by the current process:Modifications to the environment variables after the program has started, even if modified by the program itself, are ignored by the OpenMP implementation.Reacted by Grant McDermott
For a while already I've followed a form of @eddelbuettel advice and basically just set the cores to 2 before every test (like so). Tests are almost never run by users, only developers and CRAN, so it should not affect performance.
- added a commit that references this issue
on Sep 3, 2026 - added a commit that references this issue
on Sep 17, 2026
CRAN has a policy that limits the number of cores that can be used during package checks, and they have recently implemented a mechanism for checking compliance. Upon recent CRAN submission of one of my R packages which imports data.table, I got the following rejection message from CRAN:
These messages can be suppressed by adding data.table::setDTthreads(1) at the start of each example.
Is there another/recommended way to avoid using too many cores when checking a CRAN package which uses data.table? The default number of threads in data.table is max cores/2 which is over the CRAN limit (max 2 cores I believe).
Also is there documentation about this somewhere? Probably would be good to mention in the datatable-importing vignette.