Skip to content

Stress-test threading/batchSize is always on #3205

Description

@mattdowle

Two current limitations to overcome :

  • getDTthreads() doesn't return more than omp_get_max_threads(). This prevents stress testing logic and bounds on small data by increasing number of threads. I wasn't able to reproduce segfault in parallel gforce #3204 until I mimicked getDTthreads() returning 20 (greater than omp_get_max_threads()) on my machine.
> setDTthreads(20)
> getDTthreads(verbose=TRUE)
omp_get_max_threads() = 8
omp_get_thread_limit() = 2147483647
DTthreads = 20
[1] 8
  • I'm currently deliberately allowing very small batch sizes (sometimes 2 rows) in order to stress-test the test-suite. Otherwise we have to add a lot of large data tests before threading kicks in. This is good, and segfault in parallel gforce #3204 quickly came to light because of this good strategy. However, this stress-testing is left in for release. So it could be that overhead on small data is larger. Need to add a flag that is on for stress-testing in the test-suite, but turned off by default in release. It may be that overhead is not actually increased on small data and that it can be left on -- need to test.
    Update: throttle added in throttle threads for iterated small data tasks #4484 solves this 2nd part.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions