Set number of threads after numpy import

This is not a bug report but just an enhancement proposal.

I think it would be useful and important to be able to easily set the number of threads used by Numpy after Numpy import.

From the perspective of library developers, it is often useful to be able to control the number of threads used by Numpy, see for example biopython/biopython#1401.

It is not difficult to do in a simple script when we are sure that Numpy or Scipy have not been imported previously with something like:

import os
os.environ["OMP_NUM_THREADS"] = "1"
import numpy as np

However, in a library there is a good chance that the user has already imported Numpy in its main script, with something like

import numpy as np

import fluidimage

In this case, I don't see how to set the number of threads used by Numpy from the fluidimage code.

Thus, it would be very convenient to have a function np.set_num_threads.

We would have to make this pluggable somehow to adapt for different linalg implementations. For instance, OpenBLAS seems to expose a openblas_set_num_threads, but MKL would use some other function

MKL has a function void mkl_set_num_threads( int nt );

https://software.intel.com/en-us/mkl-developer-reference-c-mkl-set-num-threads

which can be call through https://pypi.org/project/mkl/

But it seems to me it would be cleaner to have a generic Numpy function for this.

Most Numpy users don't even know which linalg library is used in the background.

If its alright, I would like to work on this. I haven't contributed to numpy before, so I will have to be a bit familiar with the codebase. Hence, it would be great if I could get some directions.

@touqir14

@touqir14 see the developer documentation. You should write some tests, tests that try out the new functions, verifying that they indeed set the number of threads you desire (perhaps by writing a C-level tests-only function that calls NPY_BEGIN_THREADS and ends with NPY_END_THREADS, and in the middle somehow counts the number of threads started) and then write code to implement the functions.

note that this is actually quite tricky to tackle.
NumPy does not actually know which BLAS library is used to implement the functions, it just assumes that whatever implements it uses standard BLAS apis and abis. The underlying implementation can also be switched without relying on recompilation of numpy (so abi compatibility, this is used e.g. in debian based systems).

As there unfortunately is no standard api to determine information on the provider numpy would be relient on runtime introspection to determine the actual provider of the functions. As there are not very many of them (some I recall are openblas, atlas, blis, mkl and reference blas) it might be possible but still difficult to get working portably.

Thanks for the heads up!

…

On Sat, Sep 15, 2018 at 6:52 PM Julian Taylor ***@***.***> wrote: note that this is actually quite tricky to tackle. NumPy does not actually know which BLAS library is used to implement the functions, it just assumes that whatever implements it uses standard BLAS apis and abis. As there unfortunately is no standard api to determine information on the provider numpy would be relient on runtime introspection to determine the actual provider of the functions. As there are not very many of them (some I recall are openblas, atlas, blis, mkl and reference blas) it might be possible but still difficult to get working portably. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11826 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGX9GCm2_lM2F9HHOe8A-cw3JihOHtLkks5ubaCzgaJpZM4WPbrD> .

This is an extremely annoying problem because I want to explicitly turn off multiprocessing for my background workers and as far as I can see there is no way of properly doing that (except downgrading numpy to < 1.14). OMP_NUM_THREADS=1 will solve my problem for the most part, but it will prevent my entire python process, including the main thread, from using openmp multiprocessing while I would like to only disable it in the background workers. As of now, I have 8 background workers (multiprocessing.Process), each of which will spawn another 8 threads to do np.dot computation (which is only a very small part of what they are actually doing). That clogs up the entire CPU. Any suggestions of how I could solve this?

Edit: downgrading to numpy 1.14.5 solves the problem. Starting form numpy 1.14.6 it's there

@FabianIsensee

@FabianIsensee The problem is that NumPy does not know right now what linalg backend you are using, OpenBlas, MKL, ATLAS, ... Did you see this stackoverflow question mentioned above, which suggests functions you can wrap and call to control this for your backend.

@mattip

@mattip Thanks for pointing that out! That is a good solution to the problem, provided that the system you are running on uses openBLAS. Unfortunately this solution has no effect on my system. What's especially annoying for me is that I am providing an open source framework that runs into this problem and I cannot know what blas library each and every one of the users is going to use.
I am not familiar with the numpy releases but the multi-threaded matmul appear starting from version 1.14.6. I checked the release notes and could not spot anything that would have enabled that (or am I blind?). Since multithreading these operations did not happen prior to that there must be a way to control that from within numpy (none of my libraries changed)

The change in 1.14.6 was building manylinux1 against OpenBLAS instead of ATLAS, we were already using OpenBLAS for Windows. As far as the build goes, 1.14.6 is pretty much 1.15.x. It sounds like we need to find a solution to this that doesn't depend on the user knowing which library is in use, probably adding some function in numpy that stores its info during the build.

Thank you for the clarification!
In an ideal world I would like to either specifically enable/disable multithreading by setting some property after importing numpy or even be able to set the number of threads to use manually. Would something like this be possible?

@njsmith

Would something like this be possible?

I don't know, but it sounds like we need it. If nothing else we can pass info upstream and try to encourage the libraries themselves to add such a feature, ideally as some sort of standard library interface in BLAS (LAPACK) itself. @njsmith IIRC, you reported that there was some work going on to produce a new standard? In any case, @matthew-brett I think we could have some effect on OpenBLAS. Maybe there is already such a feature.

Although with a single dynamic library I don't see how one could coordinate between callers. Hmm, not a simple problem, almost something that needs to be handled at the OS level. This is getting beyond my expertise.

@FabianIsensee

@FabianIsensee this seems troubling

Unfortunately this solution has no effect on my system

What exactly did you try and what was the result?

Edit: formatting

Would a threading control API also allow NumPy internal functions to use multiple threads? For example, sorting functions might need multiple threads to fully utilize system memory bandwidth.

Currently it appears that no NumPy internal functions are multi-threaded. But it would seem to be possible, without affecting applications that are tuned for multi-processing, if max_threads defaulted to 1 and could be changed by the user at run-time.

Hi,
what is the status of this issue? I would very much like to use newer numpy versions but this is a complete blocker as of right now! If some package makes use of multithreading then there must be a way to control the number of threads for the user. numpy really messes up CPU utilization on high CPU count servers!
Best,
Fabian

@FabianIsensee

@FabianIsensee numpy does not use multithreading on its own for exactly this reason. However, lthere are two exceptions:

Linear algebra/OpenBLAS (which numpy uses internally) can, and also the wheels will do this. You can use OMP_NUM_THREADS=1 to disable that. Depending on the BLAS you are using, other steps may be necessary (e.g. not sure this is sufficient for MKL).
There are Intel patches in NumPy linked against MKL, which also make it into Anaconda. In that case some numpy internal functions (outside linear algebra) may be patched to use multiple threads. I expect OMP_NUM_THREADS may work, there is likely even a monkeypatched additional functions to change it run-time, but this is outside of the normal numpy implementation, so I am not willing to check right now.

If you use anaconda, and the Intel patches are an issue, quite frankly, you may want to open an Anaconda issue or ping them here, so that they know that the idea of using threaded stuff everywhere is not trivially a good idea.

@FabianIsensee

@FabianIsensee I think this is exactly what my threading composability talks are about. Please refer to https://github.com/IntelPython/composability_bench/tree/master/scipy2018_demo
In short, we've implemented two approaches as 'smp' and 'tbb4py' modules, which provide command line interface in order to modify application behavior w.r.t. number of threads or task scheduler used underneath. joblib folks develop a good alternative 'loky', which might want to use the same approach as smp module: https://github.com/tomMoral/loky/pull/135

Intel's patches also include mklinit module which solves co-existence and initialization problems of OpenMP runtime from different vendors.

It's better to open issues directly at https://github.com/IntelPython/numpy

ENH: add functions to change number of threads used by BLAS

what is the status of this issue? I would very much like to use newer numpy versions but this is a complete blocker as of right now! If some package makes use of multithreading then there must be a way to control the number of threads for the user. numpy really messes up CPU utilization on high CPU count servers!

Simply execute export OMP_NUM_THREADS=1 before running your Python script and you solved the problem.
I had the same problem about a year ago, where adding more processes would result in slower code execution due to more OpenMP threads being requested on a single core.

However, I would like to see a way to manually control the number of threads that NumPy's linalg module uses, as that would still result in a significant speed-up for my code.

@anton-malakhov

Hi everyone,
please excuse my very late reply and thank you very much for your responses! I will look into @anton-malakhov 's repository, hopefully it does what I need.
@1313e , @seberg I am aware of the OMP_NUM_THREADS 'fix', but what I really need is a solution along the lines of 'numpy.set_omp_threads(1)'. I am providing an open source python package (batchgenerators) that uses multiprocessing.Process to parallelize data augmentation for deep neural networks (that's usually 8 or so workers. If this runs on an 40C/80T machine then each worker will spawn 80 threads for linear algebra such as rotating a coordinate mesh via matrix multiplication = up to 640 threads. That clogs up everything). I cannot expect our users to set OMP_NUM_THREADS whenever they are using our package. I am basically in the same situation as I believe @mimakaev is in :-/
Best,
Fabian

there is no API for you unfortunately, the tools were designed as command-line helpers for users. loky might be a better shot if you need an API. We discussed with sklearn folks that when loky will implement functionality from smp module, it can be rebased on top of loky. Overall, there is a need for an industry-wide KB, parallelism orchestration library, similar to what Apache Arrow does for memory layout of data frames, otherwise users will be swamped under env vars like MKL_NUM_THREADS, NUMPY_NUM_THREADS, NUMEXPR_NUM_THREADS.. and so on. API approach is good only if you know with whom you are synchronizing and if you can restore the defaults (otherwise there is a risk of affecting other modules), so it's rather context manager than a potential numpy.set_num_threads()

@FabianIsensee

@FabianIsensee I know that feeling, as I am in the same situation.
The way I fixed that is that I check if OMP_NUM_THREADS was set to 1 whenever my package is imported and it uses more than a single MPI process, and if not, it raises a warning telling you that you might want to set it to 1.
Of course, it is not a perfect solution, as the user will have to set it to 1 first and then re-execute their script, but at least you remind them of what they have to do.
And, obviously, if your code does not work at all when OMP_NUM_THREADS is not set to 1, you could instead raise an error upon package import.

We will probably use this code as a solution for now. It is hard to test this because you never know how are people/distributions compiling numpy. But this code I believe should work at least for MKL.

import os, ctypes 
def set_mkl_threads():
   
    try:
        import mkl
        mkl.set_num_threads(1)
        return 0
    except:
        pass 
    
    for name in [ "libmkl_rt.so", "libmkl_rt.dylib", "mkl_Rt.dll"]: 
        try: 
            mkl_rt = ctypes.CDLL(name)
            mkl_rt.mkl_set_num_threads(ctypes.byref(ctypes.c_int(1)))
            return 0 
        except:
            pass   

    os.environ["OMP_NUM_THREADS"] = "1" # export OMP_NUM_THREADS=4
    os.environ["OPENBLAS_NUM_THREADS"] = "1" # export OPENBLAS_NUM_THREADS=4 
    os.environ["MKL_NUM_THREADS"] = "1" # export MKL_NUM_THREADS=6
    os.environ["VECLIB_MAXIMUM_THREADS"] = "1" # export VECLIB_MAXIMUM_THREADS=4
    os.environ["NUMEXPR_NUM_THREADS"] = "1" # export NUMEXPR_NUM_THREADS=6

set_mkl_threads()

There is also a nearly-complete solution somewhere in the link below, but we didn't use it in the end because it seemed complicated.
https://github.com/tomMoral/loky/pull/135/files#diff-e49a2eb30dd7db1ee9023b8f7306b9deR110

I am writing a dedicated package threadpoolctl with the solution we wrote in loky (tommoral/loky#135). It currently handles mkl, openmp and openblas with both a context API and a functional one. I did not finish packaging it to pypi and it is still in heavy development but if some of you want to test it from master and give me some feedback, it would be awesome.

@tomMoral

@tomMoral threadpoolctl would be great to solve my original issue without Numpy modification! Thank you for this great work! We'll use it in fluidimage when it's ready (we should actually also use joblib/loky for executors of topologies but it's another story).

I tried threadpoolctl and it seems to work nicely with the setup I used to try it (Debian, pyenv, numpy/scipy installed with pip). However, I observed something strange so I filled an issue in your bug tracker.

Can we mark this as Closed? Maybe we should pivot it to "document use of threadpoolctl to control threads"?

Closing, since the threadpoolctl package handles this.

Navigation Menu

Repository navigation

Set number of threads after numpy import #11826

21 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Set number of threads after numpy import #11826

Description

Activity

mattip commented on Aug 28, 2018

paugier commented on Aug 29, 2018

touqir14 commented on Sep 15, 2018

mattip commented on Sep 15, 2018

juliantaylor commented on Sep 16, 2018

touqir14 commented on Sep 16, 2018

FabianIsensee commented on Sep 26, 2018

mattip commented on Sep 26, 2018

FabianIsensee commented on Sep 27, 2018

charris commented on Sep 27, 2018

FabianIsensee commented on Sep 27, 2018

charris commented on Sep 27, 2018

charris commented on Sep 27, 2018

mattip commented on Sep 27, 2018

21 remaining items

andravin commented on Dec 27, 2018

FabianIsensee commented on Feb 28, 2019

seberg commented on Feb 28, 2019

anton-malakhov commented on Feb 28, 2019

1313e commented on Mar 26, 2019

FabianIsensee commented on Mar 26, 2019

anton-malakhov commented on Mar 26, 2019

1313e commented on Mar 26, 2019

mimakaev commented on Mar 27, 2019

tomMoral commented on Mar 27, 2019

paugier commented on Mar 27, 2019

mattip commented on May 26, 2019

mattip commented on Aug 18, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions