Repository navigation
Parallel build enabled by default #943
Description
Activity
I don't think that it's a problem where Threads > Files, as we've actually got several sets of files that are compiled, some with only one file in the set (such as imagemorph, webp, and the tk bits), and they are fine with as much parallelism as we can throw at it for the rest of the items.
Travis is kind of an odd system, as they report 32 processors, but you only get access to ~1.5 or so, according to their docs. So you're likely to run into resource exhaustion with the default multiprocessing setup of N=cores.
If setting the MAX_PROCESSES = 1 works, that's probably the best way to go. You can probably bump that up to 4 for a reasonable speedup. We may put in something that explicitly checks for travis. We also might look at adding some documentation on the out of the box build options.
I just have the problem with setting the parallel build by default (which actually overwrites our settings). This happens both on Travis (where our parallel build is disabled), as well as on our local builds, and also our local builds fail with the same or similar exceptions. I am not quite sure, why these exceptions are raised, but it has something to do with the Threading module, which is introduced by your tweaks.
So, people who are not aware of the MAX_PROCESSES setup and which run into issues similar to the ones we have, might not find the solution (MAX_PROCESSES=1). That's why I would enable the parallel building only when you are in your environment and are sure that you can control the build. Forcing other people to set an environment variable to disable your (maybe malfunctioning) tweaks is just semi optimal.
I had an related issue with C macros. In an external library, someone defined a C macro called 'max'. Every time why where including this header, we had to '#undef max' in order to be able to use 'std::max'. With your environment variable, I feel somewhat the same...
Best wishes
Manuel (who just wants to make the world easier...)
Hi,
one of our C++/Python libraries requires pillow. When we try to build pillow on Travis, we get a compilation error:
see:
https://travis-ci.org/bioidiap/bob.io.video/jobs/35247966
I am pretty sure that this is related to the parallel build, which is enabled by default using all possible processors (see Pillow/mp_compile.py), which on Travis seems to be 32. I guess that the parallel compilation does not like it, when there are more threads than files to be compiled.
My problem with your script is that it affects both your compilation and ours, which happens after yours. Therefore, I don't think it is a good strategy to enable the parallel build when the MAX_CONCURRENCY flag is not set. I think it would be smarter to enable parallel builds only when MAX_CONCURRENCY is set (e.g. to -1 to get cpu_count() threads), to avoid conflicts with other packages.
Note that we have implemented a similar solution for parallel building, which does not seem to have the problem when less files than threads are available. Maybe you want to have a look at the "parallelCompile" function on line 669 of:
https://github.com/bioidiap/bob.extension/blob/master/bob/extension/__init__.py#L667-686
For the time being, I found the solution to set MAX_CONCURRENCY=1 before building.
Cheers
Manuel