Repository navigation
Parallel build is no longer parallel #8971
Description
Activity
I'm not convinced this is the case, or at least I don't think it's completely simple.
Looking at setuptools, here's
build_extensions()checkingself.parallel, and calling_build_extensions_parallel()if it is truthy - https://github.com/pypa/setuptools/blob/3250c25197b299658cfd4d0db67770fc29b47277/setuptools/_distutils/command/build_ext.py#L478-L484def build_extensions(self) -> None: # First, sanity-check the 'extensions' list self.check_extensions_list(self.extensions) if self.parallel: self._build_extensions_parallel() else: self._build_extensions_serial()
If I adjust setup.py with
diff --git a/setup.py b/setup.py index 5d41e27d9..6af34afad 100644 --- a/setup.py +++ b/setup.py @@ -955,7 +955,10 @@ class pil_build_ext(build_ext): tk_libs = ["psapi"] if sys.platform in ("win32", "cygwin") else [] self._update_extension("PIL._imagingtk", tk_libs) + print("Start parallel check. self.parallel =", self.parallel) + build_ext._build_extensions_parallel = lambda x: print("Parallel method called") build_ext.build_extensions(self) + int("Finish check")
then I see that
build_ext._build_extensions_parallelis called in both macOS and Linux. So setuptools thinks it is running in parallel.I think the real answer is pypa/cibuildwheel#879 (comment)
The distutils feature only builds extensions in parallel, not the files of the extension. Which makes it useless for any package that only ships a single extension (which is common)
That comment points to pybind11 having a parallel compile plugin, which seems to be a way forward.
What did you do?
make debugormake installWhat did you expect to happen?
Expect compilation to happen across all the cores, taking ~ 2-3 seconds.
What actually happened?
Single threaded, taking ~15 seconds. There are something like 60 compilation units, and I have at least 8 cores.
What are your OS, Python and Pillow versions?
Sometime between ~6 years ago and now, probably when the setuptools migrated around or we went to the newer build tool specifications, the parallel option here: https://github.com/python-pillow/Pillow/blob/main/setup.py#L381 stopped working.
I can confirm that the code there is running,
self.parallelis set correctly, but setuptools is apparently not using it anymore.