Repository navigation
Reduce AVIF wheel size? #8856
Description
Activity
Is there anything we can do to make it smaller?
Maybe there is no need to carry multiple encoders and decoders? E.g. libaom covers both ends (and is libavif preferred for encoding), no need to build w/ and ship dav1d and rav1e+svt-av1 as well? That being said, libaom seems to be the largest of them all (at least on my machine/platform), and dav1d seems to be preferred for decoding speed...
Package size changed from 5.1M unpacked to 54M PIL, is that mandatory?
The AVIF .so has made the wheels about 4 times bigger:
See also #5201 (comment)
Reacted by Hugo van Kemenade and Jeffrey 'Alex' ClarkThe ability to test/switch between codecs is a handy feature.
This made me sad too because my PIL venv install went from 6.4MB to 59MB which made my container image much bigger :(
If I had to choose a codec for removal, it would probably be rav1e. Development on rav1e has been a bit stagnant, and in my opinion AOM has come out ahead of it with the 3.12.0 release. Rav1e is also quite large. I'm running a build now without it to figure out what its contribution to the wheel size is.
I recommend including libaom for the encoder and dav1d for the decoder.
@fdintino Frankie: To build an encoder-only version of libaom, pass
-DCONFIG_AV1_DECODER=0to libaom's cmake configure command. Then pass-DAVIF_CODEC_AOM_DECODE=OFFto libavif's cmake configure command.Experimenting with pillow-11.2.0.dev0-cp313-cp313-musllinux_1_2_x86_64.whl, I find
- Currently, 19.8mb
- Without dav1d, 18.8mb (AOM, RAV1E and SVT encoders, AOM decoder)
- Without svt, 16.5mb (AOM and RAV1E encoders, AOM and DAV1D decoders)
- Without rav1e, 12.8mb (AOM and SVT encoders, AOM and DAV1D decoders)
- Without dav1d, rav1e and svt, 8.5mb (AOM encoder and decoder)
(For comparison, pillow-11.1.0-cp313-cp313-musllinux_1_2_x86_64.whl was 4.4 MB in the last release.)
Thanks all for working on this!
Here's a comparison of file size in MB between #8858 (only aomenc and dav1d codecs) and #8869 (only aom codec), comparing each size to that of 11.1.0 from January.
filename 11.1.0 11.2.0 bigger aomenc & dav1d bigger aom only bigger pillow.tar.gz 46.7 47.0 x1.0 47.0 x1.0 47.0 x1.0 cp310-cp310-macosx_10_10_x86_64 3.2 11.4 x3.5 6.3 x2.0 6.4 x2.0 cp310-cp310-macosx_11_0_arm64 3.1 8.9 x2.9 5.2 x1.7 5.7 x1.8 cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64 4.3 17.1 x4.0 8.1 x1.9 7.9 x1.8 cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64 4.4 18.9 x4.3 8.6 x2.0 8.1 x1.8 cp310-cp310-manylinux_2_28_aarch64 4.4 17.6 x4.0 6.8 x1.6 8.3 x1.9 cp310-cp310-manylinux_2_28_x86_64 4.5 19.6 x4.4 7.5 x1.7 8.4 x1.9 cp310-cp310-musllinux_1_2_aarch64 4.3 18.0 x4.2 7.0 x1.6 8.5 x2.0 cp310-cp310-musllinux_1_2_x86_64 4.4 19.8 x4.5 7.6 x1.7 8.5 x1.9 cp310-cp310-win32 2.3 2.3 x1.0 2.3 x1.0 2.3 x1.0 cp310-cp310-win_amd64 2.6 13.8 x5.3 7.2 x2.8 7.1 x2.7 cp310-cp310-win_arm64 2.4 2.4 x1.0 2.4 x1.0 2.4 x1.0 cp311-cp311-macosx_10_10_x86_64 3.2 11.4 x3.5 6.3 x2.0 6.4 x2.0 cp311-cp311-macosx_11_0_arm64 3.1 8.9 x2.9 5.2 x1.7 5.7 x1.8 cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64 4.3 17.1 x4.0 8.1 x1.9 7.9 x1.8 cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64 4.4 18.9 x4.3 8.6 x2.0 8.1 x1.8 cp311-cp311-manylinux_2_28_aarch64 4.4 17.6 x4.0 6.8 x1.6 8.3 x1.9 cp311-cp311-manylinux_2_28_x86_64 4.5 19.6 x4.4 7.5 x1.7 8.4 x1.9 cp311-cp311-musllinux_1_2_aarch64 4.3 18.0 x4.2 7.0 x1.6 8.5 x2.0 cp311-cp311-musllinux_1_2_x86_64 4.4 19.8 x4.5 7.6 x1.7 8.5 x1.9 cp311-cp311-win32 2.3 2.3 x1.0 2.3 x1.0 2.3 x1.0 cp311-cp311-win_amd64 2.6 13.8 x5.3 7.2 x2.8 7.1 x2.7 cp311-cp311-win_arm64 2.4 2.4 x1.0 2.4 x1.0 2.4 x1.0 cp312-cp312-macosx_10_13_x86_64 3.2 11.4 x3.5 6.3 x1.9 6.5 x2.0 cp312-cp312-macosx_11_0_arm64 3.1 8.9 x2.9 5.2 x1.7 5.7 x1.8 cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64 4.3 17.1 x3.9 8.1 x1.9 7.9 x1.8 cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64 4.4 18.9 x4.3 8.6 x2.0 8.1 x1.8 cp312-cp312-manylinux_2_28_aarch64 4.4 17.6 x4.0 6.8 x1.6 8.3 x1.9 cp312-cp312-manylinux_2_28_x86_64 4.5 19.6 x4.4 7.5 x1.7 8.4 x1.9 cp312-cp312-musllinux_1_2_aarch64 4.3 18.0 x4.2 7.0 x1.6 8.5 x2.0 cp312-cp312-musllinux_1_2_x86_64 4.4 19.8 x4.5 7.6 x1.7 8.5 x1.9 cp312-cp312-win32 2.3 2.3 x1.0 2.3 x1.0 2.3 x1.0 cp312-cp312-win_amd64 2.6 13.8 x5.3 7.2 x2.8 7.1 x2.7 cp312-cp312-win_arm64 2.4 2.4 x1.0 2.4 x1.0 2.4 x1.0 cp313-cp313-macosx_10_13_x86_64 3.2 11.4 x3.5 6.3 x1.9 6.5 x2.0 cp313-cp313-macosx_11_0_arm64 3.1 8.9 x2.9 5.2 x1.7 5.7 x1.8 cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64 4.3 17.1 x3.9 8.1 x1.9 7.9 x1.8 cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64 4.4 18.9 x4.3 8.6 x2.0 8.1 x1.8 cp313-cp313-manylinux_2_28_aarch64 4.4 17.6 x4.0 6.8 x1.6 8.3 x1.9 cp313-cp313-manylinux_2_28_x86_64 4.5 19.6 x4.4 7.5 x1.7 8.4 x1.9 cp313-cp313-musllinux_1_2_aarch64 4.3 18.0 x4.2 7.0 x1.6 8.5 x2.0 cp313-cp313-musllinux_1_2_x86_64 4.4 19.8 x4.5 7.6 x1.7 8.5 x1.9 cp313-cp313-win32 2.3 2.3 x1.0 2.3 x1.0 2.3 x1.0 cp313-cp313-win_amd64 2.6 13.8 x5.3 7.2 x2.8 7.1 x2.7 cp313-cp313-win_arm64 2.4 2.4 x1.0 2.4 x1.0 2.4 x1.0 cp313-cp313t-macosx_10_13_x86_64 3.2 11.4 x3.5 6.3 x1.9 6.5 x2.0 cp313-cp313t-macosx_11_0_arm64 3.1 8.9 x2.9 5.2 x1.7 5.7 x1.8 cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64 17.1 8.1 8.0 cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64 4.4 19.0 x4.3 8.7 x2.0 8.1 x1.8 cp313-cp313t-manylinux_2_28_aarch64 17.7 6.8 8.3 cp313-cp313t-manylinux_2_28_x86_64 4.5 19.7 x4.3 7.5 x1.7 8.5 x1.9 cp313-cp313t-musllinux_1_2_aarch64 18.1 7.0 8.6 cp313-cp313t-musllinux_1_2_x86_64 4.5 19.8 x4.4 7.6 x1.7 8.5 x1.9 cp313-cp313t-win32 2.3 2.3 x1.0 2.3 x1.0 2.3 x1.0 cp313-cp313t-win_amd64 2.6 13.8 x5.2 7.2 x2.8 7.1 x2.7 cp313-cp313t-win_arm64 2.4 2.4 x1.0 2.4 x1.0 2.4 x1.0 cp39-cp39-macosx_10_10_x86_64 3.2 11.4 x3.5 6.3 x2.0 6.4 x2.0 cp39-cp39-macosx_11_0_arm64 3.1 8.9 x2.9 5.2 x1.7 5.7 x1.8 cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64 4.3 17.1 x4.0 8.1 x1.9 7.9 x1.8 cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64 4.4 18.9 x4.3 8.6 x2.0 8.1 x1.8 cp39-cp39-manylinux_2_28_aarch64 4.4 17.6 x4.0 6.8 x1.6 8.3 x1.9 cp39-cp39-manylinux_2_28_x86_64 4.5 19.6 x4.4 7.5 x1.7 8.4 x1.9 cp39-cp39-musllinux_1_2_aarch64 4.3 18.0 x4.2 6.9 x1.6 8.5 x2.0 cp39-cp39-musllinux_1_2_x86_64 4.4 19.8 x4.5 7.6 x1.7 8.5 x1.9 cp39-cp39-win32 2.3 2.3 x1.0 2.3 x1.0 2.3 x1.0 cp39-cp39-win_amd64 2.6 13.8 x5.3 7.2 x2.8 7.1 x2.7 cp39-cp39-win_arm64 2.4 2.4 x1.0 2.4 x1.0 2.4 x1.0 pp310-pypy310_pp73-macosx_10_15_x86_64 3.2 11.4 x3.6 6.3 x2.0 6.5 x2.0 pp310-pypy310_pp73-macosx_11_0_arm64 3.1 8.9 x2.9 5.2 x1.7 5.7 x1.8 pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64 3.4 16.1 x4.7 7.1 x2.1 6.9 x2.0 pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64 3.4 17.8 x5.2 7.6 x2.2 7.0 x2.0 pp310-pypy310_pp73-manylinux_2_28_aarch64 3.4 16.6 x4.8 5.8 x1.7 7.3 x2.1 pp310-pypy310_pp73-manylinux_2_28_x86_64 3.5 18.5 x5.3 6.5 x1.8 7.3 x2.1 pp310-pypy310_pp73-win_amd64 2.6 13.8 x5.3 7.2 x2.8 7.1 x2.7 pp311-pypy311_pp73-macosx_10_15_x86_64 11.4 6.3 6.5 pp311-pypy311_pp73-macosx_11_0_arm64 8.9 5.2 5.7 pp311-pypy311_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64 16.1 7.1 6.9 pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64 17.8 7.6 7.0 pp311-pypy311_pp73-manylinux_2_28_aarch64 16.6 5.8 7.3 pp311-pypy311_pp73-manylinux_2_28_x86_64 18.5 6.5 7.3 pp311-pypy311_pp73-win_amd64 13.8 7.2 7.1 total 249.3 1,055 x4.4 505.7 x2.0 538.1 x2.2 average 3.5 13.7 6.2 6.6 Note 11.1.0 had 71 files compared to 81 for 11.2+, so a small part of the increase is down to extra wheels.
The initial 11.2.0 was 4.4 times bigger than 11.1.0.
These new PRs are 2.0 times (aomenc and dav1d) and 2.2 times bigger (aom), much more reasonable.
I think aom is a no-brainer for the encoder, but dav1d is basically the industry standard decoder. It is the decoder used by every major browser: Edge, Safari, Chrome, and Firefox.
And @wantehchang said:
We should only consider removing
dav1das a last resort. Although the decoder in libaom is the reference implementation and we test it with open-source and commercial AV1 test vectors and fuzzing regularly at Google, dav1d is likely to have better software security because it is being used in far more applications, including some that perform additional fuzzing of dav1d.From this, #8858 (aomenc and dav1d codecs) sound like a better choice, plus the files are a bit smaller.
Reacted by Jeffrey 'Alex' Clark and Michal PortešReacted by Jeffrey 'Alex' ClarkHi. Have you considered using the Limited API to drastically reduce the number of wheels? Also, you won't need to rebuild wheels for every Python release.
I've noticed that packages don't usually use the Limited API. Is there a reason for this?This has burst the bubble on already constrained image sizes for serverless images - and a great many serverless functions use pillow.
I'd love to see support for this migrated to an extras and available through standard package tool discovery.
@PalmtopTiger @sspencer-arine Since we're late to deliver 11.2 in #8722 (in accordance with our own release schedule), those suggestions are more likely to be implemented in 11.3 (or 12?)
I'm not familiar with the Limited API … and maybe we could do
extras… but right now I think we're just trying to get the release out the door with wheels sizes that everyone agrees are within some acceptable range.Reacted by Andrey Efremov@aclark4life thank you - but is there guidance on how to omit AVIF and any other things that came in since 11.1 if we did a local build and distributed that internally as a custom 11.2 wheel?
Reacted by Jeffrey 'Alex' Clarkis there guidance on how to omit AVIF and any other things that came in since 11.1 if we did a local build and distributed that internally as a custom 11.2 wheel?
@sspencer-arine you don't really need to build it from source -- it is sufficient to remove avif support from the official whl by deleting
PIL/_avif.*$ ls -al pillow-11.2.0-cp312-cp312-win_amd64.whl -rw-r--r--. ed ed 13807352 pillow-11.2.0-cp312-cp312-win_amd64.whl $ zip -d pillow-11.2.0-cp312-cp312-win_amd64.whl PIL/_avif.pyi PIL/_avif.cp312-win_amd64.pyd deleting: PIL/_avif.cp312-win_amd64.pyd deleting: PIL/_avif.pyi $ ls -al pillow-11.*-cp312-cp312-win_amd64.whl -rw-r--r--. ed ed 2680119 pillow-11.2.0-cp312-cp312-win_amd64.whl -rw-r--r--. ed ed 2626369 pillow-11.1.0-cp312-cp312-win_amd64.whlthis is not official advice, I'm just another user :-)
Reacted by Jeffrey 'Alex' Clarkis there guidance on how to omit AVIF and any other things that came in since 11.1 if we did a local build and distributed that internally as a custom 11.2 wheel?
If you're building from source, then by default Pillow won't include libavif if it's not present in your environment. If it is present, or you don't know and would like to ensure that it's not included, you can disable it using a build option -
-C avif=disableReacted by Jeffrey 'Alex' ClarkThose who were concerned about wheel sizes will be glad to know that Pillow 11.2.1 has now been released without libavif included in the wheels.
Reacted by Michael Kushinski, Petr Přikryl and Jeffrey 'Alex' ClarkHey, are there any plans/ideas how to make it smaller and ship it without need of extra building on project side?
@petrprikryl I suspect we'll have more decisions made before the next release but ideas welcome!
@petrprikryl #8858 has every trick I've been able to come up with to reduce the binary size. On most platforms the increased uncompressed size is between 3 and 5 MB. I doubt it can be made much smaller—after some optimizations I made yesterday I am out of ideas—so the decision will probably center around whether that is a size to allow its inclusion in the distributed wheels.
Reacted by doublexReacted by doublexReacted by Jeffrey 'Alex' Clark, doublex and Petr PřikrylReacted by doublex and Petr PřikrylThank you for all your work on #8858, it's much appreciated!
With a quick calculation, the total size of all the wheels is up from ~260 MB to ~469 MB, about 1.8 times bigger. That's much better than the ~4 times we had originally and I think it's an acceptable increase.
Reacted by Jeffrey 'Alex' Clark and doublexReacted by Jeffrey 'Alex' Clark and doublexReacted by doublex and Storm HegReacted by doublex, Petr Přikryl, Storm Heg and Thibaud ColasPillow 11.3.0 has now been released with AVIF support in the wheels.
Reacted by Hugo van Kemenade and Wan-Teh Chang
The AVIF .so has made the wheels about 4 times bigger: #8722 (comment)
Is there anything we can do to make it smaller?