Repository navigation
Use zstd compression #478
Description
- A zstd patch is available for squashfuse at Add zstd compression support vasi/squashfuse#23
- Zstd support has been added to SquashFS in the kernel in commit torvalds/linux@87bf54b.
- Support is ready for mksquashfs and is coming to squashfs-tools soon https://sourceforge.net/p/squashfs/mailman/message/35996905/.
Activity
It would be amazing, because electron-builder uses lzma for windows nsis targets — expected size 35 MB. AppImage uses deflate and size ~51 MB (even more — because 7za is not used, compression is not good (e.g. 7za can compress the same data to 50 MB (1.5 MB difference))).
Yes, we use lzma2 for windows portable and users don't complain that start time is slow, but anyway, for now I decided to not use xz for AppImage because of decompression speed.
Will AppImages using zstd work on older kernels, such as the ones found in Ubuntu LTS, CentOS or Debian?
If we would implement zstd, then we would implement it in FUSE, which means it would be independent of the kernel. But we would lose the ability to loop-mount such AppImages using old kernels (is anyone ever doing that, anyway?)
@develar if it is not too much trouble, could you give the following numbers please?
- Windows uncompressed __ MB, Windows compressed with lzma2 __ MB = __% compression ratio
- Linux uncompressed __ MB, Linux compressed AppImage (without xz) __ MB = __% compression ratio
But we would lose the ability to loop-mount such AppImages using the kernel
We wouldn't lose it, but it won't work on old systems, right? I assume that on newer systems, it should work out of the box, but on older ones, it'd produce an error message.
Correct. Sorry, I meant "old kernels". Corrected above.
Zstandard support has been added to squashfs-tools, so if I am not mistaken nothing is stopping us now from using Zstandard for AppImages.
Volunteers for a PR?
sudo apt install make gcczstd library
git clone https://github.com/facebook/zstd/ cd zstd sudo make install cd ..squashfs-tools
git clone https://github.com/plougher/squashfs-tools cd squashfs-tools/squashfs-tools make -j8 ZSTD_SUPPORT=1 GZIP_SUPPORT=0 COMP_DEFAULT=zstd mksquashfs cd ../..squashfuse
git clone https://github.com/vasi/squashfuse ... --with-zstd=PREFIXReacted by Hugo Locurcio and Aviohttps://github.com/vasi/squashfuse/releases/tag/0.1.101 now contains zstd support.
We should update it in a PR branch, and check whether it builds with that new version.
Any updates on this issue? I think by this point, zstd support is pretty universal
35 remaining items
The benchmark script was written by AI, so no idea how terrible it might be, but it has worked well so far haha.
The main issue this has is its
0.1sleep which means that the numbers you show are only reliable to a between -0.2 to 0 seconds. The 2nd decimal digit has no reliable value and the first decimal digit is so-so.As long as you stay in the realm of the
0.5s, it's okReacted by Samuel@Samueru-sama @probonopd the
mksquashfsdefault iszstd:15. Nothing to do withappimagetool.Luckily it's simple enough to change the level, by passing
appimagetoolthe--mksquashfs-optflag twice (becausemksquashfsdoesn't take-Xcompression-level=N— it insists on two separate CLI args; hey, what did you expect from a tool that seems bent on having inconvenient usage?):appimagetool --comp zstd --mksquashfs-opt -Xcompression-level --mksquashfs-opt 3 ...Reacted by Samuel@probonopd if you want to implement different compression levels, I'd suggest accepting "zstd:N" (just as
mount.btrfsaccepts). There are tools that callappimagetoolinternally (linuxdeploy), and an extra level of CLI arg wrapping gets a bit insane.Well, ideally we find the optimal ("balanced") settings and set them for everyone. So that we make the choice, and not every application developer has to think about which compression level to use.
Well, ideally we find the optimal ("balanced") settings and set them for everyone. So that we make the choice, and not every application developer has to think about which compression level to use.
So in essence you're suggesting making it worse than it already is (because right now I can set all the knobs, including block size, to what I know is good for my specific use case). It is a trend I find in a lot of software nowadays, let's not name names.
Have your mythical "balanced" setting, but don't make me break open and re-create the appimage if your compression level, squashfs block size etc don't suit me.
Frankly I doubt there is an optimal level. It depends on the contents, but also on the purpose and nature of the appimage. The app can start fast or slow independently from the unpacked size (maybe it takes ten seconds to scan the system or connect to a server before it's even usable). The developer might favor a small size for easy downloads, or little compression for a fast startup, and you wouldn't know. Etcetera.
Researching the "balanced" setting would be a full time job of surveying your "customers" (appimage developers), and still half of them, by definition won't be pleased with the default. It's your call, but maybe there are more useful things to implement (if anything new gets implemented at this point).
If you ask me the optimal level is zstd:1, it gives similar comp ratios to gzip while being much faster at decompression.
Something similar also happens with zstd is used on Btrfs for filesystem level compression, it is only zstd:1 that makes sense, and sometimes zstd:2 is better under some workloads but the difference is less than 2%.
@Samueru-sama If you're going to go that low, you might as well use lz4_hc, which also has compression similar to gzip but is about twice as fast as zstd and has a much smaller space overhead in the runtime.
zstd shines at mid-high compression levels, so "optimal" is probably in the range of 12-17 (15 is the default level for mksquashfs). So you'd get better compression than gzip and still get substantially faster decompression
Developers should definitely be able to choose something else if it provides a better experience for their application though
@Samueru-sama If you're going to go that low, you might as well use lz4_hc, which also has compression similar to gzip but is about twice as fast as zstd and has a much smaller space overhead in the runtime.
zstd shines at mid-high compression levels, so "optimal" is probably in the range of 12-17 (15 is the default level for mksquashfs). So you'd get better compression than gzip and still get substantially faster decompression
Developers should definitely be able to choose something else if it provides a better experience for their application though
How can I benchmark lz4_hc? I only found a github issue where the user was told to use lz4:9 instead.
Either way I just did a quick test using the librewolf AppDIr as a compressed tar file:
~/ time unzstd Librewolf-zstd-15.tar.zst Librewolf-zstd-15.tar.zst: 349124608 bytes unzstd Librewolf-zstd-15.tar.zst 0.55s user 0.25s system 152% cpu 0.529 total ~/ time unzstd Librewolf-zstd-12.tar.zst Librewolf-zstd-12.tar.zst: 349124608 bytes unzstd Librewolf-zstd-12.tar.zst 0.53s user 0.29s system 148% cpu 0.551 total ~/ time unzstd Librewolf-zstd-1.tar.zst Librewolf-zstd-1.tar.zst: 349124608 bytes unzstd Librewolf-zstd-1.tar.zst 0.50s user 0.24s system 150% cpu 0.491 total ~/ time unlz4 Librewolf-lz4-9.tar.lz4 Decoding file Librewolf-lz4-9.tar Librewolf-lz4-9.tar. : decoded 349124608 bytes unlz4 Librewolf-lz4-9.tar.lz4 0.36s user 0.18s system 99% cpu 0.538 totalAnd the sizes of the files were 86.5MiB for zstd:15, 87.7MiB for zstd:12, 116.7MiB for zstd:1 and 124.8MiB for lz4:9.
Which yeah these benchmarks give the idea that zstd:12-15 is good, but as seen in the tests above when actually opening the appimage the default zstd compression of mksquashfs was adding a considerable delay, even slower than gzip.
If you measure sub-second timings (which have a considerable variance as I can see from the various timings you quoted) use sharkdp/hyperfine. E.g.
hyperfine -r 10 'cmd1 ...' 'cmd2 ...'. Otherwise you're measuring noise.@Samueru-sama What I do is make SquashFS images of various levels, then test reading and other simple file operations. FUSE also caches files, so you get a significant speed-up after the first read. I've never had any issues with lz4_hc, what was the issue you saw about?
As for the AppImages, I'm not sure why higher zstd ratios would be so much slower, that shouldn't be the case because using squashfuse I didn't seem to notice any massive slow-downs, certainly not any that would make it slower than gzip.
When benchmarking SquashFS compression, tar files should be avoided, they give a good indication as to how fast the decompression itself is as a whole, but speed and file size might be different when compressed into blocks, like in SquashFS. Use mksquashfs then mount with squashfuse and test that instead
@Samueru-sama What I do is make SquashFS images of various levels, then test reading and other simple file operations. FUSE also caches files, so you get a significant speed-up after the first read. I've never had any issues with lz4_hc, what was the issue you saw about?
As for the AppImages, I'm not sure why higher zstd ratios would be so much slower, that shouldn't be the case because using squashfuse I didn't seem to notice any massive slow-downs, certainly not any that would make it slower than gzip.
When benchmarking SquashFS compression, tar files should be avoided, they give a good indication as to how fast the decompression itself is as a whole, but speed and file size might be different when compressed into blocks, like in SquashFS. Use mksquashfs then mount with squashfuse and test that instead
I haven't had issues with lz4_hc (because I've never used it lol), just that I asked you how to benchmark it because when I searched for it the info I got is that I should use lz4:9 instead. I asked you because you said that instead of zstd:1 one should use lz4_hc.
I just repeated my tests of zstd (defaults which seems to be zstd 15) vs gzip (and modified the script a little bit) with the librewolf appimage:
(Once again I just did here was --appimage-extract the librewolf appimage and made a new zstd appimage using appimagetool using its defaults, the title of the appimage indicates the type in the results)
~/ ./benchmark-startup.sh original-LibreWolf.x86_64.AppImage Time taken: 2.32 seconds ~/ ./benchmark-startup.sh original-LibreWolf.x86_64.AppImage Time taken: 2.15 seconds ~/ ./benchmark-startup.sh original-LibreWolf.x86_64.AppImage Time taken: 1.99 seconds ~/ ./benchmark-startup.sh original-LibreWolf.x86_64.AppImage Time taken: 1.92 seconds ~/ ./benchmark-startup.sh original-LibreWolf.x86_64.AppImage Time taken: 2.04 seconds ~/ ./benchmark-startup.sh zstd-LibreWolf.x86_64.AppImage Time taken: 2.78 seconds ~/ ./benchmark-startup.sh zstd-LibreWolf.x86_64.AppImage Time taken: 3.00 seconds ~/ ./benchmark-startup.sh zstd-LibreWolf.x86_64.AppImage Time taken: 2.79 seconds ~/ ./benchmark-startup.sh zstd-LibreWolf.x86_64.AppImage Time taken: 2.73 seconds ~/ ./benchmark-startup.sh zstd-LibreWolf.x86_64.AppImage Time taken: 2.73 secondsAnd just for the heck of it, I decided to make again the zstd1 level one appimage using the method that @mralusw suggested:
appimagetool --comp zstd --mksquashfs-opt -Xcompression-level --mksquashfs-opt 3 ./LibreWolf.AppDirAnd it did make a zstd:1 appimages I think, because this appimage is 117.6 MiB, however when I tested it wasn't faster than the original appimage (gzip), which is odd because in the older tests zstd1 was significantly faster than the others:
~/ ./benchmark-startup.sh zstd:1-LibreWolf.x86_64.AppImage Time taken: 2.18 seconds ~/ ./benchmark-startup.sh zstd:1-LibreWolf.x86_64.AppImage Time taken: 2.09 seconds ~/ ./benchmark-startup.sh zstd:1-LibreWolf.x86_64.AppImage Time taken: 1.99 seconds ~/ ./benchmark-startup.sh zstd:1-LibreWolf.x86_64.AppImage Time taken: 2.10 seconds ~/ ./benchmark-startup.sh zstd:1-LibreWolf.x86_64.AppImage Time taken: 2.06 secondsSo I went and made the appimage again manually using
mksquashfs ./LibreWolf.AppDir Manual-zstd-1 -comp zstd -Xcompression-level 1andcat Manual-zstd-1 >> runtime-x86_64This zstd:1 appimage is 122.5 MiB (No idea why there's a size difference) however it does start up quite fast like it should:
~/ ./benchmark-startup.sh manual-zstd:1-LibreWolf.x86_64.AppImage Time taken: 1.50 seconds ~/ ./benchmark-startup.sh manual-zstd:1-LibreWolf.x86_64.AppImage Time taken: 1.49 seconds ~/ ./benchmark-startup.sh manual-zstd:1-LibreWolf.x86_64.AppImage Time taken: 1.51 seconds ~/ ./benchmark-startup.sh manual-zstd:1-LibreWolf.x86_64.AppImage Time taken: 1.50 seconds ~/ ./benchmark-startup.sh manual-zstd:1-LibreWolf.x86_64.AppImage Time taken: 1.48 seconds ~/ ./benchmark-startup.sh manual-zstd:1-LibreWolf.x86_64.AppImage Time taken: 1.51 secondsSo hmmm yeah, there's something other than the zstd compression level at play here. This is the updated script I'm using, for it to work you would need to do this test using i3wm:
#!/bin/bash # Get the directory of the script DIR="$(dirname "$(readlink -f "${0}")")" # Start the application "$DIR/"$@"" & # Get the start time in nanoseconds start_time=$(date +%s%N) # Loop until the window class is 'YourAppImage' while true; do window_class=$(i3-msg -t get_tree | jq -r ".. | select(.focused? == true) | .window_properties.class") if [[ "$window_class" == "LibreWolf" ]]; then break fi # Sleep for a bit to prevent high CPU usage sleep 0.01 done # Get the end time in nanoseconds end_time=$(date +%s%N) # Calculate the time taken and convert to seconds time_taken=$(echo "scale=2; ($end_time - $start_time) / 1000000000" | bc) # Output the time it took to run killall librewolf echo "Time taken: $time_taken seconds"And if you want the appimages I made for these tests, just let know and I'll give you link.
Edit: Also if you wonder, here is the startup time of a native librewolf (no appimage):
~/ ./benchmark-startup.sh librewolf Time taken: .94 seconds ~/ ./benchmark-startup.sh librewolf Time taken: .91 seconds ~/ ./benchmark-startup.sh librewolf Time taken: .88 seconds ~/ ./benchmark-startup.sh librewolf Time taken: .93 seconds ~/ ./benchmark-startup.sh librewolf Time taken: .89 seconds ~/ ./benchmark-startup.sh librewolf Time taken: .90 secondsAnd just for the heck of it, I decided to make again the zstd1 level one appimage using the method that @mralusw suggested:
appimagetool --comp zstd --mksquashfs-opt -Xcompression-level --mksquashfs-opt 3 ./LibreWolf.AppDirThat makes a zstd:3, not a zstd:1. Look at the args. Every appimagetool
--mksquashfsarg passes the next arg tomksquashfsAnd just for the heck of it, I decided to make again the zstd1 level one appimage using the method that @mralusw suggested:
appimagetool --comp zstd --mksquashfs-opt -Xcompression-level --mksquashfs-opt 3 ./LibreWolf.AppDirThat makes a zstd:3, not a zstd:1. Look at the args. Every appimagetool
--mksquashfsarg passes the next arg tomksquashfsOh my bad, I did actually change the
--mksquashfs-opt 3to 1 in my tests, just that I pasted it here I forgot to change the number. Just checked my zsh history and it does have the1in there.And just to be extra sure I did make it again this time making sure that it is
--mksquashfs-opt 1and the resulting size is 117.6 MiB like I said before, changing it to 3 makes it 104.8 MiB instead.Apparently zstandard already has a successor: OpenZL. It needs to be checked out.
