Use zstd compression

A zstd patch is available for squashfuse at Add zstd compression support vasi/squashfuse#23
Zstd support has been added to SquashFS in the kernel in commit torvalds/linux@87bf54b.
Support is ready for mksquashfs and is coming to squashfs-tools soon https://sourceforge.net/p/squashfs/mailman/message/35996905/.

Add zstd compression and decompression support to SquashFS. zstd is a
great fit for SquashFS because it can compress at ratios approaching xz,
while decompressing twice as fast as zlib. For SquashFS in particular,
it can decompress as fast as lzo and lz4. It also has the flexibility
to turn down the compression ratio for faster compression times.

The compression benchmark is run on the file tree from the SquashFS archive
found in ubuntu-16.10-desktop-amd64.iso [1]. It uses `mksquashfs` with the
default block size (128 KB) and and various compression algorithms/levels.
xz and zstd are also benchmarked with 256 KB blocks. The decompression
benchmark times how long it takes to `tar` the file tree into `/dev/null`.
See the benchmark file in the upstream zstd source repository located under
`contrib/linux-kernel/squashfs-benchmark.sh` [2] for details.

I ran the benchmarks on a Ubuntu 14.04 VM with 2 cores and 4 GiB of RAM.
The VM is running on a MacBook Pro with a 3.1 GHz Intel Core i7 processor,
16 GB of RAM, and a SSD.

| Method         | Ratio | Compression MB/s | Decompression MB/s |
|----------------|-------|------------------|--------------------|
| gzip           |  2.92 |               15 |                128 |
| lzo            |  2.64 |              9.5 |                217 |
| lz4            |  2.12 |               94 |                218 |
| xz             |  3.43 |              5.5 |                 35 |
| xz 256 KB      |  3.53 |              5.4 |                 40 |
| zstd 1         |  2.71 |               96 |                210 |
| zstd 5         |  2.93 |               69 |                198 |
| zstd 10        |  3.01 |               41 |                225 |
| zstd 15        |  3.13 |             11.4 |                224 |
| zstd 16 256 KB |  3.24 |              8.1 |                210 |

This patch was written by Sean Purcell <[email protected]>, but I will be
taking over the submission process.

[1] http://releases.ubuntu.com/16.10/
[2] https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/squashfs-benchmark.sh

zstd source repository: https://github.com/facebook/zstd

Signed-off-by: Sean Purcell <[email protected]>
Signed-off-by: Nick Terrell <[email protected]>
Signed-off-by: Chris Mason <[email protected]>
Acked-by: Phillip Lougher <[email protected]>

It would be amazing, because electron-builder uses lzma for windows nsis targets — expected size 35 MB. AppImage uses deflate and size ~51 MB (even more — because 7za is not used, compression is not good (e.g. 7za can compress the same data to 50 MB (1.5 MB difference))).

Yes, we use lzma2 for windows portable and users don't complain that start time is slow, but anyway, for now I decided to not use xz for AppImage because of decompression speed.

Will AppImages using zstd work on older kernels, such as the ones found in Ubuntu LTS, CentOS or Debian?

If we would implement zstd, then we would implement it in FUSE, which means it would be independent of the kernel. But we would lose the ability to loop-mount such AppImages using old kernels (is anyone ever doing that, anyway?)

@develar

@develar if it is not too much trouble, could you give the following numbers please?

Windows uncompressed __ MB, Windows compressed with lzma2 __ MB = __% compression ratio
Linux uncompressed __ MB, Linux compressed AppImage (without xz) __ MB = __% compression ratio

But we would lose the ability to loop-mount such AppImages using the kernel

We wouldn't lose it, but it won't work on old systems, right? I assume that on newer systems, it should work out of the box, but on older ones, it'd produce an error message.

Correct. Sorry, I meant "old kernels". Corrected above.

Zstandard support has been added to squashfs-tools, so if I am not mistaken nothing is stopping us now from using Zstandard for AppImages.

Volunteers for a PR?

sudo apt install make gcc

zstd library

git clone https://github.com/facebook/zstd/
cd zstd
sudo make install
cd ..

squashfs-tools

git clone https://github.com/plougher/squashfs-tools
cd squashfs-tools/squashfs-tools
make -j8 ZSTD_SUPPORT=1 GZIP_SUPPORT=0 COMP_DEFAULT=zstd mksquashfs
cd ../..

squashfuse

git clone https://github.com/vasi/squashfuse
...
--with-zstd=PREFIX

https://github.com/vasi/squashfuse/releases/tag/0.1.101 now contains zstd support.

We should update it in a PR branch, and check whether it builds with that new version.

Any updates on this issue? I think by this point, zstd support is pretty universal

@ambyjkl

Hi @ambyjkl looks like no one has done any work on this. First step would be to scientifically prove that it actually provides some advantage over what we have in place now.

The benchmark script was written by AI, so no idea how terrible it might be, but it has worked well so far haha.

The main issue this has is its 0.1 sleep which means that the numbers you show are only reliable to a between -0.2 to 0 seconds. The 2nd decimal digit has no reliable value and the first decimal digit is so-so.

As long as you stay in the realm of the 0.5s, it's ok

@Samueru-sama

@Samueru-sama @probonopd the mksquashfs default is zstd:15. Nothing to do with appimagetool.

Luckily it's simple enough to change the level, by passing appimagetool the --mksquashfs-opt flag twice (because mksquashfs doesn't take -Xcompression-level=N — it insists on two separate CLI args; hey, what did you expect from a tool that seems bent on having inconvenient usage?):

appimagetool --comp zstd --mksquashfs-opt -Xcompression-level --mksquashfs-opt 3 ...

@probonopd

@probonopd if you want to implement different compression levels, I'd suggest accepting "zstd:N" (just as mount.btrfs accepts). There are tools that call appimagetool internally (linuxdeploy), and an extra level of CLI arg wrapping gets a bit insane.

Well, ideally we find the optimal ("balanced") settings and set them for everyone. So that we make the choice, and not every application developer has to think about which compression level to use.

Well, ideally we find the optimal ("balanced") settings and set them for everyone. So that we make the choice, and not every application developer has to think about which compression level to use.

So in essence you're suggesting making it worse than it already is (because right now I can set all the knobs, including block size, to what I know is good for my specific use case). It is a trend I find in a lot of software nowadays, let's not name names.

Have your mythical "balanced" setting, but don't make me break open and re-create the appimage if your compression level, squashfs block size etc don't suit me.

Frankly I doubt there is an optimal level. It depends on the contents, but also on the purpose and nature of the appimage. The app can start fast or slow independently from the unpacked size (maybe it takes ten seconds to scan the system or connect to a server before it's even usable). The developer might favor a small size for easy downloads, or little compression for a fast startup, and you wouldn't know. Etcetera.

Researching the "balanced" setting would be a full time job of surveying your "customers" (appimage developers), and still half of them, by definition won't be pleased with the default. It's your call, but maybe there are more useful things to implement (if anything new gets implemented at this point).

If you ask me the optimal level is zstd:1, it gives similar comp ratios to gzip while being much faster at decompression.

Something similar also happens with zstd is used on Btrfs for filesystem level compression, it is only zstd:1 that makes sense, and sometimes zstd:2 is better under some workloads but the difference is less than 2%.

@Samueru-sama

@Samueru-sama If you're going to go that low, you might as well use lz4_hc, which also has compression similar to gzip but is about twice as fast as zstd and has a much smaller space overhead in the runtime.

zstd shines at mid-high compression levels, so "optimal" is probably in the range of 12-17 (15 is the default level for mksquashfs). So you'd get better compression than gzip and still get substantially faster decompression

Developers should definitely be able to choose something else if it provides a better experience for their application though

@Samueru-sama

@Samueru-sama If you're going to go that low, you might as well use lz4_hc, which also has compression similar to gzip but is about twice as fast as zstd and has a much smaller space overhead in the runtime.

zstd shines at mid-high compression levels, so "optimal" is probably in the range of 12-17 (15 is the default level for mksquashfs). So you'd get better compression than gzip and still get substantially faster decompression

Developers should definitely be able to choose something else if it provides a better experience for their application though

How can I benchmark lz4_hc? I only found a github issue where the user was told to use lz4:9 instead.

Either way I just did a quick test using the librewolf AppDIr as a compressed tar file:

~/ time unzstd Librewolf-zstd-15.tar.zst
Librewolf-zstd-15.tar.zst: 349124608 bytes                                     
unzstd Librewolf-zstd-15.tar.zst  0.55s user 0.25s system 152% cpu 0.529 total

~/ time unzstd Librewolf-zstd-12.tar.zst
Librewolf-zstd-12.tar.zst: 349124608 bytes                                     
unzstd Librewolf-zstd-12.tar.zst  0.53s user 0.29s system 148% cpu 0.551 total

~/ time unzstd Librewolf-zstd-1.tar.zst 
Librewolf-zstd-1.tar.zst: 349124608 bytes                                      
unzstd Librewolf-zstd-1.tar.zst  0.50s user 0.24s system 150% cpu 0.491 total

~/ time unlz4 Librewolf-lz4-9.tar.lz4 
Decoding file Librewolf-lz4-9.tar 
Librewolf-lz4-9.tar. : decoded 349124608 bytes                                 
unlz4 Librewolf-lz4-9.tar.lz4  0.36s user 0.18s system 99% cpu 0.538 total

And the sizes of the files were 86.5MiB for zstd:15, 87.7MiB for zstd:12, 116.7MiB for zstd:1 and 124.8MiB for lz4:9.

Which yeah these benchmarks give the idea that zstd:12-15 is good, but as seen in the tests above when actually opening the appimage the default zstd compression of mksquashfs was adding a considerable delay, even slower than gzip.

If you measure sub-second timings (which have a considerable variance as I can see from the various timings you quoted) use sharkdp/hyperfine. E.g. hyperfine -r 10 'cmd1 ...' 'cmd2 ...'. Otherwise you're measuring noise.

@Samueru-sama

@Samueru-sama What I do is make SquashFS images of various levels, then test reading and other simple file operations. FUSE also caches files, so you get a significant speed-up after the first read. I've never had any issues with lz4_hc, what was the issue you saw about?

As for the AppImages, I'm not sure why higher zstd ratios would be so much slower, that shouldn't be the case because using squashfuse I didn't seem to notice any massive slow-downs, certainly not any that would make it slower than gzip.

When benchmarking SquashFS compression, tar files should be avoided, they give a good indication as to how fast the decompression itself is as a whole, but speed and file size might be different when compressed into blocks, like in SquashFS. Use mksquashfs then mount with squashfuse and test that instead

@Samueru-sama

@Samueru-sama What I do is make SquashFS images of various levels, then test reading and other simple file operations. FUSE also caches files, so you get a significant speed-up after the first read. I've never had any issues with lz4_hc, what was the issue you saw about?

As for the AppImages, I'm not sure why higher zstd ratios would be so much slower, that shouldn't be the case because using squashfuse I didn't seem to notice any massive slow-downs, certainly not any that would make it slower than gzip.

When benchmarking SquashFS compression, tar files should be avoided, they give a good indication as to how fast the decompression itself is as a whole, but speed and file size might be different when compressed into blocks, like in SquashFS. Use mksquashfs then mount with squashfuse and test that instead

I haven't had issues with lz4_hc (because I've never used it lol), just that I asked you how to benchmark it because when I searched for it the info I got is that I should use lz4:9 instead. I asked you because you said that instead of zstd:1 one should use lz4_hc.

I just repeated my tests of zstd (defaults which seems to be zstd 15) vs gzip (and modified the script a little bit) with the librewolf appimage:

(Once again I just did here was --appimage-extract the librewolf appimage and made a new zstd appimage using appimagetool using its defaults, the title of the appimage indicates the type in the results)

~/ ./benchmark-startup.sh original-LibreWolf.x86_64.AppImage
Time taken: 2.32 seconds
~/ ./benchmark-startup.sh original-LibreWolf.x86_64.AppImage
Time taken: 2.15 seconds
~/ ./benchmark-startup.sh original-LibreWolf.x86_64.AppImage
Time taken: 1.99 seconds
~/ ./benchmark-startup.sh original-LibreWolf.x86_64.AppImage
Time taken: 1.92 seconds
~/ ./benchmark-startup.sh original-LibreWolf.x86_64.AppImage
Time taken: 2.04 seconds
~/ ./benchmark-startup.sh zstd-LibreWolf.x86_64.AppImage
Time taken: 2.78 seconds
~/ ./benchmark-startup.sh zstd-LibreWolf.x86_64.AppImage
Time taken: 3.00 seconds
~/ ./benchmark-startup.sh zstd-LibreWolf.x86_64.AppImage
Time taken: 2.79 seconds
~/ ./benchmark-startup.sh zstd-LibreWolf.x86_64.AppImage
Time taken: 2.73 seconds
~/ ./benchmark-startup.sh zstd-LibreWolf.x86_64.AppImage
Time taken: 2.73 seconds

And just for the heck of it, I decided to make again the zstd1 level one appimage using the method that @mralusw suggested:

appimagetool --comp zstd --mksquashfs-opt -Xcompression-level --mksquashfs-opt 3 ./LibreWolf.AppDir

And it did make a zstd:1 appimages I think, because this appimage is 117.6 MiB, however when I tested it wasn't faster than the original appimage (gzip), which is odd because in the older tests zstd1 was significantly faster than the others:

~/ ./benchmark-startup.sh zstd:1-LibreWolf.x86_64.AppImage                     
Time taken: 2.18 seconds
~/ ./benchmark-startup.sh zstd:1-LibreWolf.x86_64.AppImage
Time taken: 2.09 seconds
~/ ./benchmark-startup.sh zstd:1-LibreWolf.x86_64.AppImage
Time taken: 1.99 seconds
~/ ./benchmark-startup.sh zstd:1-LibreWolf.x86_64.AppImage
Time taken: 2.10 seconds
~/ ./benchmark-startup.sh zstd:1-LibreWolf.x86_64.AppImage
Time taken: 2.06 seconds

So I went and made the appimage again manually using mksquashfs ./LibreWolf.AppDir Manual-zstd-1 -comp zstd -Xcompression-level 1 and cat Manual-zstd-1 >> runtime-x86_64

This zstd:1 appimage is 122.5 MiB (No idea why there's a size difference) however it does start up quite fast like it should:

~/ ./benchmark-startup.sh manual-zstd:1-LibreWolf.x86_64.AppImage
Time taken: 1.50 seconds
~/ ./benchmark-startup.sh manual-zstd:1-LibreWolf.x86_64.AppImage
Time taken: 1.49 seconds
~/ ./benchmark-startup.sh manual-zstd:1-LibreWolf.x86_64.AppImage
Time taken: 1.51 seconds
~/ ./benchmark-startup.sh manual-zstd:1-LibreWolf.x86_64.AppImage
Time taken: 1.50 seconds
~/ ./benchmark-startup.sh manual-zstd:1-LibreWolf.x86_64.AppImage
Time taken: 1.48 seconds
~/ ./benchmark-startup.sh manual-zstd:1-LibreWolf.x86_64.AppImage
Time taken: 1.51 seconds

So hmmm yeah, there's something other than the zstd compression level at play here. This is the updated script I'm using, for it to work you would need to do this test using i3wm:

#!/bin/bash

# Get the directory of the script
DIR="$(dirname "$(readlink -f "${0}")")"

# Start the application
"$DIR/"$@"" &

# Get the start time in nanoseconds
start_time=$(date +%s%N)

# Loop until the window class is 'YourAppImage'
while true; do
    window_class=$(i3-msg -t get_tree | jq -r ".. | select(.focused? == true) | .window_properties.class")
    if [[ "$window_class" == "LibreWolf" ]]; then
        break
    fi
    # Sleep for a bit to prevent high CPU usage
    sleep 0.01
done

# Get the end time in nanoseconds
end_time=$(date +%s%N)

# Calculate the time taken and convert to seconds
time_taken=$(echo "scale=2; ($end_time - $start_time) / 1000000000" | bc)

# Output the time it took to run
killall librewolf
echo "Time taken: $time_taken seconds"

And if you want the appimages I made for these tests, just let know and I'll give you link.

Edit: Also if you wonder, here is the startup time of a native librewolf (no appimage):

 ~/ ./benchmark-startup.sh librewolf                                     
Time taken: .94 seconds
~/ ./benchmark-startup.sh librewolf
Time taken: .91 seconds
~/ ./benchmark-startup.sh librewolf
Time taken: .88 seconds
~/ ./benchmark-startup.sh librewolf
Time taken: .93 seconds
~/ ./benchmark-startup.sh librewolf
Time taken: .89 seconds
~/ ./benchmark-startup.sh librewolf
Time taken: .90 seconds

@Samueru-sama

@Samueru-sama

And just for the heck of it, I decided to make again the zstd1 level one appimage using the method that @mralusw suggested:
appimagetool --comp zstd --mksquashfs-opt -Xcompression-level --mksquashfs-opt 3 ./LibreWolf.AppDir

That makes a zstd:3, not a zstd:1. Look at the args. Every appimagetool --mksquashfs arg passes the next arg to mksquashfs

@Samueru-sama

@Samueru-sama

And just for the heck of it, I decided to make again the zstd1 level one appimage using the method that @mralusw suggested:
appimagetool --comp zstd --mksquashfs-opt -Xcompression-level --mksquashfs-opt 3 ./LibreWolf.AppDir

That makes a zstd:3, not a zstd:1. Look at the args. Every appimagetool --mksquashfs arg passes the next arg to mksquashfs

Oh my bad, I did actually change the --mksquashfs-opt 3 to 1 in my tests, just that I pasted it here I forgot to change the number. Just checked my zsh history and it does have the 1 in there.

And just to be extra sure I did make it again this time making sure that it is --mksquashfs-opt 1 and the resulting size is 117.6 MiB like I said before, changing it to 3 makes it 104.8 MiB instead.

Apparently zstandard already has a successor: OpenZL. It needs to be checked out.

Navigation Menu

Repository navigation

Use zstd compression #478

zstd library

squashfs-tools

squashfuse

35 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Use zstd compression #478

Description

Activity

develar commented on Oct 7, 2017

Calinou commented on Oct 8, 2017

probonopd commented on Oct 9, 2017

probonopd commented on Oct 9, 2017

TheAssassin commented on Oct 9, 2017

probonopd commented on Oct 9, 2017

probonopd commented on Nov 14, 2017

probonopd commented on Dec 4, 2017

zstd library

squashfs-tools

squashfuse

probonopd commented on Dec 9, 2017

TheAssassin commented on Dec 9, 2017

ambyjkl commented on Apr 25, 2020

probonopd commented on Apr 25, 2020

35 remaining items

brunoais commented on Feb 14, 2024

mralusw commented on Apr 5, 2024

mralusw commented on Apr 5, 2024

probonopd commented on Apr 5, 2024

mralusw commented on Apr 5, 2024

mralusw commented on Apr 5, 2024

Samueru-sama commented on Apr 5, 2024

mgord9518 commented on Apr 5, 2024

Samueru-sama commented on Apr 6, 2024

mralusw commented on Apr 6, 2024

mgord9518 commented on Apr 6, 2024

Samueru-sama commented on Apr 6, 2024

mralusw commented on Apr 6, 2024

Samueru-sama commented on Apr 6, 2024

probonopd commented on Oct 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions