Repository navigation
Upstream AppImage patches for squashfuse and squashfs-tools #815
Description
We should try to send the AppImage patches to https://github.com/vasi/squashfuse and https://github.com/plougher/squashfs-tools to be able to drop these patches and not have to build and ship our own libraries. This would also make packaging for distros easier.
Patches to send:
- https://github.com/AppImage/AppImageKit/blob/appimagetool/master/src/mksquashfs-mkfs-fixed-timestamp.patch
- complex squashfuse patches: https://github.com/AppImage/AppImageKit/blob/appimagetool/master/src/patch-squashfuse.sh.in
The squashfuse patches are pretty complicated, and we'll see how this will turn out.
Activity
TheAssassin commentedon Jul 12, 2018
First patch sent to upstream: plougher/squashfs-tools#51
TheAssassin commentedon Jul 12, 2018
I had an idea that'd allow us to get rid of the squashfuse patches entirely. We just build squashfuse as a shared library, and bundle it. Then, appimagetool just
dlopen()s this bundled library. If that doesn't succeed, we can probably assume FUSE support is not available, libfuse2 is missing, etc. (perhaps there's even some kind oferrnowe can check).I'll try that before trying to get our patches into squashfuse. If the idea doesn't work, I'd first try to find out whether a lib can find out it's
dlopen()ed, and then adjust the patches in a way to be able to move the error handling (i.e., the error message) into AppImageKit's code. Ideally, the squashfuse people just accept our patch, and usedlopen()by default then.probonopd commentedon Jul 12, 2018
Sounds like additional work and a source of additional complexity and bugs in an area that is already working for us as of today. I don't see messing with this as a high priority. You know my mantra - never touch a running system. While in theory upstreaming the patches would be nice, in practice I'd not invest time into it, at least not until appimaged still copies in the desktop files all the time, appimageupdatetool can't use GitLab, etc - things that are really bothering us.
TheAssassin commentedon Jul 12, 2018
@probonopd I really want to get rid of those patches for multiple reasons. Most prominently, it'd allow us to let users link to prebuilt libraries. That's also a core requirement for distro packaging.
probonopd commentedon Jul 12, 2018
I see, especially Debian packaging is of course a very worthy goal. (I'm just a bit clueless how this will work in practice - since we will want to use Debian oldstable, but the latest and greatest AppImage tools on it.)
TheAssassin commentedon Jul 12, 2018
@probonopd we can still recommend our users to use the most recent versions of AppImageKit. However, stuff like libappimage should be available in the distro repos.
probonopd commentedon Jul 12, 2018
My fear is that it will be ever-outdated. I don't feel like answering questions from users having a two year old
*appimage*... I fear we might end up in the same boat as the Krita lead developer has requested Debian cease distributing his software, if we don't find a good way to handle this (e.g., by providing backports of the latest*appimage*to all still-suported versions of Debian, without having it in the normal (non-backports) repo. Don't know if this is even allowable as per Debian policy.TheAssassin commentedon Jul 12, 2018
Well, that doesn't mean we should not remove annoyances that also prevent packaging. The point is, some software (like AppImageLauncher) is meant to be included into distros, but it can't because it depends on libappimage. We could at least allow building libappimage.
Also, even Debian announced a rolling release distro now, based on sid. They're making progress in providing a more recent distro for desktop users.
probonopd commentedon Jul 12, 2018
Agree
TheAssassin commentedon Jul 13, 2018
We might be able to generate a wrapper library using https://github.com/yugr/Implib.so that implicitly loads the squashfuse libraries on first call.
Thinking about it, however, we use the squashfs functionality a lot for non-FUSE operations as well, and if we'd rely on the fact that when
dlopening squashfuse fails means FUSE is not available, we'd need to separate all the other squashfs related calls. Therefore I think it'll be best to implementdlopenbased solution right in squashfuse, in a way that allows for some error handling (i.e., some sort of struct that keeps the references to the loaded fuse library that can be managed by users of squashfuse).probonopd commentedon Jul 13, 2018
I fear we are complicating it too much here - while making it easier for distributions to package our software is a noble goal, it is not one that we should sacrifice simplicity of how our software works for.
TheAssassin commentedon Jul 13, 2018
I actually propose to move this out of our maintenance scope, and have squashfuse worry about it instead of us.
probonopd commentedon Jul 13, 2018
As long as the moving doesn't create work for us. This issue is again an example of where I think things are already working well enough that I don't see it's a good investment of time.
For me personally, I am not willing to invest time to change anything so that it works in the way Debian thinks it should work. Because in the end, this just creates work for me, and as the result, we will have outdated and unmaintained versions of our software in Debian, which will only create additional work, and we will have to tell users "use the continous builds from this website instead, because everything else is bit-rotten".
Let's focus on things we don't have yet, or things that are not working yet.
TheAssassin commentedon Jul 13, 2018
The idea is to invest some time now and move the dlopen stuff into squashfuse's codebase, given they accept it. Then, any recent squashfuse binary will work for libappimage. That's a big plus IMO.
probonopd commentedon Jul 15, 2018
I see. Let's ask the squashfuse team then what they think about it.
TheAssassin commentedon Jul 15, 2018
@probonopd that's the plan, but it's scheduled for tomorrow.
azubieta commentedon Nov 27, 2018
It seems that the squashfuse-tools project where the patch was purposed is not quite active. Also I found in the tread a reference to use another project instead, see https://reproducible-builds.org/docs/system-images/#squashfs-metadata--compression
TheAssassin commentedon Nov 27, 2018
@azubieta are you referring to squashfuse or squashfs-tools (that's two separate projects)?
https://github.com/squashfskit/squashfskit is the reference you mean, I guess. Doesn't seem much more active either, I'd say.
bignaux commentedon May 1, 2019
For what it worth, when i was contributing to appimage, i sent 2 patchs on squashfuse and they were merged quickly ( https://github.com/vasi/squashfuse/pulls?q=is%3Apr+author%3Abignaux+is%3Aclosed ) .
TheAssassin commentedon May 1, 2019
We'll probably be settling on squashfskit in the future anyway, they already merged the required patches.
TheAssassin commentedon Aug 14, 2019
squashfs-tools now have their own reproducibility stuff, my PR was closed. See plougher/squashfs-tools#51 (comment).
probonopd commentedon Aug 15, 2019
Well... that project seems to have its own set of issues... squashfskit/squashfskit#4
TheAssassin commentedon Aug 16, 2019
License wise that's fine. An author cannot insist on a specific way of attribution. They just need to put it somewhere else.
But after all that's all off topic. Seems like squashfs-tools have come up with something satisfying. So we should move on to the latest version at some point and change our code to use these new variables they set up.