Consider using DwarFS

[probonopd]

https://github.com/mhx/dwarfs is making bold claims:

So in this comparison, mkdwarfs is more than 6 times faster than mksquashfs, both in terms of CPU time and wall clock time.
In terms of compression ratio, the DwarFS file system is more than 10 times smaller than the SquashFS file system.

It'd be interesting to do some real-world comparisons by recompressing existing AppImages in DwarFS.

Reference:

• https://github.com/Phantop/appdwarf

[probonopd]

Thanks for your tests @FryingPanBrock. Interesting!

It would be valuable to do very systematic testing using the various zstandard compression levels and block sizes vs. DwarfFS, and then consider app size, app launch speed, zsync efficiency for AppImageUpdate, and (not so important) filesystem creation time.

Definitely something to be looked into. Maybe these tests could be scripted, unfortunately I don't have the time to do so now.

[Samueru-sama]

There is now an appimage runtime that supports dwarfs written in rust: https://github.com/VHSgunzo/uruntime

Here are some tests using the latest LibreOffice-still.basic.AppImage as reference, which still uses the old runtime and likely gzip compression.

• libreoffice.original.AppImage being the original appimage

• libreoffice.squashfs.zstd22.AppImage being the appimage with the new static runtime, 128K block size and zstd22 compression.

• libreoffice.dwarfs.l7.AppImagemade running mkdwarfs -l7 which uses zstd22 by default as well iirc and then I just added to the runtime with cat ./dwarfs.image >> runtime

Here the initial benchmarks:

TLDR: dwarfs makes the smallest appimage at 221MiB, however it also takes longer to start, about 100ms higher than the original appimage and about 500ms slower than the appimage using the static runtime with zstd22 squashfs and 128K blocks.

It is very likely that the dwarfs settings can be changed to still make smaller images that startup faster though.

The biggest problem I see with the new runtime is that it is almost 6 MiB, which means it only comes useful for appimages bigger than 60 MiB I would say.

[Samueru-sama]

Update: I here are some tests using different mksquashfs parameters instead of the defaults:

mkdwarfs -f --set-owner 0 --set-group 0 --no-history --no-create-timestamp --header runtime \
-i squashfs-root -o libreoffice.dwarfs.optimized.AppImage --compression zstd:level=22 -S20 -B9

mkdwarfs can also add the runtime instead of using cat.

Now it is smaller than squashfs while having about the same startup speed.

---

Edit: More testing with the brave appimage, this time with -S22 -B16 in dwarfs:

TLDR: dwarfs is 9% smaller and 21% faster

[Samueru-sama]

I made 2 dwarfs appimages that use this runtime:

https://github.com/Samueru-sama/GIMP-AppImage/releases/tag/continuous
https://github.com/Samueru-sama/OBS-Studio-AppImage/releases/tag/continuous

The runtime is now fully compatible with the static type2-runtime, including updates with appimageupdatetool work perfectly.

Edit: Also these appimages bundle all the libs and the ld-*.so with sharun which makes them work on any linux system.

[probonopd]

Hi @Samueru-sama

Not entirely sure how exactly this userland-execve magic is working but I have to say it is working nicely on Windows 11 WSL2 with Debian 👍

[Samueru-sama]

Not entirely sure how exactly this userland-execve magic is working

sharun is just a binary that you symlink to inside the AppDir and based on the name of the symlink it will find the ld-*.so and call the binary in ./shared/bin while passing every single directory that contains libraries to --library-path when calling the ld-*.so. This means that when hardlinks get used /proc/self/exe is the name of the binary instead of ld-*.so, and since every single directory with libraries gets passed down with the --library-path flag there is no need to use patchelf to set rpaths.

It also sets several env variables on its own if it detects certain directories that need them in the AppDir.

Excellent. Your approach is even immune to the “missing GLIBC x.y.z” errors which plague so many AppImages created now, contrary to the philosophy of packaging Windows-style widely compatible software that works forever. Writing this from Ubuntu 20.04.6 LTS. Using dwarfs with your approach looks obviously better.

This approach is something that began with go-appimage as far as I know, however note that you don't need dwarfs in other to make appimages that work on any linux system, you can see some other appimages that I made that work on any linux system here, not all use sharun either.

[probonopd]

@Samueru-sama does your sharun approach work with applications that need GLX, like https://github.com/AlbrechtL/welle.io/?

Reference:

• Fatal: Could not initialize GLX probonopd/go-appimage#301

[Samueru-sama]

@Samueru-sama does your sharun approach work with applications that need GLX, like https://github.com/AlbrechtL/welle.io/?

Reference:

* [Fatal: Could not initialize GLX probonopd/go-appimage#301](https://github.com/probonopd/go-appimage/issues/301)

Yes, you can see it in the OBS Studio AppImage I bundled the gpu libs and set __EGL_VENDOR_LIBRARY_DIRS

The deploy script that sharun uses an script, named lib4bin has a mode that uses strace to get all the libraries that get dlopened and copies them to the AppDir.

I used it in the OBS AppImage because otherwise I was getting incompatible mesa errors when testing it on alpine linux.

The only downside is that this means the application has to launched in the CI, although it was as simple as using xvfb-run to simulate a working display.

[probonopd]

@Samueru-sama I am really becoming a fan of your approach. Do you think you could build welle.io on Alpine Linux and then bundle it with your approach? That would be a killer demo 💯

[Samueru-sama]

@Samueru-sama I am really becoming a fan of your approach. Do you think you could build welle.io on Alpine Linux and then bundle it with your approach? That would be a killer demo 💯

I used the following script on Artix linux, use the well-io aur package binaries instead of compiling it:

#!/bin/sh

set -eu

PACKAGE=welle.io
DESKTOP=welle-io.desktop
ICON=welle-io.png

export ARCH="$(uname -m)"
export APPIMAGE_EXTRACT_AND_RUN=1
export VERSION="$(pacman -Q $PACKAGE | awk 'NR==1 {print $2; exit}')"

APPIMAGETOOL="https://github.com/AppImage/appimagetool/releases/download/continuous/appimagetool-$ARCH.AppImage"
UPINFO="gh-releases-zsync|$(echo $GITHUB_REPOSITORY | tr '/' '|')|continuous|*$ARCH.AppImage.zsync"
LIB4BN="https://github.com/ghraw/VHSgunzo/sharun/refs/heads/main/lib4bin"

# Prepare AppDir
mkdir -p ./"$PACKAGE"/AppDir/shared/lib \
	./"$PACKAGE"/AppDir/usr/share/applications \
	./"$PACKAGE"/AppDir/etc
cd ./"$PACKAGE"/AppDir

cp -r /usr/share/welle-io   ./usr/share
cp -r /usr/share/locale     ./usr/share
cp -r /usr/share/glvnd      ./usr/share

cp /usr/share/applications/$DESKTOP              ./usr/share/applications
cp /usr/share/applications/$DESKTOP              ./
cp /usr/share/icons/hicolor/256x256/apps/"$ICON" ./

ln -s ./usr/share  ./share
ln -s ./shared/lib ./lib

# ADD LIBRARIES
wget "$LIB4BN" -O ./lib4bin
chmod +x ./lib4bin
xvfb-run -d -- ./lib4bin -p -v -r -e /usr/bin/welle-*
rm -f ./lib4bin

# DELOY QT
mkdir -p ./shared/lib/qt6/plugins
cp -r /usr/lib/qt6/plugins/iconengines       ./shared/lib/qt6/plugins
cp -r /usr/lib/qt6/plugins/imageformats      ./shared/lib/qt6/plugins
cp -r /usr/lib/qt6/plugins/platforms         ./shared/lib/qt6/plugins
cp -r /usr/lib/qt6/plugins/platformthemes    ./shared/lib/qt6/plugins
cp -r /usr/lib/qt6/plugins/styles            ./shared/lib/qt6/plugins
cp -r /usr/lib/qt6/plugins/xcbglintegrations ./shared/lib/qt6/plugins
cp -r /usr/lib/qt6/plugins/wayland-*         ./shared/lib/qt6/plugins
cp -r /usr/lib/qt6/plugins/qml*              ./shared/lib/qt6/plugins
cp -r /usr/lib/qt6/qml                       ./shared/lib/qt6

ldd ./shared/lib/qt6/plugins/*/* 2>/dev/null \
  | awk -F"[> ]" '{print $4}' | xargs -I {} cp -nv {} ./shared/lib || true

find ./shared -type f -exec strip -s -R .comment --strip-unneeded {} ';'

# Prepare sharun
ln ./sharun ./AppRun
./sharun -g

# MAKE APPIAMGE WITH FUSE3 COMPATIBLE APPIMAGETOOL
cd ..
wget -q "$APPIMAGETOOL" -O ./appimagetool
chmod +x ./appimagetool

./appimagetool --comp zstd \
	--mksquashfs-opt -Xcompression-level --mksquashfs-opt 22 \
	-n -u "$UPINFO" "$PWD"/AppDir "$PWD"/"$PACKAGE"-"$VERSION"-"$ARCH".AppImage

mv ./*.AppImage* ../
cd ..
rm -rf ./"$PACKAGE"
echo "All Done!"

And here is the appimage if anyone wants to do a quick test

[probonopd]

This is what happens with Debian on Windows 11 with WSL2:

[Samueru-sama]

I just booted an ubuntu 20.04 vm to test it:

And here is on a voidlinux musl vm: