Is it really any linux?

Here is GIMP running in Ubuntu 10.04

Here is Cromite running in NixOS without any FHS-wrapper

Here is Cromite running in Ubuntu 14.04

Here is WINE running foobar2000 in Ubuntu 14.04

Here is QEMU running in Ubuntu 12.04

Here is FeatherPad running in Ubuntu 6.10 👀

Here is aarch64 Trelby running on 32-bit ARM debian 👀

This is possible because this system had a 64bit kernel and CPU. We barely depend on the host userland besides some POSIX utils like sh.

Here is Eden running in FreeBSD using Vulkan with NVIDIA via Linuxulator 👀

What’s the minimum supported kernel version?

• Short answer: 2.6.17 (Ubuntu 6.10 era).

• Anything older than 2.6.17 is not possible due to no openat (needed by dwarfs), which means we are unable to execute sharun.

glibc on archlinux is compiled with --enable-kernel=4.4, that does not mean it is unable to run on kernels older than that, it will work as long as it doesn’t attempt to use a syscall not present in such kernels. For example GIMP3 runs perfectly in Ubuntu 10.04 (kernel 2.6.32) as shown above.

However one problematic syscall is statx, which is kernel 4.11 which Qt depends on and apps will crash when missing.

To fix this and a several other potential issues, our fork of sharun now has a compatiblity layer for older kernels, for more details see the Anylinux-sharun README.

Supporting old kernels has run into some interesting issues, for example:

• A missing syscall should return ENOSYS so the caller knows and can fall back if possible. Some 2.6-x kernels instead return the syscall number itself: getrandom returns 318, clone3 returns 435. This causes glibc to think the syscall works, while Rust’s std panics with range start index 318 out of range for slice of length 16. The fix is just to notice rax == nr and treat it as ENOSYS.

• On 3.8.0-19 prctl(PR_SET_NO_NEW_PRIVS) followed by execve fails with EPERM. Yes prctl returns 0 and the very next execve("/bin/sh", ...) comes back EPERM lol? The fix is just probe if execve works, else don’t use prctl.

How come this only became possible in 2024?

• For an application to be truly portable we need to ship our own dynamic linker (ld-linux.so).
• It turns out it is not possible to have a relative dynamic linker with executables.
• polyfill glibc attempted to fix this issue with an experimental tool that replaces PT_INTERP with PT_LOAD and has the payload look for the relative dynamic linker but this never got finished.
• We can execute the dynamic linker first and then pass the binary to launch to bypass this limitation, go-appimage had been doing this since ~2019.
• But that runs into issues with /proc/self/exe.
• sharun had to be made to fix the /proc/self/exe issues. And as far as I know, brioche had been using the same approach before sharun as well.
• Once all the pieces were ready, the next step was changing the way we deploy AppImages and sorting all the bugs that came with that, AppImage was originally made with the idea of relying on the host glibc and a set of libraries that always had to come from the host.

I didn’t understand any of this

• You know when you have a shell script that it has shebang right? #!/bin/sh for example. And lets see our script is in /usr/bin/myscript. Well when you execute that file, you actually just tell the kernel to execute /bin/sh /usr/bin/myscript.
• So if we wanted to have a truly portable shell script, we just would need to bundle our own sh and always execute sh /path/to/script. And this is true for shell scripts (with a few minor exceptions not worth mentioning here).
• In shell scripting there is this special parameter called $0, it tells you the path of the script that is being executed, remember this since it is very important.

So what’s the problem with dynamic binaries?

• The equivalent of $0 in binaries is reading /proc/self/exe, this is a magic symlink set by the kernel that points to the current running process.
• /proc/self/exe is set by the kernel when you execute a binary, this is not something we can ask the kernel to change.
• So when a binary that was executed with the dynamic linker (ld-linux.so /path/to/binary) if that binary checks /proc/self/exe it will get the path to ld-linux.so instead of /path/to/binary and many apps break horribly as result, some will even report ld-linux.so as the window class lol.
• This stupid problem is what has prevented true 100% binary compatiblity in linux for decades 😹

This problem is what sharun fixes, by using userland-execve and bypassing the kernel execve, since it is the kernel what sets /proc/self/exe, if we use userland-execve we can control where /proc/self/exe actually points to instead and fix this.

Why bundle glibc instead of musl?

• Using musl would mean any hardware accelerated application will not work with the proprietary nvidia driver.
• musl runs into performance issues because the default allocator is not great, this even affected the type2 AppImage runtime.
• It does not really save space, the libc is a small fraction of the entire AppImage size, the reason distros like alpine linux are small is because they optimize most of their packages for size like this example that results in a libicudata.so that is less than 1 MiB in size while most other distros do not bother to do this optimization and ship a 30 MiB libicudata.so. Many of these optimizations are already used in the debloated packages repo.
• With glibc, we are able to dlopen optional libraries on the host even when those link to musl. If we used musl the opposite is usually not possible as musl lacks a lot of symbols that libraries expect from glibc. For example here is the Qt6-demo dlopening alpine’s GTK3 to use the GTK3 platform theme and look native on the system:

We only use musl where it is very useful, that is when making static binaries.

Why not statically link everything?

• That is super hard, some libraries are not meant to be statically linked as well and that means a ton of patches are needed.
• Statically linking everything means we are not able to dlopen any library from the host, even optional ones like the example I just gave about dlopening the host GTK from Qt apps to follow the system theme.
• It means goodbye to the proprietary nvidia driver.
• It means you are no longer able to use vulkan layers like mangohud or lsfg-vk.
• It means you are forever stuck with the version of MESA that was statically linked. Remember, you can use the host Mesa if needed by setting SHARUN_ALLOW_SYS_VKICD=1 and that is something you will want to do if you plan on using the same AppImage for several years in the future.
• Static linking some dependencies is still desired however, as that reduces the final size of the AppImage, but a fully static binary is a very bad idea.

Why not use solo or detour?

These solutions allow statically linked programs to dlopen host libraries, amazing no? Well that runs into several problems:

• What happens if my application needs OpenGL 4.6 but the host Mesa only supports OpenGL 4.5? Sadness.
• What happens if I end up statically linking LLVM and the host’s Mesa links to a different version of LLVM? More problems.
• Also it seems none of the solutions implement dlmopen, so you are likely to run into a lot of symbol collisions with host libraries depending on what you end up building.

Using the host Mesa you are also going to run into bugs that had already been fixed in Mesa, we used to allow our AppImage to use the host vulkan drivers along with the bundled drivers, that ended up being a bad idea.

Also you are not forced to use our bundled drivers always, you can always set USE_HOST_MESA_DRIVERS=1, this will help if you plan to use the same AppImage several years into the future, but it is not guaranteed to work forever due to glibc symbol nonsense.

We don’t run into these problems with Nvidia, because Nvidia releases its proprietary driver linking to super old versions of glibc so you can be certain it will always work.

UPDATE: We now have a similar feature via cross-libc-dlopen, enabled via USE_HOST_DRIVERS_EXPERIMENTAL=1 in quick-sharun.

This feature is only going to be used if the applications meets the following conditions:

• The application doesn’t depend on a recent version of OpenGL. (A good test is checking if the application works with the softpipe driver since that only supports OpenGL 3.3)
• The application does not have a hard dependency on vulkan. (Most vulkan apps require relative new versions of vulkan (1.2 or newer) which only began to show up in Mesa 20.0 ~Ubuntu 20.04).
• The application has a fallback software renderer. Who knows what can happen in future, it is likely for example that OpenGL might not be installed by default anymore in the next decade and now we have applications that no longer work.

Why DwarFS instead of SquashFS?

DwarFS is a lot faster than SquashFS while being smaller at the same time.

DwarFS also offers PGO like optimizations, which allows us to make small appimages that start instantly.

AppImage has no thumbnail?

Because we use DwarFS instead of SquashFS, you need an AppImage thumbnailer that supports DwarFS:

• appimage-thumbnailer
• simple-appimage-thumbnailer

I get ERROR: Can't find a valid SQUASHFS superblock in NixOS

Once again this is because we use DwarFS instead of SquashFS, NixOS has something called appimage-run which lets you run old type appimages that need an FHS env and some host libraries, appimage-run manually mounts the appimage instead of letting it execute itself which results in that error since it expects it to be SquashFS.

None of this is needed for our appimages, they run directly in NixOS, so all you have to do is disable appimage-run.

Why is there no usr directory in the AppImages?

Because it causes more issues than it solves.

• /usr is the typical installation prefix for an application.

• $APPDIR/usr makes no sense, it just causes projects to code exceptions for appimage that do something along these lines: getenv(APPDIR) + usr + xyz. Instead we make APPDIR the installation prefix directly. This means we can take any application and patch away the /usr prefix for $APPDIR and make them portable without the need for projects to support AppImage. Here are some examples where projects checking for $APPDIR just made things worse: 1 2

• NOTE: $APPDIR/shared is an internal directory that sharun uses for itself, you should never copy anything manually there.