Adventures in VDI

After spending an inordinate amount of time (up until 2:30AM today) struggling with getting GPU-accelerated desktops to work consistently in Proxmox, I thought I’d just jot down a few notes and move on to other things.

Background

I’ve been running xorgxrdp desktops with varying degrees of graphical acceleration for years, and there have always been a few recurring challenges:

Mapping the GPU into the VMs or LXC containers (which typically results in seeing a /dev/dri device tree inside the guest machines).
Having GPU support (xorgxrdp-glamor has fixed most of that, although H.264 encoding of the rendered desktop and streaming that is still done via software encoding)
Being able to run applications with accelerated graphics–which these days means ensuring flatpak applications can deal with the various libraries, drivers and devices.

This week I’ve been struggling with both ends of this:

I have a Fedora LXC container that has the host’s iGPU mapped correctly (Proxmox now has a simplified way to pass-through device namespaces into LXC containers) and works spectacularly well, but where I cannot get flatpak applications to run.
I have a Fedora Bluefin VM that has a VirGL (virtio-gl) GPU device (also riding on the host’s iGPU) where I can only get software rendering to work. I can run flatpak inside without any issues, since it runs its own kernel.

Discrete GPU PCI pass-through isn’t an option for some of these scenarios (and I have that working for other VMs just fine).

Bluefin And VirGL

This is the “easy” one–I initially thought there was some kind of issue with locked-down configurations (it is an “immutable” distribution), but I can see the virgl device using eglinfo, and it shows up as a “platform” device:

❯ eglinfo -B
...

Device platform:
Device #0:

Platform Device platform:
EGL API version: 1.5
EGL vendor string: Mesa Project
EGL version string: 1.5
EGL client APIs: OpenGL OpenGL_ES
OpenGL core profile vendor: Mesa
OpenGL core profile renderer: virgl (Mesa Intel(R) Graphics (ADL GT2))
OpenGL core profile version: 4.3 (Core Profile) Mesa 24.3.4
OpenGL core profile shading language version: 4.30
OpenGL compatibility profile vendor: Mesa
OpenGL compatibility profile renderer: virgl (Mesa Intel(R) Graphics (ADL GT2))
OpenGL compatibility profile version: 4.3 (Compatibility Profile) Mesa 24.3.4
OpenGL compatibility profile shading language version: 4.30
OpenGL ES profile vendor: Mesa
OpenGL ES profile renderer: virgl (Mesa Intel(R) Graphics (ADL GT2))
OpenGL ES profile version: OpenGL ES 3.2 Mesa 24.3.4
OpenGL ES profile shading language version: OpenGL ES GLSL ES 3.20

Device #1:

Platform Device platform:
EGL API version: 1.5
EGL vendor string: Mesa Project
EGL version string: 1.5
EGL client APIs: OpenGL OpenGL_ES
OpenGL core profile vendor: Mesa
OpenGL core profile renderer: llvmpipe (LLVM 19.1.7, 128 bits)
OpenGL core profile version: 4.5 (Core Profile) Mesa 24.3.4
OpenGL core profile shading language version: 4.50
OpenGL compatibility profile vendor: Mesa
OpenGL compatibility profile renderer: llvmpipe (LLVM 19.1.7, 128 bits)
OpenGL compatibility profile version: 4.5 (Compatibility Profile) Mesa 24.3.4
OpenGL compatibility profile shading language version: 4.50
OpenGL ES profile vendor: Mesa
OpenGL ES profile renderer: llvmpipe (LLVM 19.1.7, 128 bits)
OpenGL ES profile version: OpenGL ES 3.2 Mesa 24.3.4
OpenGL ES profile shading language version: OpenGL ES GLSL ES 3.20

However, glxinfo -B cannot see the virgl device at all inside xrdp:

❯ glxinfo -B 
name of display: :10.0
display: :10  screen: 0
direct rendering: Yes
Extended renderer info (GLX_MESA_query_renderer):
    Vendor: Mesa (0xffffffff)
    Device: llvmpipe (LLVM 19.1.7, 128 bits) (0xffffffff)
    Version: 24.3.4
    Accelerated: no
    Video memory: 15982MB
    Unified memory: yes
    Preferred profile: core (0x1)
    Max core profile version: 4.5
    Max compat profile version: 4.5
    Max GLES1 profile version: 1.1
    Max GLES[23] profile version: 3.2
Memory info (GL_ATI_meminfo):
    VBO free memory - total: 0 MB, largest block: 0 MB
    VBO free aux. memory - total: 13503 MB, largest block: 13503 MB
    Texture free memory - total: 0 MB, largest block: 0 MB
    Texture free aux. memory - total: 13503 MB, largest block: 13503 MB
    Renderbuffer free memory - total: 0 MB, largest block: 0 MB
    Renderbuffer free aux. memory - total: 13503 MB, largest block: 13503 MB
Memory info (GL_NVX_gpu_memory_info):
    Dedicated video memory: 0 MB
    Total available memory: 15982 MB
    Currently available dedicated video memory: 0 MB
OpenGL vendor string: Mesa
OpenGL renderer string: llvmpipe (LLVM 19.1.7, 128 bits)
OpenGL core profile version string: 4.5 (Core Profile) Mesa 24.3.4
OpenGL core profile shading language version string: 4.50
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile

I’ve been wracking my brain trying to figure out why since all the right packages seem to be installed and present in the LayeredPackages section of rpm-ostree status:

❯ rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: last run 11h ago
Deployments:
  ostree-image-signed:docker://ghcr.io/ublue-os/bluefin:latest
                   Digest: sha256:f54878b23f649344c50dcf79ae0e6311d5ba589da95c5e63f37172f0b11530e0
                  Version: latest-41.20250215 (2025-02-15T00:41:52Z)
                     Diff: 6 upgraded
          LayeredPackages: btop dialog egl-utils fira-code-fonts glib2-devel
                           libappstream-glib mosh neovim qemu-device-display-virtio-gpu-gl
                           sassc stow syncthing the_silver_searcher virglrenderer
                           xorgxrdp-glamor xrdp

● ostree-image-signed:docker://ghcr.io/ublue-os/bluefin:latest
                   Digest: sha256:53707ff4a24b114b121472133b64bc2146e747104d3409ad73121f5c29ee912e
                  Version: latest-41.20250214 (2025-02-14T05:01:09Z)
          LayeredPackages: btop dialog egl-utils fira-code-fonts glib2-devel
                           libappstream-glib mosh neovim qemu-device-display-virtio-gpu-gl
                           sassc stow syncthing the_silver_searcher virglrenderer
                           xorgxrdp-glamor xrdp

  ostree-image-signed:docker://ghcr.io/ublue-os/bluefin:latest
                   Digest: sha256:53707ff4a24b114b121472133b64bc2146e747104d3409ad73121f5c29ee912e
                  Version: latest-41.20250214 (2025-02-14T05:01:09Z)
          LayeredPackages: btop dialog fira-code-fonts glib2-devel libappstream-glib mosh
                           neovim qemu-device-display-virtio-gpu-gl sassc stow syncthing
                           the_silver_searcher virglrenderer xorgxrdp-glamor xrdp

…and the drivers appear to be loaded, etc.

So after a few hours poring over kernel and server logs of all kinds and triple checking configs, I created a new, “plain” Fedora VM with a completely identical configuration, and, surprise, it has no trouble using GPU-accelerated rendering:

❯ glxinfo -B
name of display: :10.0
display: :10  screen: 0
direct rendering: Yes
Extended renderer info (GLX_MESA_query_renderer):
    Vendor: Mesa (0x1af4)
    Device: virgl (Mesa Intel(R) Graphics (ADL GT2)) (0x1010)
    Version: 24.3.4
    Accelerated: yes
    Video memory: 0MB
    Unified memory: no
    Preferred profile: core (0x1)
    Max core profile version: 4.3
    Max compat profile version: 4.3
    Max GLES1 profile version: 1.1
    Max GLES[23] profile version: 3.2
OpenGL vendor string: Mesa
OpenGL renderer string: virgl (Mesa Intel(R) Graphics (ADL GT2))
OpenGL core profile version string: 4.3 (Core Profile) Mesa 24.3.4
OpenGL core profile shading language version string: 4.30
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
...

So I am chalking this one up as some sort of Bluefin bug, and moving on–however, I need to run flatpak applications, so I had to spend another few hours trying to shave yet another yak:

LXC and Bubblewrap

This is one of those “Googling for it only turns up my own questions” situation. I’ve been poking at variations of this problem since I adopted Proxmox, and the gist of the problem is that:

flatpak applications use bwrap to (re)map process namespaces.
LXC, apparmor et al impose a number of default restrictions you should be able to tweak via various /etc/pve/lxc/<id>.conf settings, including setting various privileged, nesting and capability flags.
But none of them actually work.

Also, infuriatingly, LXC containers running under Ubuntu’s LXD just work, and nobody seems to have figured out exactly why.

Right now, my test case is something as simple as this:

❯ bwrap --bind / / --proc /proc --dev /dev --unshare-pid /usr/bin/true
bwrap: Can't mount proc on /newroot/proc: Operation not permitted

This should be fixable by editing the .conf file to allow for remounting proc, trying to allow for additional capabilities or tweaking the host kernel to allow for cloning namespaces (my last attempt was setting kernel.unprivileged_userns_clone=1), but none of it seems to work, and most of the changes I’ve made to try to approximate what I think LXD does yield a hung container inside Proxmox.

And no, a privileged container doesn’t work either. Not even one with an unconfined policy.

Searching for this only digs up my own previous attempts from previous years, as well as posts in a few forums and mailing-lists, and that is part of the reason why I’m not adding the various configuration settings to this post (I don’t want it to turn up as a false positive in later search results).

But after another late night hacking at it, I decided to just go and look for non-flatpak alternatives to the apps I want to run. I suppose not many people try to run graphical applications inside Proxmox LXCs, so this isn’t a common issue–yet.

Conclusion

Both of these yaks remain unshaved, and any insights and suggestions are welcome–but keep in mind I’ve tried all the obvious ones…

Tao of Mac

Adventures in VDI

Background

Bluefin And VirGL

LXC and Bubblewrap

Conclusion

This page is referenced in: