Homelab Update

Every now and then I spend an hour or so in the evenings poking at my homelab, which kind of adds up into actual productivity over a couple of months.

So here are my notes from that time:

Pimox, The Great Little Hypervisor

I haven’t yet gotten around to upgrade my KVM server to Proxmox (I’ve decided to get new hardware first, and I’m eyeing the HX90g¹), but on a particularly rainy evening I decided I was going to put one of my 4GB Raspberry Pi 4s to good use and decided to install Pimox on an external SSD drive:

This is, of course, completely unsupported, but the great thing is that it works beautifully with LXC containers. Just go to the LXD Image Server and grab an arm64 rootfs of your favorite distribution, give it the URL, and you’re in business.

Among other things, this allowed me to catch up on the 7.x release (which I hadn’t yet had a chance to try). And I bet it should work on just about any ARM SBC that runs Debian, which opens up a lot of possibilities for cheap, low-power development sandboxes.

I’m now using it to set up a few ARM test environments, and although I haven’t yet figured out a clean way to use cloud-init with LXC containers, it’s only a matter of time.

And if you want to get rid of the annoying warning about it being an unsupported version, just run this as root:

sed -Ezi.bak "s/(Ext.Msg.show\(\{\s+title: gettext\('No valid sub)/void\(\{ \/\/\1/g" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js && systemctl restart pveproxy.service

This is something I’m publishing here because I’m positive other people will have similar issues:

My KVM server is nominally running Ubuntu 20.04 LTS, but it is actually an Elementary install that I stripped the desktop off and put in the closet. Since all the core packages are exactly the same, I didn’t give it a second thought, but of late it had been failing to upgrade the kernel when running apt dist-upgrade, and it was never the right time to shut down all the clients and have a proper look.

This is what I was seeing:

Setting up initramfs-tools (0.136ubuntu6.7) ...
update-initramfs: deferring update (trigger activated)
Setting up linux-firmware (1.187.33) ...
update-initramfs: Generating /boot/initrd.img-5.4.0-90-generic
E: /usr/share/initramfs-tools/hooks/plymouth failed with return 1.
update-initramfs: failed for /boot/initrd.img-5.4.0-90-generic with 1.
dpkg: error processing package linux-firmware (--configure):
 installed linux-firmware package post-installation script subprocess returned error exit status 1
Processing triggers for initramfs-tools (0.136ubuntu6.7) ...
update-initramfs: Generating /boot/initrd.img-5.4.0-90-generic
E: /usr/share/initramfs-tools/hooks/plymouth failed with return 1.
update-initramfs: failed for /boot/initrd.img-5.4.0-90-generic with 1.
dpkg: error processing package initramfs-tools (--configure):
 installed initramfs-tools package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 linux-firmware
 initramfs-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)

…as it turned out, Elementary‘s default plymouth settings required the Inter font to be installed in the system. Otherwise, the post-install shell script fails, hence blocking update-initramfs.

That was an easy enough fix, but since I had removed Inter when I got rid of all the desktop environment packages, this also means Elementary isn’t keeping track of all package dependencies – such are the pitfalls of maintaining a derived distribution, I guess.

I am still using ArchiveBox to take snapshots of interesting articles and other useful webpages, but with over 2000 items its built-in search functionality just wasn’t working out for me.

Fortunately, it supports Sonic, a nice little Rust-based indexing engine that can take the full-text extracts and provide lightning-quick responses, and I set that up as a sidecar container.

This sort of setup is highly recommended if, like me, you’re fed up with bits of the Internet dropping off just when you need to refer to them.

Most of my ArchiveBox is now obscure blogs and electronics-related stuff that tend to vanish after a few years, but since I imported my del.icio.us account (remember that?) and let it run for a few days, I also have a delightful set of 10-year old tech notes that are still surprisingly online and very useful to this day.

Another thing I did was to set up watchtower to do regular updates of some containers. It is an amazingly deploy-and-forget thing, and although I had to resort to SSH and docker-compose to set it up on my Synology (because its Docker management doesn’t allow for manifests), it was a painless and quite rewarding experience.

For extra peace of mind, I set it up to notify me of any updates via Pushover and to exclude a few designated containers.

Here’s an example of doing that and excluding Syncthing (I actually don’t exclude it, but I had the file around anyway, and it is a good example of something you likely want to update yourself):

version: "3.2"

services:
  watchtower:
    image: containrrr/watchtower
    container_name: watchtower
    hostname: ${HOSTNAME}
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - TZ=Europe/Lisbon
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_LABEL_ENABLE=true
      - WATCHTOWER_SCHEDULE=0 30 3 * * *
      - WATCHTOWER_ROLLING_RESTART=true
      - WATCHTOWER_TIMEOUT=30s
      - WATCHTOWER_NOTIFICATIONS=shoutrrr
      - WATCHTOWER_NOTIFICATION_URL=pushover://shoutrrr:${PUSHOVER_API_KEY}@${PUSHOVER_USER_KEY}/
    cpu_count: 1
    cpu_percent: 25
    mem_limit: 64m

  syncthing:
    image: syncthing/syncthing
    container_name: syncthing
    hostname: ${HOSTNAME}
    restart: always
    network_mode: "host"
    labels:
        - "com.centurylinklabs.watchtower.enable=false"
    volumes:
        - /home/${USER}/.config/syncthing:/var/syncthing
        - /home/${USER}/Sync:/Sync
    environment:
      - TZ=Europe/Lisbon
    cpu_count: 1
    cpu_percent: 50
    mem_limit: 256m

This has saved me a fair amount of time manually upgrading some of the Docker containers I run on my Synology, and is heartily recommended.

I was getting a bit annoyed with my Fedora container not showing file previews so I decided to investigate, which led me down a particularly niche rabbit hole.

As it turns out, GNOME wisely decided to sandbox its thumbnail generator to avoid security issues, but that took me a long time to figure out since there was no logging of any kind to indicate why it was failing.

That was quickly fixed by setting:

lxc config set gnome security.nesting true

…in the host. This also helps with flatpak and other forms of sandboxing.

Although we’re an Apple-centric home with a NAS, there is occasional need to send small files between various kinds of machines, and AirDrop has its foibles even among Apple devices.

So I’ve set up a local Snapdrop instance to transfer files and URLs via WebRTC between just about any browser, and it’s been working great:

There are loads of similar solutions out there (I also like ShareDrop), but I liked the look, feel and overall simplicity of Snapdrop, so I did a few changes to it:

• Got rid of the nginx Docker container it was using for SSL and serving static files over TLS (doing anything with SSL on a home LAN is a mess, and it wasn’t really necessary).
• Moved static file serving to its (tiny) index.js by adding express to the otherwise pure websocket server.
• Commented out all of the stuff that would require SSL or make no sense in a home setting (PWA support, notifications, etc.).
• Changed its peer discovery mechanism (which is the only thing the server does, really, since WebRTC file transfers are fully peer-to-peer) to understand clients were on the same LAN.

The original code was quite simple, but it was following the assumption that clients from the same IP address would be in the same LAN, which is sort of broken.

The end result takes almost zero resources and is easily deployable via Piku, so that was a very satisfying quick hack that I just cleaned up and released on GitHub.

I’ve been doing a few more, of course, but they aren’t fully baked yet.