I’ve been running a mixed Proxmox cluster for years – four nodes of wildly different capability, from an Atom x5-Z8350 with 2 GB of RAM (a z83ii, currently offline after years of faithful service as a baseline torture device) up to an i7-12700 with 128 GB (borg, my main homelab server).

This year, somewhere along the way between writing agentbox and all the hype around agentic sandboxes I got tired of the eternal compromise between LXC containers and full virtual machines, and ended up building pve-microvm – a Debian package that adds QEMU’s microvm machine type as a first-class managed guest in Proxmox VE.

This isn’t a quick hack. Well, the first version was, actually, but it’s gone quite a bit farther than that, and certainly farther than I expected.

It now ships a custom kernel, patches the Perl internals to provide Proxmox web UI integration, and, due to my usual fascination with offbeat operating systems, ended up supporting (as of this writing) 21 guest OS types from Debian to NetBSD to Plan9.

Yes, I completely brought it upon myself to run Plan9 in a microVM, and yes, it works.

After a few rounds of cluster cleanups and migrations, it’s now my daily driver for running Gitea, Caddy reverse proxies, mini-firewalls, and the AI agent that’s helping me clean up this post.

Proxmox gives you two main options out of the box:

• LXC containers start instantly, share the host kernel, and are spectacularly efficient. But they’re not isolated – a kernel exploit in one container compromises everything. You can’t run a different OS. You can’t easily nest Docker inside them without ending up (eventually) wrestling with fuse-overlayfs gymnastics. And certain workloads (anything needing custom kernel modules, or CAP_SYS_ADMIN in anger) simply don’t fit.

• Full VMs give you hardware isolation via KVM/VT-x, but they boot SeaBIOS or OVMF, sedately walk through GRUB as they yawn their way out of bed, probe a forest of emulated legacy devices (IDE controllers, VGA, USB hubs, PCI bridges), and typically take 5-10 seconds to reach a login prompt. Each one carries the overhead of that entire emulated chipset sitting in memory.

What I wanted was the security boundary of a VM with the startup characteristics of a container. QEMU’s microvm machine type – originally developed for Firecracker-style workloads – strips all of that away. No BIOS, no GRUB, no legacy devices. Direct kernel boot into a minimal virtio-only environment. The result: sub-300ms boot to a fully networked guest with a QEMU agent, running inside its own KVM hardware isolation boundary.

Now, let me be clear: I’m not spawning hundreds of these things. I have Azure for that – but I do want to run Gitea Actions workers, have a very limited set of hardware resources, and got fed up with the time it took for one particular VM to boot repeatedly…

pve-microvm is a single .deb that patches Proxmox’s qemu-server Perl modules at install time. When you set machine: microvm on a VM config, the standard config_to_command function delegates to my MicroVM.pm, which builds an (almost) completely different QEMU command line:

qemu-system-x86_64 -M microvm,x-option-roms=off,pit=off,pic=off,\
  isa-serial=on,rtc=on,acpi=on,pcie=on \
  -kernel /usr/share/pve-microvm/vmlinuz \
  -initrd /usr/share/pve-microvm/initrd \
  -append "console=ttyS0 root=/dev/vda rw quiet" \
  -device virtio-blk-pci-non-transitional,drive=drive-scsi0 \
  -device virtio-net-pci-non-transitional,netdev=net0 \
  ...

No chipset emulation. No PCI bridges. No VGA. The guest gets a single serial console (which PVE’s xterm.js connects to natively), virtio block devices, and a virtio network interface. Everything rides PCIe transport with non-transitional (modern-only) virtio devices rather than the MMIO transport microvm was originally designed around – for reasons I’ll come to in a moment.

The package ships:

• A tiny (12MB) pre-built Linux 6.12.22 kernel compiled from x86_64_defconfig with a minimal overlay – virtio, vsock, virtiofs, 9p, and the modules Docker needs (overlay, veth, bridge, netfilter, BPF), because, well, I’m pragmatic.
• A 1 MB initrd that probes virtio devices, finds the root filesystem by label or device path, and does a switch_root in ~150ms
• pve-microvm-template – builds root filesystems from any of 12 supported OCI base images, with optional SSH, Docker, and guest agent
• pve-oci-import – pulls an OCI image directly into a PVE-managed disk
• Web UI extensions – a “Create µVM” button, machine type dropdown, conditional panel hiding for irrelevant settings, and an amber bolt icon in the resource tree
• A systemd service (pve-microvm-early.service) that ensures patches are applied before pvedaemon starts on boot – critical for onboot=1 VMs

Like in aerodynamics, most speed comes from eliminating everything that isn’t strictly necessary. A standard VM spends most of its boot time in firmware and bootloader, so a microVM skips all of that.

SmolBSD (a NetBSD guest using virtio-mmio transport) boots in 31ms. A full Debian with Docker and the QEMU agent is ready in under 8 seconds – and most of that time is apt package installation during first boot. Subsequent boots hit the 300ms mark consistently, even on my humble hardware.

A fun rabbit hole I went into when someone asked me to add SmolBSD support:

There’s a reason SmolBSD gets to use virtio-mmio and the Linux guests don’t. A QEMU microvm machine type can carry its virtio devices over two transports: the bare-bones MMIO interface it was originally built for, or PCIe. MMIO is the lighter of the two – no PCIe host bridge, no ACPI – which is how a NetBSD guest shaves itself down to 31 ms.

But on QEMU 10.x the MMIO path has (as far as I can tell) a device-probing bug for Linux guests: only virtio-blk binds, and the network, serial and balloon devices are never claimed by their drivers for some reason. NetBSD probes MMIO correctly and is perfectly happy; Linux (at least the kernel I am using) is not.

For every Linux guest I therefore fall back to PCIe with non-transitional (modern-only) virtio devices, which binds all of them reliably. The cost is about 50 ms of extra bring-up – which, against a 300 ms boot, I’ll take without complaint.

I think the above is actually a bug in my kernel configuration, but haven’t really had time (or maybe even the right hardware) to tackle it – this is something I’d love more people to look at and contribute patches.

There’s a deliberate consequence of this direct kernel boot trick that’s easy to miss: the kernel doesn’t live inside the guest. It sits on the Proxmox host at /usr/share/pve-microvm/vmlinuz, and the guest disk holds nothing but a root filesystem – userland, no /boot, no GRUB, no per-guest kernel package, no initramfs of its own.

That also means there’s no “boot the installer ISO and click through it” path, so instead the rootfs gets built straight from an OCI image with pve-microvm-template (Debian, Alpine, Fedora, Rocky, Amazon Linux and friends). In the weird cases, we import a prepared ext4/raw disk with qm importdisk. You don’t install an OS – you assemble a root filesystem.

Decoupling the kernel from the rootfs is what makes this interesting to run at scale. Every Linux microVM on the node boots the same vmlinuz – one kernel, built once from a stock x86_64_defconfig with a microvm overlay, so you can audit and update it in exactly one place: drop a new vmlinuz on the host, restart the guests, done. No guest ever pulls a broken kernel from an apt upgrade, because no guest has a kernel to upgrade, and the rootfs images stay tiny and completely kernel-agnostic.

Container-style kernel consistency, VM-style isolation.

On my cluster right now, I have a fair smattering of these already. Four off the top of my head are:

• gitea (VM 114, on an Intel N5105) – Bare-metal Gitea with SQLite, Caddy HTTPS, local actions runner, Avahi discovery. 2 cores, 2 GB RAM, 32 GB disk. Boots in ~3s, mostly because Gitea does a lot of housekeeping.
• smith (VM 9022, on my i7) – the main piclaw agent, the system that manages the cluster, releases piclaw and generally keeps tabs on everything. 2 cores, 6 GB RAM, 48 GB disk. Runs Docker internally, and has 3 smaller, volatile siblings scattered throughout the cluster that don’t run Docker but have different roles (CI/CD, wipe-and-reinstall agent instance for testing upgrades, etc.)
• exo (VM 9021, on my i7) – Distributed inference coordinator for running LLMs across multiple machines. CPU-only, 2 GB root.
• virtualdsm (VM 300, tnas) – Synology DSM running inside a microVM with Docker, inside Terramaster NAS hardware. Yeah, I know I’m weird, but it was needed when my Synology went sideways and I haven’t nuked it yet. Uses the stock Debian kernel rather than my custom one, because DSM needs specific module paths.

I also have a dormant 9Front (Plan9) one, as well as a little menagerie of OpenWrt, OPNsense, OSv unikernels, gokrazy Go appliances, and various Alpine/Fedora/Rocky/Amazon Linux configurations filed away as standard Proxmox backups. The 21 guest OS types aren’t theoretical – each one has been booted and validated, and sometimes smith will go and thaw one out to do regression tests.

The z83ii (that ancient Atom x5-Z8350 with 2GB RAM) was invaluable as a baseline test platform, because If a microVM can boot and run usefully on a fanless 2016-era Atom with 2 GB of total system memory, it’ll work anywhere. And it could run six before it started slowing down…

There’s no magic to a microVM config – it’s an ordinary qm guest with a particular machine type and a kernel command line. Here’s gitea (the VM 114 above) as it sits in /etc/pve/qemu-server/114.conf:

agent: 1
args: -kernel /usr/share/pve-microvm/vmlinuz -append "console=ttyS0 root=/dev/vda rw quiet"
boot: order=scsi0
cores: 2
machine: microvm
memory: 2048
name: gitea
net0: virtio=BC:24:11:00:6E:01,bridge=vmbr0
onboot: 1
scsi0: local-lvm:vm-114-disk-0,size=32G
serial0: socket
tags: microvm
vga: serial0

The only microvm-specific lines are machine: microvm, the args carrying the kernel and its cmdline, and serial0: socket / vga: serial0 wiring the console through to xterm.js. Everything else – cores, memory, the virtio NIC on vmbr0, the scsi0 disk on local-lvm, onboot, the guest agent – is exactly what you’d write for any Proxmox VM.

Note there’s no -initrd in args: MicroVM.pm injects it automatically when it sees the shipped kernel, along with the balloon, vsock and (when configured) virtiofs devices. That’s the whole point – a microVM is a normal guest that happens to boot a host-provided kernel, not a special object you have to learn a new tool to manage.

A microVM attaches to the network exactly like any other Proxmox guest. The interface spec is a bit of a mouthful (virtio-net-pci-non-transitional device on the PCIe bus), and it lands on whatever Linux bridge and VLAN you point it at:

qm set 900 --net0 virtio,bridge=vmbr0           # single NIC
qm set 900 --net0 virtio,bridge=vmbr0,tag=100   # tagged onto VLAN 100

Because it’s an ordinary KVM guest, the standard PVE firewall applies – per-VM nftables rules work the way they do for any VM, which is the part that actually matters when you’re running untrusted code.

Inside the guest, networking is handled by systemd-networkd rather than cloud-init: DHCP by default (matched on Type=ether, so it survives cloning with no MAC pinning), or a one-line /etc/microvm-static-net for a static address. Earlier versions leaned on cloud-init for this, and I found it too brittle; moving it to systemd-networkd made cloning reliable and I stopped having to debug templates half the time.

Network isolation, another mainstay of the current hype around agent isolation, is a solved problem I have no interest in re-solving inside the package because I think Proxmox’s own SDN already does it properly – a simple VLAN zone with a separate VNet per trust domain keeps untrusted guests on their own segment with no path to the LAN, and if I ever bothered with that on a home LAN (well I have thought about it… but got no time), a VXLAN zone with a designated exit node would let me funnel egress through a single firewalled choke point.

The microVM just lands on whatever VNet I point net0 at, so the policy lives in Proxmox, not in a one-off ruleset I’d have to babysit.

There’s also a non-networked path for host/guest plumbing: each guest gets a vsock CID (VMID + 1000), which I use for SSH-agent forwarding (host keys into the guest without ever exposing them on the wire) and for virtiofs/9p directory sharing, because… I forget. I know I needed it at one point, even if right now most of my microvm instances just do SMB mounts (which were a pain to do under Docker and LXC)

The NIC count is… moderately sane. I currently allow six virtio interfaces per guest (net0-net5) on the spurious grounds that they’re twice the maximum number of physical interfaces across all of my host machines. It’s just defined in MicroVM.pm, and the very few people who need more than that are welcome to tweak it (and if you think you want a dozen you almost certainly want VLANs instead).

Storage was… trivial? every PVE backend works, because the disk is just a virtio-blk-pci device and Proxmox hands it the same path it hands any VM. LVM and LVM-thin, ZFS, Ceph/RBD, NFS and CIFS, plain directory storage – all fine, with snapshots, linked and full clones, vzdump backups and qm importdisk behaving exactly as they do elsewhere. A microvm is a normal PVE guest that happens to boot differently.

Migration is the only place things are iffy. a) I can’t do live migration on any of my hardware (none of it is enterprise grade), and b) true live migration isn’t viable with the current QEMU microvm machine type anyway – it simply doesn’t implement it.

Offline migration (i.e., reshuffling instances around) works fine, though, and because microvm boot in well under a second, you can run a quick HA relocate cycle – stop, migrate, start – provided your disks live on shared storage.

I’ve measured it at around two seconds moving a small guest between nodes on shared CIFS. Not seamless VM motion, but perhaps good enough for most uses, and as far as I can figure out, ha-manager drives it the same way it would any guest.

Mind you, I still run LXC containers. They’re the right choice when:

• You trust the workload (or it’s your own code)
• You need zero boot overhead
• You want direct filesystem access from the host
• You want slightly less memory allocation overhead (oh, and there’s shared page cache if you’re really lucky)

MicroVMs win when:

• You need actual kernel isolation – running untrusted code, different kernel versions, nested Docker, anything with aggressive CAP_ requirements
• You want a reproducible image you can snapshot, back up (vzdump works), offline-migrate, and clone (linked clones too)
• The workload does something unusual to the kernel (BPF programs, custom netfilter rules, kernel modules) and you don’t want that leaking into your host (which is why most of my tunnels now land in a microvm)
• You’re running non-Linux guests – NetBSD, Plan 9, FreeBSD-based firewalls – which simply can’t run as LXC
• You want a VM that’s up before you’ve finished the command.

The overhead difference between LXC and a microVM on modern hardware is surprisingly small. On borg (that i7-12700), the idle memory footprint of a minimal microvm is ~40 MB, and the CPU overhead of the hypervisor is unmeasurable for most workloads.

The memory figure you set at boot is genuinely just a ceiling: As far as I can tell KVM still only backs the pages a guest actually touches, and the host’s same-page merging dedupes identical pages – kernels, libc, shared base-image layers – across every microvm on the node, so like on grown-up cloud hypervisors the real cost tracks the working set, not the nominal allocation.

Much to my embarrassment, I didn’t even bother with live resizing for a long while, but I added balloon support to the kernel recently, with free-page-reporting and deflate-on-oom. PVE’s balloon target drives proper auto-ballooning, and a virtio-mem device gives genuine fine-grained hot-add and hot-remove, so, again, this works like a normal VM.

Now for the catches…

First one is pretty obvious: I’m patching someone else’s product, and even though I survived the Perl 4 to Perl 5 transition decades ago and I have Codex to help these days, patching the Perl internals is fragile–every qemu-server upgrade can break my setup.

I mitigate this by a) not blindly running upgrades and b) using dpkg triggers (the package re-patches automatically after upgrades), but it’s still a race condition waiting to happen – I’ve had partial PVE upgrades leave the system in a state where pvedaemon couldn’t even compile the patched module.

And then come the weird failure modes–one upgrade had a nasty little side effect regarding root devices:

• Root is found at root=/dev/vda because the guest boots under the microvm machine type with virtio-blk on PCIe
• but if a guest ever comes up under the standard chipset instead (a half-applied patch, or an onboot=1 VM starting before the early-boot service has re-patched after a host reboot), the same disk enumerates as /dev/sda, root isn’t found, and the VM looks like it has lost its filesystem.

The data is always there, just at the wrong path. I’ve patched that particular twist in a way that the dpkg trigger and early-boot service ordering try to mitigate it, so the initrd now falls back to /dev/sda when /dev/vda is missing – but until (if ever) Proxmox supports microVMs natively, expect the occasional papercut.

Package version mismatches can cascade. I learned this the hard way – a partial apt upgrade on one of my nodes left libpve-cluster-perl and libpve-network-perl at incompatible versions, which broke all Perl module loading, which meant pvedaemon couldn’t start, which meant VMs couldn’t be managed.

So… always do full dist-upgrade, never partial upgrades on PVE nodes with this.

Another catch (for some) is that there is no VGA and no graphical console – the serial console is your only interface. If something goes wrong during boot, you’re reading kernel panics on a terminal. This is fine for servers, less fine for desktop-oriented guests, and Plan9 really doesn’t like it.

The next one is that there’s no USB at all – no controllers, no passthrough, nothing on the bus – so I had to get the web UI to hide those options the moment you flip a guest to microvm. This might be a deal-breaker if you want to, say, run a Zigbee controller (and is why my home automation stuff is still in an LXC).

The kernel is opinionated. My custom 6.12.22 kernel includes exactly what I need and nothing else. If your workload needs a module I haven’t included, you’ll need to rebuild it or use the stock Debian kernel (which works fine but is 3x larger and boots slower).

Finally, GPU and PCI passthrough are disabled, not impossible. And this is another thing I would love people with more money hardware to really dive into, since I actually had an RTX 3060 passed through and working in early testing.

In the end, I decided to temporarily disable GPU support for simplicity – the package strips hostpci* today and there’s no vIOMMU plumbing wired up, so it’s off by default.

There’s nothing stopping anyone from adding it back (save some QEMU microvm caveats around the minimal ECAM PCIe bus and IOMMU setup), and I’d genuinely love to be able to do proper PCI and vIOMMU testing – I just don’t have the spare hardware for it, and I chose to focus this on running agents rather than chasing accelerator passthrough.

The package modifies three files in the PVE Perl stack:

• Machine.pm – extends the machine type regex to accept microvm as a valid value
• QemuServer.pm – adds a use PVE::QemuServer::MicroVM import and a delegation check at the top of config_to_command: if the VM has machine: microvm, hand off to my module entirely
• MicroVM.pm – the complete command builder, installed at /usr/share/perl5/PVE/QemuServer/MicroVM.pm

Since I’m obsessive about recoverability, the patches are reversible (pve-microvm-patch revert), and the original files are backed up. The dpkg trigger system (interest-noawait qemu-server) ensures automatic re-application after PVE updates. The early-boot service ensures patches are live before any VM auto-starts, preventing that sda/vda mixup I mentioned above:

pve-microvm-early.service (Before=pvedaemon.service pve-guests.service)
    └── /usr/share/pve-microvm/pve-microvm-patch apply

If you’ve read this far, you’re either a brave person or missed the GitHub link, so I’ll just drop the potted version here:

# On any PVE 9.x node:
wget https://github.com/rcarmo/pve-microvm/releases/latest/download/pve-microvm_0.3.12-1_all.deb
dpkg -i pve-microvm_0.3.12-1_all.deb

# Create a Debian microVM template:
pve-microvm-template --vmid 9000 --storage local-lvm --profile standard

# Clone it into a real VM:
qm clone 9000 100 --name my-microvm --full
qm set 100 --machine microvm --memory 1024 --cores 2
qm start 100

The template takes about 60 seconds to build (pulls the OCI image, installs packages, writes the root filesystem). After that, cloning is near-instant if you use linked clones.

Besides more testing (that I just don’t have the hardware, time or even focus for, since like most of my projects I just want to use the thing), what’s left is mostly polish: maybe a nicer configuration layer, network-off-by-default with egress allow-lists for untrusted guests for the paranoid, GPU passthrough if I ever wire up vIOMMU properly, and eventually AArch64 support for ARM-based PVE nodes (which I’ve run in the past, and I think might also have microvm support in their QEMU versions).

And, of course, someone should probably upstream this. I reached out informally to a couple of people at Proxmox and was told I should join a mailing-list to discuss this, but a) that is awfully 90s and b) I just don’t have the time to do more than maintain this for myself. And yeah, I get that this would have to go through the entire enterprise QA pipeline (I get that a lot, believe me).

The source is at github.com/rcarmo/pve-microvm, the patches are fairly small, it does not break Proxmox itself since 99% of the features come from QEMU/KVM, so… I’m just literally giving it away.

But if you’re running a homelab, know your way around Linux and want container-speed VMs with actual isolation, this might be what you’ve been looking for since… well, statistically, maybe never–but you can have it now!

This is a very quick follow-up to my Mac emulation hacks from a couple of weeks ago, and worth noting for the fun value and a little bit of AI.

I love old arcade games (especially some NeoGeo titles), so it was only natural that I gravitated to them while I was trying to get Mac color rendering to work on an ESP32–if there’s a piece of software that was extremely attuned to its hardware, it’s arcade games, often written to map directly into hardware.

And I love R-Type in particular, so even though I originally thought of getting Metal Slug to run on the ESP32-S3 because of its shared 68000 heritage with the Mac, I ended up wondering how fast I could make that run.

Turns out the M72 boards Irem did for R-Type ran an 8086-like CPU (the NEC V30, which has a few extensions) and a Z80 in tandem, and that the emulator wasn’t at all hard to recompile if you stubbed out things like audio (which is done by the Z80).

I decided to start with the hardest/smallest target (the plain CYD with a plain ESP32), which can barely run the emulator in one core and has almost no free RAM–to the point where after a few iterations it was rendering something, but clearly wouldn’t make it without rebuilding the whole emulator from scratch.

Getting it to render frames effectively (as in, rendering one frame without any visible stutters inside the frame), is exactly the kind of problem I am having on the Mac emulator because a) you typically need enough RAM to manage the framebuffer and b) all ESP CYD displays have limitations regarding display (typically SPI) bandwidth.

For a little bit of inside baseball (yeah, I’ve been spending time with US folk again) the real hassle (especially on the smaller ESP32) was handling memory maps, palette RAM, tile/sprite priority, and frame timing. You can finagle things a bit by reassigning one of the cores to “just” do rendering, and there are various DMA modes depending on chipset, but all of which proved to be enough distraction for me to upgrade to an S3-powered display as soon as I could.

So I just focused on clean frame renderings, even if the time required to produce them made it feel like a slideshow, so much so that after figuring out the backgrounds were a static texture composited behind the main sprites, I decided to skip that.

It would have been amazing to see running on the smaller one, though.

Then I got piclaw to port the entire thing to the ESP32-S3, and all of a sudden there was enough horsepower to run and render at around 50fps:

I’m so happy with the results that I am considering getting this to run on an ESP32-P4 and see what we can do about audio and using the USB host port on that for a controller, but I really should focus on backporting the rendering techniques into a Mac emulator…

Either way, this was a great way to refine my approach at getting AI agents to tackle long, grinding, intricate problems, and the code is up on GitHub if anyone cares to check it out.

However, before handing it over to agents, I had to specify how to do this, and right now, after half a dozen embedded development and hardware porting projects since Christmas, the strategy is pretty well established:

• Get something to run on a host harness, running VNC, plain SDL or just framebuffer dumps
• Derive milestones from that (still quite manual) job. Maybe even more harnesses (like target CPU opcode harnesses for JITs, sprite subroutines, etc.)
• Tackle the first few milestones on a simpler (but also more limited) hardware/software target
• Build reusable debugging/introspection tools for each milestone that the agents can use later to have a feedback loop
• Expand out from the above.

That’s why my first hack for these things is just to point a webcam at the display (or generate a frame, or a known good end-to-end output dump) and get them to render a test pattern:

From then on, the agents can use the camera and other test patterns to verify that they are rendering correctly (of course it’s useless for video, but any SOTA model these days can take useful feedback from images), and, as a bonus, I get their snapshots on the piclaw web interface and can verify that they are actually doing what I want them to do.

I already knew what I wanted to achieve (in short, to explore and document techniques to render fast graphics on these boards), and I had a camera pointing at the target devices like in previous hacks, but one of the things I wanted to explore with this setup was to mitigate long context problems:

• Even if you use things like /goal (which I do, but with bounded horizons) models will inevitably deviate from the actual goal
• As context piles up, they will also inevitably hyper focus on tangentially relevant issues (because they see code issues and zero in on those rather than take a broader view of what needs to be achieved)
• Dead ends and back-tracking to reassess better approaches becomes nearly impossible

What I did was very simple. piclaw allows me to easily have multiple sessions running, and a few weeks ago I implemented a chat tool, with hilarious results:

…plus “agents” or sessions also have the ability to introspect each other’s state (goals, messages, current activity, compaction status, etc.) and schedule themselves, so setting up an @auditor / overseer that can keep track of other agents is trivial–all I needed was to write a SKILL.md file that told the auditor to:

• Observe commits, logs, tests, and artifacts; judge progress from concrete evidence towards the set goal, not just sessions being generally “active” but treading water.
• Enforce strict, reproducible completion gates (no interpreter fallbacks, ROM/global seeding, scanner bypasses, or synthetic shortcuts like skipping steps or faking code).
• Nudge active sessions once with a concrete, evidence-backed step, a measurable success signal, and any corrections to make.
• Require commit/push hygiene with a quality bar for commit messages
• Never edit target-session code or implement fixes, keeping itself to steering only via chat and audit log entries.
• Escalate from steering to actual interruptions only after repeated ignored guidance
• Keep a running log with a summary of what was done every cycle (state, output/structural/strategy/steering aspect) and write out a neat Markdown template in the web UI

I gave that file to Opus 4.8 (I definitely still don’t trust Opus to write code, but I did want a different, complementary model steering Codex 5.5), told it which sessions to monitor, and let it go on its merry way.

For this particular case, I did have to intervene once or twice to highlight rendering and palette issues (which I can do in piclaw’s web interface on my iPad), but that was it:

And I think this approach has legs–I’m now using it to grind through the porting/testing/quality aspects of other things I’m doing, and will eventually try it with local models (if I ever get good enough hardware to run them).

Note that I did not want to create a fancy multi-agent system where every agent talks to each other: I wanted to have long-term oversight and steering.

And this is not a delegation pattern either (there is also a delegate plug-in that allows each session to delegate chores to simpler models).

I deliberately chose this approach because, in general, I’ve found multi-agent systems with “party lines” and loose couplings to be a complete waste of tokens unless there is a clear hierarchy and very well scoped outcomes–just like in a human team, really…

This is the (very) abridged story of how I got After Dark running on my own flavour of the Cydintosh–specifically, Flying Toasters on an ESP32-S3 board, zooming along at 65 FPS, which is both completely pointless and one of the more satisfying things I’ve done this month.

Because.

When I first got wind of the Cydintosh, I immediately dug out one of my Cheap Yellow Displays and tried to get the software running on it, only to find out two things:

• My Cheap Yellow Display was (predictably) different (same 240×320, but an ESP32-D0WD)
• The resistive touch screen mine had, together with the relatively small size, made it unusable in practice

I mean, it ran, but… Here, you be the judge:

The photo above was me trying to push the envelope a bit.

The truth is that even with proper 1:1 pixel scaling, portrait rendering and some creative interpretations of how to push faster screen updates through the SPI bus, and even considering the Mac Plus emulation and basic Wi-Fi access worked really well (because of the genius trick of exposing ESP32 hardware through to the emulator), it was painfully slow.

I did the obvious thing and ordered a couple more, larger displays with a slightly more powerful chip and proper capacitive screens:

And the results were glorious: the new boards are labelled ESP32-8048S043C, and they come with an 800×480 capacitive panel, an ESP32-S3, and 8MB PSRAM, which is more than enough to run the emulator and a more complete version of Mac OS, running the original Musashi-based umac emulator at a fairly good speed (certainly faster than the original Mac Plus or even the Classic) and at full panel screen resolution (480×800) in black and white:

However, the original Cydintosh image was ancient. The emulator worked OK, but I needed to do a bunch of upgrades to the firmware on my personal fork:

• Rewrote the display layer to support multiple board profiles, with the S3 target running a 480×800 framebuffer rotated to landscape
• Bumped emulated Mac RAM to 1MB (the original used 128KB, barely enough for System 3.2)
• Fixed portrait/landscape orientation, touch mapping, and the GT911 capacitive driver
• Added an MQTT layer so the thing can actually do smart-home control (the original project’s raison d’être)
• Patched umac to suppress Sony eject requests so disk mounting worked properly
• Built an impromptu web flasher and a serial log capture tool, because flashing ESP32-S3 devices is its own special kind of hell

Getting there took a few weeks of evenings and a bit of creative hacking.

Everything went pretty swimmingly until I actually injected After Dark into the System folder:

Without any way to input text (and no, Key Caps doesn’t really work for this), I had to resort to removing bits from the control panel with ResEdit until it mostly worked (removing the dialog was not trivial, and in the end I asked Codex to just disassemble the INIT(?) resource and skip over the dialog).

A few minor firmware tweaks later, I have a “real” Mac that runs After Dark as a screensaver perfectly:

In case you’re wondering, the toasters fly at a solid 65-67 FPS.

Which is faster than they ran on my actual Macintosh SE/30 back in 1991, because that machine was doing it in 1-bit black and white on a 68030 and this is a $15 display board doing it in 16-bit colour on a dual-core 240MHz chip with more RAM than my first three computers combined.

Not too shabby, even if it’s in black and white (which is what the original Mac ROM can handle).

And that is why I have gone down the rabbit hole of trying to port this to an LC ROM, on yet another ESP32 board (a P4). It turns out there’s not enough RAM on the other boards to hold a colour frame buffer, not enough bandwidth to do the delta refresh hacks I did in the original version, and not enough CPU power and storage for the required emulation and system changes…

I’m still grinding through the mechanics of paring down BasiliskII to fit, but in the meantime this board is sitting on my desk doing nothing but rendering endless toasters and reminding me some things are worth the sheer fun involved every time I glance at it.

This was the weirdest WWDC26 keynote in a while, and some of the past ones were visibly phoned in. It was rife with weirdness and flashbacks.

To my surprise, a few of my wish list items actually made it. Naming the next macOS “Golden Gate” was not on my bingo card, though; a little too trippy and a lot too lofty for what is, by Apple’s own tacit admission, a Snow Leopard year: catching up rather than charging ahead.

The self-deprecating tone ran through the whole thing, from a hippy bus that was equal parts weird and funny to the unmistakable sense of a company that spent the past year watching the industry sprint past it on AI and is now, not running but sedately pacing, to catch up.

Much to my surprise, two of my top annoyances got airtime: they’re tackling Spotlight and Mail search, the exact failures I called out, although whether either works once it ships is anyone’s guess.

They’re also doubling down on automation, at least superficially, with vibecoded Shortcuts and a renewed push for third-party Actions. Vibecoding Safari extensions and Shortcuts is the genuinely interesting part: it points at automation rather than novelty, which is more than I can say for yet another Image Playground. None of it erases the brittleness and legacy gaps that made me want a real platform to begin with, but it’s at least pointing the right way. Tab grouping and change detection in Safari are a fun party trick, no more.

And yes, there’s a new, as-yet-unproven Siri (with a completely pointless AI moniker) you summon by holding the power button (part Spotlight, part walkie-talkie, plus a floating gelatinous orb in Vision Pro), and a Siri app trying to be a catch-all bucket for every interaction.

The new voice struck me as a little cringe and overly American, which is an odd note to land on when you want me talking to my machines all day. The feature set is fuzzy: on paper it can touch far more of my data, and moving photos to the shared library by voice would be neat if it works. But Siri has been stuck at “if it works” for fifteen years, and the one thing I actually want (for it to handle my mail and calendar properly) wasn’t demoed in any useful detail.

I wondered whether the automation push would reach HomeKit, and the answer is a shrug: the new camera detection is cute, but a YOLO model has done exactly that for a decade, and the automation logic I actually need stays vague. The rest of my list didn’t show at all: no hypervisor on the iPad, no running my own code without the annual toll, nothing on iCloud sync, the Watch, or SwiftUI. Maybe the sessions turn something up (which is why this is an early read), but my expectations haven’t budged.

The framing around Apple Foundation Models was the bigger tell: we already know there’s Gemini underneath, which leaves me wondering how much Apple is adding beyond the wrapper. Liquid Glass got the same treatment by being walked back in the most face-saving way imaginable, with the old Accessibility transparency slider re-warmed and trotted out as an improvement. Disingenuous is the word, twice over.

Update: Also much to my surprise, they actually mentioned unifying the corner radii, which I completely missed. I must have tuned it out after the 300 random percentage performance improvements they quoted against… no real baseline, really.

Anyway, Apple heard the parts of everyone’s complaints that a) did not force them to walk back Liquid Glass and b) fit the AI story it needed to tell, and stayed quiet on a lot of the boring structural stuff that’s been broken for years. Yes, they are committing to improving performance and fixing some of the most egregious issues, and that’s not nothing; hearing Spotlight and Mail search admitted out loud is more than I expected, but it is mostly Apple’s technical debt catching up with them, and, of course, Apple catching up with everyone else where it regards AI, but on its own terms and at its own pace.

Oh, and they deprecated pretty much all of my hardware, too. Kind of expected, much like the usual geographical restrictions, which mean a good chunk of this may not reach Portugal for a year, if at all.

I’m going to give it a couple of days until the dust settles, watch the Platforms State of the Union tomorrow, and then mull things over a bit more. And maybe, somehow, we can chalk up this WWDC as a sort of a win, in the long run.

Michael Tsai’s annual roundup of WWDC wish lists went up this week, and the thing that struck me most wasn’t any single request–it was the mood. There seem to be fewer wish lists than last year, several people openly admitted they couldn’t be bothered to write one, and the ones that did are pretty much bereft of any “aspirational” wishes.

Read More...

Since today is a bank holiday for me, I decided to consolidate a few more of my notes into a post. What follows is a set of guiding “principles” that I’ve found useful over the past year or so and that I’ve codified into various bits of scaffolding I reuse across my projects.

Read More...

Allergy season is finally fading (at least for me), but today was the first time I had to turn on the AC in the office, and it was great to realize that despite the recent Wi-Fi changes and almost four years of potential HomeKit foibles, my ESP32 hack is still working perfectly.

Read More...

A few months after writing up the Cudy AX3000 units and moving the house over to OpenWRT, I ended up revisiting the one bit I had deliberately waved away as “good enough”: roaming.

Read More...

I know this blog has strayed a fair distance from its Mac-centric origins, but I’ve been keeping a mental list of all the things that are broken, missing or inexplicably neglected in Apple’s software, and it’s gotten long enough that writing it down feels like a public service¹.

Read More...

I’m done waiting for Apple to fix things. And one of the things I think should exist is a decent way to run Linux binaries on my iPad.

Read More...

Last weekend my DS1019+ decided, for some unfathomable reason, to stop working after I took it out of the closet, dusted it and put it back, and I have feelings about it.

Read More...

The Ternus announcement got me thinking about the one thing I keep wishing Apple would build and almost certainly never will: a family-scoped AI assistant that actually works across all our devices.

Read More...

I’ve been mulling the UX arc I’ve been going through over the past couple of years, and I think it was mostly the same for everybody:

Read More...

Regular readers will know that I’ve spent most of the past two years shoehorning LLMs into single-board computers, partly as a learning exercise and partly because there are lots of local/”edge” applications where semantic reasoning (no matter how limited) and “interpretation” of sensor data are actually useful.

Read More...