Running NixOS on Proxmox LXC

As a sort of quick follow-up to my earlier post on running NixOS on ARM LXC/LXD I decided to do the same on Proxmox and document the process for its LXC environment.

Although I have both ARM and Intel machines running Proxmox, I focused on Intel since I wanted to replace the NixOS VM I had running with something lighter, and LXC containers have essentially no overhead. Doing the same for my ARM servers should be almost exactly the same.

Just as before, I decided to live on the bleeding edge and use a trunk image.

So I visited Hydra and downloaded the latest nixos-system-x86_64-linux.tar.xz tarball directly into the LXC templates shared storage I have set up on my Proxmox cluster (it’s also the backup storage, but that’s a detail).

This is where I ended up spending most of my time, since there is one significant catch to the process: Proxmox likes to run various hooks when creating containers, and these are simply not there in the NixOS tarball.

So I had to create the container manually, and this was the command I ended up using:

❯ pct create $(pvesh get /cluster/nextid) \
  --arch amd64 \
  backup:vztmpl/nixos-system-x86_64-linux.tar.xz \
  --ostype unmanaged \ # this prevents Proxmox from trying to use LXC hooks
  --description nixos \
  --hostname nixos \
  --net0 name=eth0,bridge=vmbr0,firewall=1 \
  --storage local-lvm \
  --unprivileged 1 \
  --features nesting=1 \
  --cmode console \
  --onboot 1

This was pieced together from various sources, but the key points are that besides using the --ostype unmanaged flag, I also had to go back to the Proxmox UI to set resource sizes and fix the network settings to use DHCP.

Don’t forget to increase the volume size from the default 4GB, otherwise you’ll run out of space during the very first build.

Now comes another fun part: Although the container images now have a working Proxmox console (which used to be a problem in earlier releases), the default configuration.nix is now actually set up for LXD, not LXC (which was a bit of a surprise since that’s changed since my ARM adventure). It’s only been a month, but things seem to move fast in the NixOS world.

Also, even though it’s built for container use, the tarball still has a lot of stuff that doesn’t work inside an LXC environment, so I had to disable a few systemd units.

But first, it’s a good idea to update the system and set a root password:

❯ nix-channel --update
❯ passwd

Then comes the part I dread, which is using nano (I just detest the keybindings and wish NixOS bundled vim) to edit the configuration:

❯ nano /etc/nixos/configuration.nix

After a couple of iterations, this is what I ended up with–note that I had to comment out the lxd.nix import and that besides temporarily enabling password-based ssh login for root, I kept the origintal NixOS networking settings:

{ modulesPath, config, pkgs, ... }:

{
  imports =
    [
      # Include the default lxc/lxd configuration.
      "${modulesPath}/virtualisation/lxc-container.nix"
      # Include the container-specific autogenerated configuration.
      #./lxd.nix - this has to be commented out from the system tarball
    ];
  boot.isContainer = true;

  # I had to suppress these units, since they do not work inside LXC
  systemd.suppressedSystemUnits = [
    "dev-mqueue.mount"
    "sys-kernel-debug.mount"
    "sys-fs-fuse-connections.mount"
  ];

  # A few packages I like to have around
  environment.systemPackages = with pkgs; [
    openssh
    vim
    tmux
    htop
    binutils
    man
    starship
  ];
  programs.zsh.enable = true;
  users.defaultUserShell = pkgs.zsh;

  # Enable password-based SSH login for root
  services.openssh = {
    enable = true;
    settings = {
      AllowUsers = null; # everyone
      PasswordAuthentication = true; # this is just a sandbox
      PermitRootLogin = "yes";
    };
  };

  # from here on it's the same as the default tarball configuration

  networking = {
    dhcpcd.enable = false;
    useDHCP = false;
    useHostResolvConf = false;
  };

  systemd.network = {
    enable = true;
    networks."50-eth0" = {
      matchConfig.Name = "eth0";
      networkConfig = {
        DHCP = "ipv4";
        IPv6AcceptRA = true;
      };
      linkConfig.RequiredForOnline = "routable";
    };
  };
  system.stateVersion = "24.11"; # Did you read the comment?
}

Once this is done, you can proceed as usual:

❯ nixos-rebuild switch --upgrade

For now, I have backed up the container as-is (since it will be easier to clone it than to start from the template again) and am moving stuff from my NixOS VM to it. I’ll probably revisit this in a few weeks since I need mDNS and a few other things that I haven’t had time to set up yet, but at least I have a working base to build on.

I’m a bit surprised that the NixOS team has apparently moved so quickly to LXD as the default container environment and I wish I could get container creation to work via the Proxmox GUI (just for the sake of consistency with the other images I use), but at least the process is straightforward once you understand the quirks.