# HOWTO Run Vagrant with LXC on your Mac

Update: I’ve since moved from [VirtualBox][vb] to [Parallels][pa], and updated this accordingly. Also, [vagrant-lxc][vl] has now been upgraded to 0.4.0, which means there are some (very minor) differences.

After fooling around with [Vagrant][v] and [LXC][l] for a few days (thanks to [Fabio Rehm’s][fr] awesome [plug-in][vl]) , I decided to pull everything together into a working, reproducible setup that allows me to develop and test on four different [Linux][li] distributions, as well as prepare [Puppet][p] scripts for production deployments – and all without the hassle of running multiple [VirtualBox][vb] instances on my Mac,

If you’re running [Linux][li] natively, you can skip ahead to the last few sections, since most of this is about setting up a suitable [Linux][li] VM to host all the containers on the Mac.

By using [LXC][l] inside one “host” VM, this setup uses far less resources – and with a few tweaks, it also lets me access my Mac’s file system from both inside the parent VM and from the [Vagrant][v] environments running as containers inside that VM:

In a nutshell, the advantages are:

• Run everything on a single VM – as many different [Linux][li] distros as you want
• Run multiple machines simultaneously with nearly zero overhead
• Significantly less clutter on your hard disk (boxes stay inside the VM)
• No need to install [Vagrant][v] directly on your Mac
• Your Mac’s filesystem is transparently mounted inside each environment with zero hassle

Assuming you have [VirtualBox][vb] or [Parallels][pa] set up already, here’s how to go about reproducing this:

## Set up a suitable “parasite” host

The first step is to set up an [Ubuntu][u] 64-bit VM with an extra host-only network interface. Here’s the relevant screen for [VirtualBox][vb]:

This uses [VirtualBox’s][vb] built-in DHCP server and will be the interface through which you’ll connect to the VM.

And here’s the equivalent in [Parallels][pa]:

The reason I prefer setting up a second interface is that I’d rather use this than fool around with port mapping in the default interface (which I use to NAT out of my laptop), and it mirrors around 99% of my production setups (i.e., a dedicated network interface for management)

Now grab the Ubuntu MinimalCD image and do a bare minimum install, setting the host name to something like containers and skipping everything else – i.e., Choose only OpenSSH Server near the end).

(For the purposes of this HOWTO, I’m going to assume you created a local user called… user – replace as applicable)

I recommend that you don’t set up LVM (it’s somewhat pointless in a VM, really), don’t mess with the defaults, and go with 13.04 or above (I started out using this with 12.04 LTS and had a few issues with 13.04, but those seem to have been mostly fixed in the latest updates).

## Required Packages

Once you’re done installing, it’s package setup time:

sudo apt-get update
sudo apt-get dist-upgrade

# grab the basics we need to get stuff done
sudo apt-get install vim tmux htop ufw denyhosts build-essential

# grab vagrant-lxc dependencies
sudo apt-get install lxc redir

# the easy way to ssh in
sudo apt-get install avahi-daemon


Once that’s done, install the [VirtualBox][vb]/[Parallels][pa] guest software in the usual way (i.e., use the relevant menu option to mount the CD image, run the installer as root, reboot the VM, etc.)

## Networking tweaks

Now for a few networking changes that make it simpler (and faster) to access your new VM.

First off, disable DNS reverse lookups in sshd by adding the following line to /etc/ssh/sshd_config (they’re pointless in a local VM and cause unnecessary connection delay):

UseDNS=no


Then set up eth1 by editing /etc/network/interfaces to look like so:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

auto eth1
iface eth1 inet dhcp


## The Easy Way In

Now we’ll set up avahi so that your VM announces its [SSH][s] service to your Mac using [Bonjour][b] – to do that, create /etc/avahi/services/ssh.service and toss in this bit of XML:

And, if you’re planning on doing web development, I also suggest setting up /etc/avahi/services/http.service with something like:

To get avahi to start in a headless machine, you need to disable dbus support in /etc/avahi/avahi-daemon.conf – which, incidentally, is also the right place to ensure [Bonjour][b] advertisements only go out via eth1.

So you need to add these two lines to that file:

allow-interfaces=eth1
enable-dbus=no


Now you’re ready to enable the network interface:

sudo ifup eth1
sudo service avahi-daemon restart


If you use ufw, you must open all traffic to lxcbr0:

sudo ufw allow in on lxcbr0
sudo ufw allow out on lxcbr0


…and change DEFAULT_FORWARD_POLICY to "ACCEPT" in /etc/default/ufw - otherwise your host system won’t be able to talk to your containers, and they won’t be able to reach the outside.

## The First Neat Bit

You now have a nice, clean way to login to the machine – you can always connect via ssh [email protected]/* <![CDATA[ */!function(t,e,r,n,c,a,p){try{t=document.currentScript||function(){for(t=document.getElementsByTagName('script'),e=t.length;e--;)if(t[e].getAttribute('data-cfhash'))return t[e]}();if(t&&(c=t.previousSibling)){p=t.parentNode;if(a=c.getAttribute('data-cfemail')){for(e='',r='0x'+a.substr(0,2)|0,n=2;a.length-n;n+=2)e+='%'+('0'+('0x'+a.substr(n,2)^r).toString(16)).slice(-2);p.replaceChild(document.createTextNode(decodeURIComponent(e)),c)}p.removeChild(t)}}catch(u){}}()/* ]]> */, regardless of which address your virtualizer’s DHCP server grants the VM, and in case you ever have to put the VM on a LAN to work from another machine, you can bridge eth1 and things will just work™.

## Housekeeping

If you’re going to be using the VM for development alone, it’s rather convenient to set up password-less sudo by tweaking /etc/sudoers to read:

%sudo   ALL=(ALL:ALL) NOPASSWD: ALL


Also, and since we’re going to rely on [VirtualBox’s][vb] shared folders, we need to change the vboxsf group’s GID to 1000 and add your user (user in this example) to it in /etc/group:

vboxsf:x:1000:user


This doesn’t seem necessary for [Parallels][pa], but I recommend you check the permissions on /media/psf regardless.

This is key to getting shared folders working nicely with [Vagrant][v], because [VirtualBox][vb] adds the vboxsf as the next available GID on the VM (which is 1001 under [Ubuntu][u] on a fresh install) but the vagrant UID/GID is typically 1000 (since it’s the first user created in a guest system).

So you need to make sure these match, or else you’ll have all sorts of problems with file permissions on shared filesystems.

You should also tweak /etc/hosts and /etc/hostname accordingly:

$cat /etc/hosts | head -2 127.0.0.1 localhost 127.0.1.1 containers.yourdomain.com containers$ cat /etc/hostname
containers


…and, finally, placing your SSH key into the user‘s authorized_keys file (manually, since the Mac, alas, doesn’t ship with ssh-copy-id:

me@mac:~$cat ~/.ssh/id_rsa.pub | pbcopy me@mac:~$ ssh containers.local
Password:
Last login: Tue May 28 17:28:32 2013 from 192.168.56.1
user@containers:~$mkdir .ssh; tee .ssh/authorized_keys # Cmd+V - Ctrl+D user@containers:~$ chmod 400 .ssh/*


## Installing [Vagrant][v] with [LXC][l] support

After this is done, you should be able to fire up your VM and [SSH][s] to it by typing:

ssh [email protected]/* <![CDATA[ */!function(t,e,r,n,c,a,p){try{t=document.currentScript||function(){for(t=document.getElementsByTagName('script'),e=t.length;e--;)if(t[e].getAttribute('data-cfhash'))return t[e]}();if(t&&(c=t.previousSibling)){p=t.parentNode;if(a=c.getAttribute('data-cfemail')){for(e='',r='0x'+a.substr(0,2)|0,n=2;a.length-n;n+=2)e+='%'+('0'+('0x'+a.substr(n,2)^r).toString(16)).slice(-2);p.replaceChild(document.createTextNode(decodeURIComponent(e)),c)}p.removeChild(t)}}catch(u){}}()/* ]]> */


And be magically there. So now you can install [Vagrant][v] and [Fabio Rehm’s][fr] awesome plug-in by doing something like:

wget http://files.vagrantup.com/packages/0219bb87725aac28a97c0e924c310cc97831fd9d/vagrant_1.2.4_x86_64.deb
sudo dpkg -i vagrant_1.2.4_x86_64.deb
vagrant plugin install vagrant-lxc


…which is all you need to get started. To get suitable boxes and start working, check out the plug-in README, and this article on building your own base boxes.

## The Second Neat Bit

Now set up a [VirtualBox][vb] shared folder pointing to your home directory ([Parallels][pa] does this by default) – it will show up as a /media/sf_username mount inside the VM.

The really neat bit is that if you cd into that, set up a Vagrantfile, do a vagrant up --provider=lxc and then vagrant ssh into the [LXC][l] container, you’ll be able to access your Mac’s file system via /vagrant from inside the container.

Wrap your head around that for a second – you can develop on your Mac and test your code on a set of different [Linux][li] environments without ferrying copies across. And, of course, you can use [Puppet][p] to provision those environments automatically for you.

## Port Forwarding

Finally, a small fix - to be able to access forwarded ports from your containers on versions 0.3.4 and 0.4.0 of the plug-in, you need to tweak it a bit:

cd ~/.vagrant.d/gems/gems/vagrant-lxc-0.4.0/lib/vagrant-lxc/action

vi forward_ports.rb

# now edit the redirect_port function to remove the 127.0.0.1 binding like so:

def redirect_port(host, guest)
#redir_cmd = "sudo redir --laddr=127.0.0.1 --lport=#{host} --cport=#{guest} [email protected]/* <![CDATA[ */!function(t,e,r,n,c,a,p){try{t=document.currentScript||function(){for(t=document.getElementsByTagName('script'),e=t.length;e--;)if(t[e].getAttribute('data-cfhash'))return t[e]}();if(t&&(c=t.previousSibling)){p=t.parentNode;if(a=c.getAttribute('data-cfemail')){for(e='',r='0x'+a.substr(0,2)|0,n=2;a.length-n;n+=2)e+='%'+('0'+('0x'+a.substr(n,2)^r).toString(16)).slice(-2);p.replaceChild(document.createTextNode(decodeURIComponent(e)),c)}p.removeChild(t)}}catch(u){}}()/* ]]> */_ip} 2>/dev/null"
redir_cmd = "sudo redir --lport=#{host} --cport=#{guest} [email protected]/* <![CDATA[ */!function(t,e,r,n,c,a,p){try{t=document.currentScript||function(){for(t=document.getElementsByTagName('script'),e=t.length;e--;)if(t[e].getAttribute('data-cfhash'))return t[e]}();if(t&&(c=t.previousSibling)){p=t.parentNode;if(a=c.getAttribute('data-cfemail')){for(e='',r='0x'+a.substr(0,2)|0,n=2;a.length-n;n+=2)e+='%'+('0'+('0x'+a.substr(n,2)^r).toString(16)).slice(-2);p.replaceChild(document.createTextNode(decodeURIComponent(e)),c)}p.removeChild(t)}}catch(u){}}()/* ]]> */_ip} 2>/dev/null"

@logger.debug "Forwarding port with #{redir_cmd}"
spawn redir_cmd
end


And that’s it, you’re done. Enjoy.

# Here Be Dragons

Another thing I’m experimenting with is using [btrfs][bt] - [Ubuntu][u] 13.04 ships with a version of [LXC][l] that is [btrfs][bt]-aware and significantly further reduces the amount of disk space required (lxc-clone actually performs a filesystem snapshot, which is terrific). That’s a little beyond the scope here, but it’s worth outlining here for ease of reference.

## Setting up [Ubuntu][u] with btrfs

If you really want to try that, here’s a short list of the steps I’ve followed so far:

• Create a new VM with 512MB or RAM, two network interfaces (as above) and three separate VDI volumes:
• 2GB for /
• 512MB for swap
• 8GB for /var
• Install from the [Ubuntu][u] 13.04 Server ISO
• Set up the partitions using [btrfs][bt] for both data volumes (/ and /var), with noatime set (that’s about the only useful flag you can set inside the installer)
• Once the partitioner sets up your volumes, switch to a new console and remount them with:

• Once you’ve installed the system and before rebooting, go back to the alternate console and edit /etc/fstab to have noatime,ssd,compress=lzo,space_cache,autodefrag as options to (you can use compress-force=lzo if you want everything to be compressed).
• To get rid of the “Sparse file not allowed” message upon booting, comment out the following line in /etc/grub.d/00_header and run update-grub:

• Proceed as above, and hike up the RAM on the VM to your liking. If all goes well, you’ll have a much smaller VM footprint due to the use of lzo compression, a relatively small swapfile and copy-on-write support.

There are only three caveats:

1. You need to do this for the moment (should be fixed in updated boxes after 0.3.4)
2. The VM will be a little more CPU-intensive when doing disk-intensive tasks (I find it a sensible trade-off)
3. The usual tricks for shrinking disk files on either [VirtualBox][vb] or [Parallels][pa] no longer apply to [btrfs][bt] volumes (so far even [GParted Live][gp] hasn’t really worked for me), so it’s probably best to save a copy of the VM somewhere.