Backup 3.0

Okay, so I took the plunge and started using to, well, back up my s.

Not being one to use something I don't understand (and having had a lot of bad experiences with backup software in the past across every platform), I took my time and figured out as much of its innards as I could, taking notes along the way.

The Master Plan

It was fairly easy to define a "Home Folder" profile that backed up everything except my data, as well as excluding stuff like the cache, , Thunderbird and Firefox profile info, and local cached messages (I just had to hunt and peck them off the Library).

Another notorious offender, the Microsoft User Data folder (that , in a bout of asinine design, stuffed inside Documents rather than under the Library) was picked off in the same way.

The whole thing (mostly preferences, heaps of source code, site artwork and miscellaneous documents) weighed in at 3.6GB after pruning, and coped well with my removing a couple of files and directories manually (i.e., using the command-line), automatically un-listing them from the backup roster.

So, What's In a Package?

For the curious, here is a file tree for a snapshot (which is nothing more than a package containing some metadata and a .sparseimage:

foo.FullBackup/ --- Contents -+- Backup.catalog -+- cat
                              |                  |
                              |                  +- stat0
                              +- backup.log.gz
                              +- InfoPlist.strings
                              +- Contents --- Backup.sparseimage

cat seems to be a straightforward NSData serialized stream listing all of the files (essentially a cache, since you can get mostly the same information from scanning the .sparseimage), but I have no idea of what stat0 is.

The .sparseimage seems to be appended to directly (without a temporary file cache) for disk-based backups (it is stored under /private/tmp for DVD-based ones, and the whole thing is then burned at once).

The good news from this is that, in a pinch, you can mount it straight from the .

The bad news is that, from this moment on, your backups are completely -centric, since there is no trivial way to deal with this file format in other OSes - I don't think that matters much to most people, but to me it's a bit of an annoyance, since I much prefer platform-agnostic backup file formats - after all, you never know what sort of machine you'll be using two years from now, and two years is a perfectly sensible recovery window.

I like to aim for ten years (even though the media probably won't last that long), and not having something like a .zip or a .tar archive file seems a bit risky. But hey, that's me.

The gzipped log is just that - a log, listing the files added to an archive, that is appended to and compressed at the end of the backup - and the .plist includes the sort of info you'd expect (backup dates, version numbers, the user that initiated the backup, etc.).

Here's a trimmed sample:

"Media Type" = "Filesystem";
"Ancestry" = ();
"Backup Archive Version" = "2";
"Archive Format" = "DiskImage";
"FinishDate" = "2005-10-02 10:43:46.901 +0000";
"UUID" = "515F035C-5AA6-456B-BC82-07E14B7B9CF5";
"Package Version" = "1";
"Master" = "YES";
"StartDate" = "2005-10-02 10:43:24.608 +0000";
"Source Filesystem Descriptions" = (
"OS Version" = "0x1042";
"Home Directory Real Path" = "/Users/rcarmo";
"Target" = {
    "Backup Configuration" = {Spanning = 1; "Spans to Same Media" = YES; Splitting = 1; };
    "Destination Path" = "/Backups";
    "Filesystem Description" = {
    Name = Hotwired;
    "Target Archive Version" = 1;
    Type = DiskImage;
    UUID = "0222E415-CE82-4E1D-8EC8-406B1030B574";
"User's Full Name" = "Rui Carmo";
"Incremental Catalog" = "NO";
"Application Version" = "3.0";
"Device/Disc Label" = "Hotwired";
"Home Directory" = "/Users/rcarmo";
"Package Size" = "11931374";
"Contained Backups" = ("515F035C-5AA6-456B-BC82-07E14B7B9CF5");
"User's Short Name" = "rcarmo";
"ScheduledStartDate" = "2005-10-02 10:43:24.592 +0000";
"Device/Disc Name" = "Hotwired";
"Archive Info" = {"Archive Components" = ("Backup.sparseimage"); };
"BackupSet" = {
    "BackupSet Archive Version" = 1;
    Name = "Personal Data & Settings";
    QuickPicks = (
        {Key = ""; Version = "1.0"; },
        {Key = ""; Version = "1.0"; },
        {Key = ""; Version = "1.0"; },
        {Key = ""; Version = "1.0"; },
        {Key = ""; Version = "1.0"; }

The .plist gets a lot more interesting as your plan grows, of course, since it starts listing exclusions:

            "Prune Paths" = (
                {doNotCrossFilesystemBoundaries = 1; pattern = "~/dowser"; },
                {doNotCrossFilesystemBoundaries = 1; pattern = "~/Library/Mail"; },
                {doNotCrossFilesystemBoundaries = 1; pattern = "~/Music"; },
                {doNotCrossFilesystemBoundaries = 1; pattern = "~/Desktop"; },
                    doNotCrossFilesystemBoundaries = 1;
                    pattern = "~/Library/Application Support/Camino";

deserves credit for re-using their existing technology in just the right way to make this a near-trivial and relatively open format, and all in all, I think that the packages have pretty much everything you need to deal with them manually if needed be (provided you have a ), so your data should be relatively safe and somewhat future-proof.

Incremental vs. Full

did a full backup the first time I tested it and then started doing incremental backups for each file set, which is nice.

But here's the flip side: when I removed the full backup file from the destination folder, it didn't notice anything and just went on doing incremental backups - which seems to be a great way to lose a lot of data if you're not careful.

Thankfully, you can force a full backup from the File menu (but, for some reason, not from the plan dialog boxes).

My guess is that it relies on the backup plan state stored on your own machine and only looks at the actual full/incremental files when restoring (or at least I hope it does), so you might want to force full backups once in a while.

The Good, The Bad and The Ugly

Overall, the experience was pretty much painless, and it seems to do the job adequately. Other folk will surely post their own opinions, flaws and caveats, but for now, here are mine:

Pretty and Simple

In typical  fashion, there were a lot of nice UI touches - such as the partially colored sunshade icon used for temporary files, the detailed history list of tried and failed (aborted or interrupted) backup attempts for each set, and the scheduled backups of personal data and settings (but come on, 1GB is nowhere near the sort of storage I'd expect from a paid service - add a zero and I'd find this a whole lot more interesting).

A particularly nice touch is that, when backing up to CD or DVD, prompts you with a "Work To Do" dialog saying "8 CDs or 1 DVD". It supports non- recorders, too, and the dialog expands to let you pick speed and erase options for re-writable media, as well as other settings (verification after write is the default).

It does not seem to be much aware of the media size, though (i.e., I don't think it distinguishes between 650MB and 800MB CDs).

Restoring seems functional (I only restored a couple of files for testing), and lets you pick optional destinations. Since I did not have to split backup sets across discs, I have no idea of how it handles that.

Could be Faster and Smarter

has an unwelcome tendency to re-scan the whole file set even when you haven't made any changes to it (I suppose they're not using file system change notifications), so that if you go in, check the file set and then return to the backup plan dialog without making any changes, it will not update the backup size instantly - no, you have to wait a good deal until it re-displays the overall set size.

It's a pain, even if you do change something trivial (like removing a single file from the set), and I shudder to think how long it will take on my iBook (which isn't backed up yet, mostly because around 99% of the data on it is replicated on my other machines).

On my G5, scanned my whole home directory much faster than - by what seemed to be a factor of two.

There are plenty of other things to improve - for instance, removable media backups need to have better UI feedback to be able to give a better forecast for image build, burn and verification stages, and I'm a bit miffed that I can't run for other users by entering an administrative password, but that's a minor detail (I guess most people will have the administrative user run for them anyway).

Still, it's something that I feel that should be there (if I can install things by entering the administrative password, why can't I back up other users' files by authenticating myself in the same way?), so I went and sent that feedback in.

Sadly, still had not upgraded the feedback form to support 3.0 at the time of writing. Ah well.

Some Feature Creep Required

Being able to specify full backups as the default would be nice, too, as well as a more platform-neutral backup file format, but that's my mindset wanting ever so much more control over things.

And speaking of , I'm disappointed with what does with "dot files" (the traditional way to store preferences), since they aren't listed anywhere (nor is there an option to show hidden files and folders).

I know they get backed up (they show up in the log), but restoring them seems to be a bit tricky (I suppose I can always mount the image in ).

Hints and Tips

Finally, here are a few hints and tips that should make it easier for you (these will eventually be relocated to the node and expanded upon):

  • If you own several s and are backing up to the same location(s), you definitely want to rename your backup plans to something a bit more descriptive than the defaults - include your name and machine name on the backup plan's name, otherwise you'll have trouble later - adds timestamps to the file names and doesn't seem to get confused, but "Rui's Home Folder on Desktop" is a lot harder to confuse than a bunch of almost identical "Home Folder" sets.
  • Check if your applications have plugins available. For instance, there's this plugin for Delicious Library that creates a custom "quick pick" list for its data, and it may make sense for you to back up specific applications' data in different ways.
  • Grab and use it to figure out how you want to build your backup plan - you will find that you have a lot more stuff than you were expecting, and that you probably don't need to back up everything. Besides the usual media folders, you'll find that there is a lot of stuff inside the Library folder that you can probably do without (although that depends a lot on what sort of applications you use).
  • If you're backing up to removable media, you will need at least as much free space as the size of the media ( generates a temporary disk image under /private/tmp and then burns that to the DVD, which means it requires around 5GB of free space). It apparently does so incrementally when splitting across several discs (i.e., it generates one image, burns it, ejects the disc, generates the next one, and so on), but I couldn't confirm that.

This page is referenced in: