28 Aug 2014 idcmp   » (Journeyer)

Backing up OS X onto Linux. Finally.

I've tried all sorts of black voodoo magic to make this work, but finally I have something repeatable and reliable. Also, please excuse the horrible formatting of this post.

History

I started with Time Machine talking to my Linux box with netatalk. This worked fine until one day Time Machine told me it needed to back up everything all over again.

Then it started to tell me this almost weekly.  Apparently when this happens, the existing backup is corrupt and all that precious data is at worst irretrievable or at best tedious to retrieve. These are not attributes I want associated with my backup solution.

Then I did an rsync over ssh to my Linux box.  This is fine exception that it lacks all the special permissions, resource forks, acls, etc, etc that are hidden in a Mac filesystem.

Then I tried SuperDuper! backing up to a directory served up via netatalk mounted via afp:// on OS X.  This worked, but was mind numbingly slow. Also, it would mean I'd have to pay for a tool if I wanted to do incremental backups. This gets expensive as I also back up a few friends OS X laptops on my Linux file server.

I tried SuperDuper! backing up over Samba, but hdiutil create apparently doesn't work over Samba. Workarounds all needed the purchased version of SuperDuper!.

There's *another* work around for SuperDuper! where I can use MacFUSE and sshfs, but the MacFUSE author has abandoned the project and recommends people not to use it.

Sheesh.

The Solution

Ultimately, the goal is to make a sparsebundle HFS+ disk image, put it on a Samba mounted share and rsync my data over to it. You'd be surprised how many niggly bits there are for this.

Install Rsync


First, I grabbed the 3.1.x version of rsync from Homebrew - install Homebrew as per the directions there, then run:

brew install https://raw.github.com/Homebrew/homebrew-dupes/master/rsync.rb

If you've been digging through voodoo magic, then you'll be happy to hear this version of rsync has the all the rsync patches you'll read about (like --protect-decmpfs).

Samba

Nobody needs another out of date blog entry explaining how to setup Samba. Follow some other guide, make sure Samba starts automatically and use smbpasswd to create an account.  

I recommend using the name of the machine being backed up as the account name. I'm calling that machinename for the rest of this post.

Make sure you can mount this share on OS X via smb:// ( Finder > Go > Connect to Server... ).  Make sure you can 1) create a file, 2) edit and save the file, 3) delete the file.  I'm going to assume you've mounted this share at /Volumes/Machinename

Backup Into

Now lets make something for us to backup into.  Figure out how big the disk is on the source machine (we'll assume 100g) then run:

hdiutil create /tmp/backup.sparsebundle -size 100g -type SPARSEBUNDLE -nospotlight -volname 'Machinename-Backup' -verbose -fs 'Case-sensitive Journaled HFS+'

Yes, you're creating it in /tmp, this is to work around hdiutil create not liking Samba.

Next you'll want to copy this sparse bundle onto your Samba share:

 cp -rvp /tmp/backup.sparsebundle /Volumes/machinename


This will copy a bunch of files and should be successful without any warnings.  Now lets mount this sparse bundle:

hdiutil attach /Volumes/machinename/backup.sparsebundle -mount required -verbose

You should now have /Volumes/Machinename-Backup mounted on your system. Fun story, OS X recognizes that this disk image is hosted off the machine, so it mounts this disk image with "noowners" (see mount man page). That's going to be a problem for our backup, so we need to tell OS X it's okay to use userids normally:

sudo diskutil enableOwnership /Volumes/Machinename-Backup

Preparing Rsync

There are a handful of files which recommend to be excluded:


.DocumentRevisions-*/
.Spotlight-*/
/.fseventsd
/.hotfiles.btree
.Trashes
/afs/*
/automount/*
/cores/*
/private/var/db/dyld/dyld_*
/System/Library/Caches/com.apple.bootstamps/*
/System/Library/Caches/com.apple.corestorage/*
/System/Library/Caches/com.apple.kext.caches/*
/dev/*
/automount
/.vol/*
/net
/private/tmp/*
/private/var/run/*
/private/var/spool/postfix/*
/private/var/vm/*
/private/var/folders/*
/Previous Systems.localized
/tmp/*
/Volumes/*
*/.Trash
/Backups.backupdb
/.MobileBackups
/.bzvol
/PGPWDE01
/PGPWDE02

Store this in a file somewhere. I stored mine as exclude-from.txt in /Volumes/Machinename

Okay, now we're ready to run rsync.  I think the correct arguments to rsync are: -aNHAXx --nfs-compression --protect-decmpfs --itemize-changes --super --fileflags --force-change --crtimes

So, we run:

rsync all-those-args-above --exclude-from=/Volumes/Machinename/exclude-from.txt / /Volumes/Machinename-Backup/

When The Backup Is Done

This will take a little while. When it's done, you can then bless your backup so it could be booted:

sudo bless -folder /Volumes/Machinename-Backup/System/Library/CoreServices

Then you can umount your backup:

hdiutil detach /Volumes/Machinename-Backup

Periodically, and after your first run, you should compact down your sparsebundle disk image:

hdiutil compact /Volumes/Machinename/backup.sparsebundle -batteryallowed

You can now log into your Linux server and tar up the backup (apparently XZ > bzip2 for compression size).

 tar Jcvf machnename-backup.tar.xz backup.sparsebundle

Depending on the size of that tarball, you could upload it to Google Drive, Drop Box, etc. Before you do, you'll probably want to encrypt it. I used OpenSSL:

openssl aes-256-cbc -a -salt -in machinename-backup.tar.xz -out machinename-backup.tar.xz.aes-256-cbc

Many of these steps will take *hours* if you have a lot of data, so you may consider just backing up parts of your system more frequently and doing your whole system once every-so-often.

Why This Is Nice

One thing I really love about this setup is that each piece of the puzzle does one thing and does it well. If Samba is too slow, I could go back to netatalk. The sparsebundle disk image hosts HFS+ properly with all its OS X-specific voodoo and rsync's job is to copy files. If there's a better file copier, I could drop that in.

Conclusion

I left a lot out. I know. I'm kind of expecting you to have a rough idea of how to get around OS X and Linux, figure out how to put most of the above in a shell script, decide when to do backups, how to store those tarballs, etc. Hopefully though this will help someone that just needs some of the key ingredients to make it work.  



Syndicated 2014-08-27 23:49:00 (Updated 2014-08-29 17:52:35) from Idcmp

Latest blog entries     Older blog entries

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!