Somehow (thankfully) I've managed to avoid much to do with backups in my career as a sysadmin. It's always been someone else's responsibility, that's been fine by me.
For some reason, I've been more paranoid lately, and with all of the crazy shit I've been doing to daedalus, I wanted to have a backup, just in case. Maybe it's old age.
So a while ago, I went and had a stab at setting up Amanda, since pretty much whenever I've been near a bunch of sysadmins and someone asks for a recommendation on what to use, "Amanda" is the answer.
I jumped through all of the hoops to have caesar be the backup server, and daedalus be a client. I set up a virtual tape library on caesar. I think I then lost interest for a while, until I was going to upgrade daedalus to Etch, and I wanted to have a successful backup first.
I had the holding disk and the virtual tape library on the same filesystem, and it was nice and big, yet I kept running into this error about insufficient space. "dump larger than available tape space" was the complaint.
I tried making the filesystem larger. I tried adding more virtual tapes. Nothing would placate it. After another round of Googling, I discovered that the problem was because Amanda would only only span a maximum of four tapes, and the filesystem I was trying to backup was way bigger than that. So I told it to use up to 10 tapes in a run (runtapes 10) and then lo and behold, I got a successful backup out of the sucker.
So that was mostly all good, but occasionally a backup would fail, or fail once an succeed on a retry, because tar exited non-zero, with "file changed as we read it". I did some more Googling tonight, and discovered this Amanda/GNU tar compatibility matrix.
It turns out that the version of tar and the version of Amanda shipped in Debian Etch aren't compatible with each other. Suck! Oh wait, they're maintained by the same person! Argh!
So aside from the occasional backup failing due to a file being written to at the time of the backup, it's working fairly well. My DSL link isn't the fastest, so a full backup of daedalus can take over 48 hours to complete, but I'm not paying for inbound bandwidth at home, so it's not a big deal.
I initially thought that to do a restore, I'd have to run amrecover on daedalus, and then I'd have to stream the contents of multiple tapes back to it at the appalling uplink speed of my DSL connection, but fortunately amrecover has a sethost command, so I can restore just the files I need on another machine local to caesar at LAN speeds, and just transfer the files I restore to daedalus. So that's all pretty neat.
I quite like the FTP-client-like interface of amrecover. It's cool how you can use setdate and just go back in time.
I had the opportunity to do a test restore recently when one of my users nuked his Maildir directory by accident. Unfortunately I was unable to recover the needed files in that case, and I'm not entirely sure what the problem was, so there's some more testing to be done before I'll be fully confident that I can rely on it.
There's a lot of configuration options available as well, and I think I got the config I'm currently using from a template or something, so there's probably a fair bit of tweaking I can do.
Oh, and I'm constantly reminded of this quote from a previous life, where technical details bubbled too far up non-technical management:
"Who is Amanda, and what is her role in the backups?"