Posted by Scott Laird
Wed, 16 Aug 2006 04:36:14 GMT
I mentioned a few weeks ago that I was trying to add generic database backups to my installer, so I can do database backups before each upgrade. I got backups working a couple releases ago, and they’re useful enough that I’ve extracted them so everyone can use them.
The latest installer release includes a pair of command-line tools that make it easy to use the installer’s backup and restore code without using the rest of the installer. Run rails-backup and it’ll produce a .yml file containing all of your data. Then run rails-restore BACKUPFILENAME and it’ll restore the backup. The dump and restore formats are database-independent, so it should be possible to use these tools to migrate between database engines, but this hasn’t been heavily tested yet.
One warning about the restore–rails-restore doesn’t restore the schema, just the data. So you’ll need a way to build a database with the right schema revision. Look in the .yml file to see which schema version you need (search for schema_info), and then run rake migrate VERSION=xx to move your database to the right schema revision.
To use these, just install the rails-app-installer gem, and they should appear in your path automagically. Report bugs to the rails-app-installer bug tracker.
Tags backups, installer, ruby, rubyonrails | 4 comments
Posted by Scott Laird
Fri, 06 Aug 2004 03:38:22 GMT
Backups suck. They always have, they always will, it’s just the nature of the beast. The big problem is that open-source backup solutions tend to suck really bad, and decent commercial solutions obey Joel’s pricing law, where no enterprise software costs between $3,000 and $100,000. At Internap, we paid well over $100k to Legato for their buggy, inflexible backup software, and we paid a similar amount of money for backup hardware. That’s what it took to do good backups of several hundred machines per night. On a smaller scale, people seem to have decent luck with Arkeia, but I’m not all that convinced that it’s much cheaper then Legato or Veritas once you pile on dozens of clients.
I’ve wanted a decent open-source solution for years. In ‘97 or so, I used Amanda for a while, but it had a kind of bizarre approach to tapes, weird security needs, and it didn’t scale very well. It still seems to be under some sort of active development, but I was using version 2.4.0 in the late ’90s, and the current release is 2.4.4p3; draw your own conclusions. I’ve been spending around one afternoon per year since then looking for something free that can handle a couple dozen mixed Linux and Windows systems, and can back up to a mix of tape and disk, and I’ve come away empty-handed every time.
Until today. Somehow, Bacula slipped past me without my noticing it. Besides the catchy name and motto (“It comes by night and sucks the vital essence from your computers”), it seems to have the features that I’m looking for. It understands tape changers as well as filesystem backups. It’s network based with clients for Windows and Unix (and sort of OS X). It looks like it does an okay job at backup parallelism. It doesn’t have plugins for hot backups of popular databases, but most of the time, it’s easier to dump the database to a file and then back up the file, at least when you’re dealing with smallish databases.
It’s currently at version 1.34, although Debian seems stuck at 1.32f4 for some reason. I’ll be doing some testing tonight at home, since I’ve been without a comprehensive backup solution for years, but I’m feeling pretty good about this.
Having said that, it still lacks a few things that I’d consider rather useful:
- It can’t migrate backups between storage devices. No staging from disk to tape, for instance.
- It doesn’t do anything with optical disks. I’ve had good luck with doing full backups of important data to CDs or DVDs; it’d be nice to integrate that into one central backup system. This is better for smaller-scale setups, though.
- It can’t stripe a single backup across multiple drives.
- It’s not clear if it can interleave backups onto a single tape. When you’re backing up slow hosts over a big network, this can be critical. For LAN usage with cheap tape drives, it’s less critical.
They maintain a to-do list online. I haven’t really looked into their security history; there are a couple worrisome comments about sscanf in their to-do list, but it can’t be any worse then Legato Networker–it had holes that you could drive a bus through along with a few really fun failure modes. We triggered a great DDoS on ourselves one night in 2000 with every single backup client in the company sending 64-byte UDP packets to our backup server as fast as they could transmit them. We really loved them for that.
Update (8/6/2004): I let it rip last night. First, its UI needs work. Most configuration is done via text files, which is okay, but monitoring and management can be done via either a bad CLI without command-completion, or via a bad GUI which is basically just the CLI in a window with a scrollbar. Windows backups don’t include open files *or* the registry. Mac backups don’t include resource forks, but I knew that going into it. Finally, I’m not getting any email notification of success or failure, but that might be a mail problem on my backup server. All in all, it’s still better then most of the free solutions that I’ve looked at, but still not 100% there. It’s usable if you only care about Unix systems and you’re willing to spend a bit of time learning and scripting. It’s not usable if you’re looking for an out-of-the-box system that’ll work with Windows.
A friend suggested that I look at Box Backup. It doesn’t do tapes at all, which is a bit of a shortcoming in my mind, but it looks like it might be better for pure-Unix backups. On the other hand, it’s designed to do continuous backups, not nightly snapshots, which is probably a better design, if you can handle the load.
I should really write the backup strategy guide that I keep meaning to write. Leave a comment if you’re interested.
Posted in Computer System Administration | Tags backups, bacula, linux | 7 comments
Posted by Scott Laird
Tue, 24 Feb 2004 11:42:21 GMT
Somewhere in the middle of my MPx200 adventure, something ate all of my repeating annual calendar entries, including birthdays and anniversaries. Personally, I blame PocketMac, but it’s really my fault for not keeping a good backup of my calendar.
That’s not to say that I don’t have a backup–I sync iCal with my home web server every few minutes–but there’s no way to track changes or dig up old versions of my calendar. Once entries are deleted, they’re gone for good. What I really want is revision control for my calendar.
And now I have it. Through a feat of astounding geekiness, I’ve finished installing the subversion version control system on my home web server, and my calendar WebDAV share is actually a subversion repository. What that means is that every time iCal exports a new calendar to my home server, it’s transparently checked into subversion in the background. I can get subversion to give me a list of all of the changes that have been made over time, generate diffs between specific versions, and even revert to older versions.
It’s actually pretty easy. Just install Apache 2, and then install Subversion with the ‘mod_dav_svn’ Apache module. Configure up a virtual server and then add these lines to your virtual host config files:
<Location /xxx>
Dav svn
AuthType Basic
AuthName "Auth Required"
AuthUserFile /etc/apache/htpasswd
Require valid-user
SVNPath /path/to/svn/repository
SVNAutoversioning on
AuthzSVNAccessFile /etc/apache2/svn_authz
</Location>
Now create /path/to/svn/repository with svnadmin, reload apache, and you should have a nice, revision-controlled web-based file server. Most modern desktop OSes can mount WebDAV-enabled web servers just like a more-traditional file server. Both OS X and XP can do it natively, and there should be at least a dozen ways to accomplish it in Linux. This is almost certainly the easiest (cheap) way to get reliable version-controlled storage.
Thanks to Sterling Hughes and Justin Erenkrantz for providing inspiration and documentation.
Posted in Computer System Administration | Tags apache, backups, ical, subversion, webdav | 2 comments
