So, the comments on yesterday’s post about my nasty RAID failure encouraged me to spend some time looking at ZFS on OpenSolaris, and I really like what I see. I’ve ordered some new hardware, so I should have lots to write about by next weekend.

Reading the ZFS docs reminded me of my Holy Grail of Storage: a storage system that could actually do reasonably smart things with 3–5 drives. Imagine a system where you could start with 3 drives and simply plug new drives in as you need more space, without worrying about RAID or data layout. When you run out of slots, then just unplug the oldest, smallest drive and plug in a new, larger one, and the data will resync, giving you more disk space without needing any special work on your part. For bonus points, you’d be able to designate specific bits of your data as more or less important, so Bittorrent files might not be replicated at all, while your Word documents might be replicated onto every available drive.

I’ve wanted that for years, but I’ve largely dismissed it as a pipe dream, because it doesn’t fit cleanly into the drive/RAID/LVM/filesystem model that everything uses. The only thing that I’ve seen that even comes close is Drobo, and it’s supposedly fairly slow and really just too “magic” for me to trust.

I realized this morning that it’d be easy to build a storage system like this using ZFS. Just create a zpool with 3 drives to start, and then create zfs filesystems with copies=2 on top of it. When you add new drives, just add them to the pool. Blindly removing a single old drive will only leave you with a single copy of some of your files, but that shouldn’t be fatal, and ZFS can copy everything off of it if you give it a chance. There are some corner cases that will give you less redundancy–if you manage to fill the system 98% full before adding a new drive, then all of the replicas of new data will probably end up on the same disk. There are a couple obvious workarounds, and Sun will probably add replication rebalancing at some point, if it isn’t there already.