I now have Solaris up and running and reasonably stable-looking, after only 12 hours of work. A number of things turned out to be bigger issues than I’d anticipated, largely because it’s been years since I last used Solaris and, frankly, Solaris’s disk partitioning and formatting tools suck.

  • My first problem is still unresolved: my BIOS refuses to boot from the IDE DVD drive that I installed. Once the system boots, it works just fine, so I’m not sure what’s up. Maybe a BIOS bug. Fortunately, the system’s perfectly happy booting off a USB DVD drive, and (amazingly) Solaris is happy installing from it.
  • The GC-RAMDISK card that I was looking forward to testing is a complete failure so far. I don’t know if I have a bad card or if it’s simply incompatible with both SATA chips in the system, but the BIOS completely ignores it if its plugged into the motherboard, and Solaris fails to talk to it on either bus. If it’s plugged into the MB, then I get a device failed initialization error; if it’s plugged into the PCI-X SATA card then I get device on port 5 still busy after reset. I’ve swapped cables and RAM. I’d really like to get it to work, so I’m going to try it with an older system before RMAing it.
  • Actually getting a working Solaris install took me 3 tries. The first time I installed it to the wrong drive (the first disk on the PCI-X card, not the first disk on the motherboard), and it was unable to mount the root partition after rebooting. Next, I managed to install it onto an EFI partition, and that wouldn’t boot either. Finally, I installed it onto the second drive on the right bus, and that worked.
  • Since Solaris’s installer doesn’t support ZFS yet, I had to manually copy the root filesystem onto a newly created ZFS filesystem mirrored across a pair of drives. The directions are helpful, but I kept screwing things up. First, I accidentally created the ZFS pool using the entire disk, which made ZFS re-label the drives with EFI, which makes them unbootable. Then, I missed the line in the directions that says to run format -e instead of using format; that left me with a pair of nicely partitioned drives that still used EFI. The third try worked, and the system is now booting off of ZFS via GRUB without problems. Er…
  • Well, one problem–I can’t change the GRUB menu.lst file for some reason. I don’t know where GRUB is looking for it, but it’s not in /boot/grub/menu.lst on my boot array. My changes are being completely ignored. I can live with this for the weekend.
  • OpenSolaris doesn’t ship with drivers for the ASUS P5K WS’s onboard Ethernet chips. I had to grab them from Marvell, but it was easy enough to install.
  • Creating an 8-drive ZFS filesystem is trivial. One command takes care of RAID, logical volume management, creates the filesystem, and mounts it: zpool create -f space raidz2 c0t1d0 c0t2d0 c0t3d0 c0t4d0 c3t2d0 c3t3d0 c3t4d0 c3t5d0.
  • ZFS performance is decent. Here’s a bonnie with and without ZFS compression, using 10 GB of data on a box with 2 GB of RAM:

              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
    Machine    GB M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU  /sec %CPU
    zfs        10 105.0 55.3 163.3 27.3 121.0 30.4 119.2 88.4 287.1 36.2   169  1.8
    zfs+c      10 112.9 59.7 181.5 30.3 127.8 29.1 118.1 86.0 424.9 52.2   198  2.1
    
    • 163 MB/sec writing and 287 MB/sec reading is good enough for me. I was expecting slightly higher numbers, but there’s nothing here to complain about. Adding compression improves writing a bit and makes a big difference reading. It’s quite a bit faster then GigE, which was my goal.