A new server (part 1)

A few days ago, I mentioned that my home NAS box had failed, and that I was considering replacing it with a PC server running OpenSolaris and ZFS. I’ve read a pile of ZFS docs, and it looks like the best option available to me today, so I decided to order some suitable hardware.

At that point, pretty much everything broke down. I have a hard enough time keeping track of which hardware works with Linux this week, and OpenSolaris is completely new to me. Sun’s list of officially-supported hardware is pretty sparse, and digging through their mailing list archives gets frustrating quickly. From what I can tell, it boils down to:

  • Current Intel and AMD CPUs are all fine.
  • Most of Intel’s chipsets are fine.
  • Most of nVidia’s AMD chipsets are fine.
  • nVidia and Intel video chips are good.
  • Most common Ethernet chipsets are either supported natively or have drivers available.
  • The only SATA controllers that work are Intel’s ICH southbridges, Silicon Image’s PCI and PCI-E chips, Marvell’s PCI chips, and nVidia’s southbridges. It’s not clear that Marvell’s PCI-E chips work. Most motherboards with additional, non-southbridge SATA ports probably won’t work.
  • Venturing too far outside of this list will probably result in problems.

I was looking for a motherboard with 8 SATA ports, and was hoping that the Intel D975XBX2 (“Bad Axe 2”) would work, but 4 of its 8 SATA ports belong to a Marvell PCI-E SATA chip that doesn’t appear to be supported. I went through every single 8-port motherboard in Newegg’s database and ended up flunking all of them out for some reason–unsupported chipset, unsupported SATA chip, not enough slots, too expensive, and so on. In the end, I ended up ordering an Asus P5K WS (the ‘WS’ is important–the P5K is a different board). It only has 6 on-board SATA ports, but it includes a PCI-X slot. That’ll let me use the Supermicro AOC-SAT2-MV8, which is far and away the cheapest 8-port SATA card on the market. That’ll give me a total of 14 SATA ports, which should be enough for a whatever I want to throw at it. The Marvell PCI-X chip at the heart of the Supermicro card is the same one used in Sun’s Sun Fire x4500 48-drive server, so it’s safe to assume that Sun has put a lot of effort into the driver.

Most of the test of the system is fairly generic–a cheap nVidia 7200GS video card (the cheapest PCI-E card that NewEgg carries), a nice case and power supply, RAM, and a boatload of drives.

The one odd component that I’ve added is a Gigabyte GC-RAMDISK with 1 GB of RAM. The GC-RAMDISK is a battery-backed SATA ramdisk; it looks like a hard drive to the system and can survive up to 18 hours without power. I’ve had my eye on this thing for years, and it looks like it’ll be a perfect external log device for GFS. I had to ask to see how ZFS will behave if the device fails, and it looks like manual intervention may be required after an 18+ hour power outage, but it should be pretty minimal. I’m planning on posting some benchmarks here once I’ve had a chance to try it out.

Assuming that I’m able to get this whole mess to work at all, I should have lots to write about here over the next week or so. I’m going to start by explaining why I want to use Solaris instead of Linux or *BSD, and why I’m building something instead of buying a pre-build NAS box.

Posted by Scott Laird Sat, 20 Oct 2007 04:56:00 GMT


Lightweight Home Security with Indigo and Asterisk

I mentioned last week that I’ve been working on building sort of a lightweight home security system for my house so my 4-year-old daughter won’t be able to sneak out of the house again.

I spent about a day researching possible solutions, to see if there was something simple that I could buy that would make me happy, and I couldn’t find anything on the market that was cheap and would tell me which of my 6 exterior doors had been opened loudly enough to hear from across the house. There’s no point in putting a door buzzer on the basement door if you can’t hear it from the master bedroom. The only solutions that I found were from professional alarm companies, and they probably would have charged me a couple thousand for installation plus $30-$50 per month for monitoring. I’m just not willing to pay that much, and it’s not really what I was looking for–it’s gross overkill for my problem.

So, I decided to build it myself. After a few hours’ searching, I decided to use Perceptive Automation’s Indigo home-automation software for the Mac. It’s commercial software, but I like its user interface and capabilities better then any of the open-source solutions that I’ve seen. It’s under active development, supports just about everything that I need, it’s client-server so I can access it from any Mac I own, and it comes with a nice web interface that I can get to via my phone. I figured that it’d be cheaper to pay the money for Indigo then to spend most of a week hacking away at one of the open-source packages to get them to do what I want. Plus, er, pretty much every open source home automation program that I could find was written in Perl, and I’ve been successfully avoiding Perl for almost 5 years now. The last thing I really want to do is spend a week modifying someone else’s Perl. That kind of thing gives me nightmares.

So I fired Indigo up on an old Powerbook that I had laying around and bought a bunch of hardware from MacHomeStore and SmartHome.com. The important bits are a W800RF32A wireless reciever and a whole bunch of $12 DS10a door and window sensors. The sensors broadcast their state over the air, and the receiver feeds them into my Mac.

That took care of the input side of the equation, but I still needed some relatively cheap way to play audio around my house. I poked around for a while looking at random X10 hardware and radio solutions before I realized that most VoIP phones support audio paging. Since my house is full of VoIP phones and I run my own Asterisk server, all I’d really need to do was write a couple dozen lines of Asterisk dialplan code, and everything should just work. I ended up ordering two new VoIP phones (Grandstream GXP-2020s) because two of the phones that I have are too old to be usable for paging. I probably could have used $45 Budgetones, but I have future plans for the GXP-2020’s big displays, so I decided to spend a bit more for them.

It took me about 30 minutes to unpack everything, install Indigo, and have it receiving data from a test sensor. Each sensor assigns itself a random 8-bit ID when it’s powered up, so the first problem was mapping semi-random sensor IDs onto logical names. Indigo comes with a blob of sample AppleScript for doing this, and it only took me 5 minutes to modify it so that one of Indigo’s internal variables changed state to reflect the state of the door sensor–true when the door is closed and false when opened. Five more minutes and I had a nifty web page with a green blob that turned red when the door opened. A half-hour after that, I had a PNG floorplan of my house that I could use as a backdrop for a bunch of little red/green blobs:

After this, it’s all just a matter of plumbing. First, I modified Indigo’s sensor-handling AppleScript example into something that knows how to talk to Asterisk, using one of the Asterisk/AppleScript integration examples on the voip-info.org Wiki. Here’s I ended up with:

(*
  Door sensor tracking code for Indigo and Asterisk.
  By Scott Laird <scott@sigkill.org>
  http://scottstuff.net

  This code handles incoming security events, maps each numeric
  device ID onto a logical name, and then tells Asterisk to page all
  VoIP phones with a device-specific message.

  Net effect: opening the front door causes "Front door opened" to echo
  throughout the house.

  I'm calling doDialOut directly rather then using an Indigo trigger for 4
  reasons:
    1.  I have the extension name handy here, while I'd have to parse it back out of the variable name if I used triggers.
    2.  Creating an identical trigger for each of 10+ sensors is a pain in the neck.
    3.  We need to do *something* with unknown sensor events, but creating variables for them is pretty clearly wrong.
    4.  My AppleScript is lousy, and I can't get triggers to call doDialOut correctly.
*)

using terms from application "IndigoServer"
  on receive security event of eventType with code devID
    set extension to "unknown"

    -- Map sensor IDs onto logical names.  In a real language,
    -- I'd use some sort of hash and skip the if ... else if ... code,
    -- but I don't see anything suitable in AppleScript.  Sigh.  I'm
    -- stuck writing code in blub.
    if devID is 95 then
      set extension to "frontdoor"
    else if devID is 175 then
      set extension to "deckdoor"
    else if devID is 99 then
      set extension to "stairs"
    else if devID is 201 then
      set extension to "backyard"
    else if devID is 55 then
      set extension to "upgarage"
    else if devID is 155 then
      set extension to "downgarage"
    else if devID is 253 then
      set extension to "slider"
    end if

    -- If we get a request for an unknown devID, then send it on to Asterisk 
    -- so we get an audible indication that *something* happened.  Since batteries
    -- falling out of DS10 modules can cause the ID to change, I'd rather not ignore
    -- these.  YMMV, however.
    if extension is "unknown" then
      my doDialOut(devID)
    else
      -- Indigo variable names are derived from Asterisk extension names.

      -- In retrospect, doorClosed_ is a great name for doors, but not so hot
      -- for doorbells or motion sensors.  Feel free to change this.
      set var to ("doorClosed_" & extension)
      set seen_var to ("lastSeen_" & extension)
      set change_var to ("lastChanged_" & extension)
      set timestamp to (current date) as string

      -- Figure out if it opened or closed.  There are actually 4 different
      -- results that the sensors can return (normal/active X min/max),
      -- but we only care about normal/active.
      if eventType is sec_SensorNormal_min then
        set val to "true"
      else if eventType is sec_SensorNormal_max then
        set val to "true"
      else
        set val to "false"
      end if

      -- Create the Indigo variables if they don't already exist.
      if not (variable var exists) then
        make new variable with properties {name:var, value:val}
      end if

      if not (variable seen_var exists) then
        make new variable with properties {name:seen_var, value:timestamp}
      end if

      if not (variable change_var exists) then
        make new variable with properties {name:change_var, value:timestamp}
      end if

      -- Did the value of this variable just change?
      if not (value of variable var is val) then
        set value of variable change_var to timestamp

        -- If it just changed and it's now false, then send an alert.
        -- The DS10a sensors send a signal once per hour, even if nothing's changed.
        -- So we don't want to alert *unless* something's changed.
        if val is "false" then
          my doDialOut(extension)
        end if
      end if

      -- We need to keep track of the time that each DS10 was last seen.
      -- This way we can spot bad batteries.
      set value of variable seen_var to timestamp
      set value of variable var to val
    end if
  end receive security event

  -- doDialOut tells Asterisk to dial a specific extension using
  -- Asterisk's management interface.  This doesn't require that Asterisk
  -- runs on the same machine as Indigo.
  --
  -- To get this to work in your environment, you'll need to change the IP
  -- address, username, and password, and possibly the contexts and/or 
  -- extension names used at the bottom.
  on doDialOut(extension)
    log "Paging extension " & extension using type "Dialer"
    set expectscript to "set timeout 20;
spawn telnet 10.0.0.1 5038;
expect \"Asterisk Call Manager/1.0\";
send \"Action: login
username: my_username
secret: secretsecret

\";
expect \"Message: Authentication accepted\";

send \"Action: originate
Exten: " & extension & "
Context: security-paging
Channel: Local/all@paging
Priority: 1
Callerid: Security: " & extension & " <0>

\";

sleep 1;

send \"Action: logoff

\";
"
    set results to do shell script "/usr/bin/expect -c  '" & expectscript & "'"
  end doDialOut
end using terms from

In addition to simply monitoring the current state of the sensors, this also tracks their last change as well as the last time each sensor sent out a “nothing’s changed” report. Eventually I’ll add monitoring for this so I can spot failing batteries immediately, and not two months later when I discover that two doors aren’t monitored anymore. Indigo tracks each variable internally, and gives you a handy window for viewing and modifying them:

Once all of that was in place, it was time to add paging logic to Asterisk. I created two new contexts, security-paging to contain the messages that need to be played back and paging to handle the phones. The AppleScript above basically glues the two ends together, sending ‘deckdoor@security-paging’ to ‘all@paging’. Here’s the relevant bit of my Asterisk config:

[security-paging]
  exten => frontdoor,1,Playback(security/frontdoor)
  exten => deckdoor,1,Playback(security/deckdoor)
  exten => backyard,1,Playback(security/backyard)
  exten => stairs,1,Playback(security/stairs)
  exten => upgarage,1,Playback(security/upgarage)
  exten => downgarage,1,Playback(security/downgarage)
  exten => slider,1,Playback(security/slider)

  exten => _.,1,SayNumber(${EXTEN})

[paging]
  exten => all,1,Page(Local/203@paging&Local/204@paging&Local/205@paging&Local/206@paging)

  ; Cisco 7940; no Call-Info support, but you can create a new line and turn on auto-answer on the phone.
  exten => 203,1,Dial(SIP/203aa)

  ; Sipura 841.  It needs a semicolon before answer-after.
  exten => 204,1,SipAddHeader(Call-Info: \;answer-after=0)
  exten => 204,n,Dial(SIP/204)

  ; Grandstream GXP-2020s, although the same config will work for most modern phones.
  exten => 205,1,SipAddHeader(Call-Info: answer-after=0)
  exten => 205,n,Dial(SIP/205)

  exten => 206,1,SipAddHeader(Call-Info: answer-after=0)
  exten => 206,n,Dial(SIP/206)

Then I just had to record some audio files to use for annoucements. The easiest way to do this is just to call yourself up and leave voicemail, and then copy the VM files over into Asterisk’s sound file directory, usually /var/lib/asterisk/sounds. You could get better quality recordings with a good microphone and audio-processing app, but I’m not sure that there’s really a point, given the quality of most speakerphone speakers.

All told, it took me about 8 hours of research to put this all together, and maybe 10 hours to implement it all, including learning a bit of AppleScript. Now I have a programmable system for monitoring my house, and all of the pieces in place for adding X10 or Insteon components as they make sense. It’s all under my control; if I can code it, then I can make it happen.

So, does it all work? Yep–Monday morning it caught my daughter trying to sneak into the garage. Mission accomplished.

Posted by Scott Laird Wed, 15 Aug 2007 16:16:00 GMT


Backyard Flowers

I spent most of today trying to catch up on the three weeks’ worth of yardwork that I’d missed while I was in California. I didn’t make all that much progress, but along the way I spotted a few photogenic wildflowers growing in the middle of my back lawn. I still don’t own a real macro lens, but I bought a Canon 500D close-up lens last month before my trip but never had a chance to use it. So took a break from the yard work, slapped it on the 100-400, grabbed a couple flashes, and headed out into the yard. Here’s my favorite:

Flower

All in all, I’m pretty happy with the pictures. The 100-400L plus 500D gives roughly 1:1 magnification at the long end, with way more working distance then I’d expected. It may actually be too long for this sort of photography–I kept having to back up to give myself more room between the lens and the flower, especially at the long end of the lens. Fortunately, my 7-year-old assistant was able to keep the light on the flowers, even as I kept backing further and further away. Good help is hard to find :-).

Posted by Scott Laird Sun, 22 Apr 2007 06:27:37 GMT


A whole new office

When my wife and I moved into our house almost 7 years ago, I claimed one of the bedrooms as a home office. I crammed a desk, an 8’ folding table, and 2 big bookshelves into the room, filling it up with Linux systems, big CRT monitors, 2 or 3 printers, a couple scanners, and other big, heavy bits of hardware. I wanted to maximize my desk space, so I put the desk and table up against the walls and worked with my back to the door most of the time, never quite understanding why I didn’t enjoy working in there as much as I’d expected.

In 2002, I bought a PowerBook and discovered that I could work from any room in the house; I wasn’t tied down to big chunks of glass and 30 lb computers anymore. So I mostly stopped using my office for anything but storage. There are a couple servers in the corner, but most of the space is full of hardware that I don’t really need.

I finally realized a month or two ago that I actually want a quiet place to work, and that most of the problem with my home office is that the layout is designed to maximize the number of big CRTs that it can hold. That’s irrelevant now, and has been for years. What I really want is a quiet, comfortable room with one or two LCDs, a place to read, and some place to plug in my laptop.

So, I’ve started on a quest to fix my office. My long-term goal is to repaint, replace the nasty old carpet with something more livable, and maybe even pick up a couch to fit along one wall, but for now I’m starting by moving the desk around so my back is away from the door and getting rid of the big folding table. Then I’ll add a bit of extra lighting so it doesn’t feel like a cave anymore, and replace the two big bookshelves with someplace to cleanly store cables, random camera gear, CDs, and a couple dozen books.

I moved the desk around last night, then moved my 24” LCD onto it. Then I dug the speakers that I bought in 2001 and have never really been able to use out of the mess, plugged them into a nice little surround sound decoder, and wired my Xbox 360 and Wii into the LCD and decoder.

I’m only half-done with the “clean the room up” phase of the plan, but it’s already drastically better. Getting the desk away from the door made a huge difference. I don’t know why I didn’t do it years ago.

Posted by Scott Laird Fri, 05 Jan 2007 07:03:15 GMT


The sysadmin's home office

It’s been an interesting experience setting up my home office over the past few days. I remember my parents going through a similar experience when I was in high school–they bought a copier, a fax machine, a new business phone, waited for the telco to install a new line, dealt with the yellow pages people, and spent a couple days in the car dealing with all of the paperwork and supplies required.

So fast-forward to the modern sysadmin-turned-consultant’s home office:

  • Computer: have several. Can produce more from spare parts if needed.
  • Copier, fax: have one of those already.
  • Office phone line: 5 minutes via the website of my VoIP provider.
  • Office phone: dug an old phone out of a box and plugged it into my Asterisk server. It’s nice having a home PBX.
  • Advertising: this blog, networking via friends, releasing newer and more exciting open-source software.
  • Business email and web server: I already have those. I can create more accounts and sites as needed.
  • Business licenses: they’re all available online now, although Washington State’s site requires either IE or Netscape 4, and putting Safari into “lie about your User-Agent string” isn’t good enough.

It’s amazing how easy it is to set up a small business office when your house is already higher-tech then most companies.

Posted by Scott Laird Tue, 06 Sep 2005 22:48:00 GMT


Tree Removal for Fun and Profit

I’m still recovering from this last weekend–we ripped a 60 foot maple tree out of our back yard. One of the tree’s three main trunks was rotting, and it’d been dropping branches onto our yard and fence for years. It had to go. The problem was that the tree was less then 10 feet from our fence, only 20 feet from our garage, our neighbor’s garage, and the road, and 25 feet from our house. The hard part was getting the tree down without destroying anything.

We mostly succeeded–our garage’s gutter has two little dings, and we cracked one panel of our fence, but we managed to reduce the tree down to a 4-5 foot wide stump. We hauled 5 pickup-loads of branches to the city’s yard cleanup station, and hauled 2 dumptruck-loads of logs and firewood to friends and family. We’re left with a small mound of branches and leaves, but we’ll be able to stuff that into our yard waste can over the next few weeks.

Next up, I need to rent a stump grinder and get rid of the stump. That’ll probably take a few hours–it’s a lot of stump. Once that’s done, we’ll be ready to level things out, add a bit more topsoil, and then re-plant our back yard. It’s been ailing for years–the combination of the tree’s shade, damage from my sister-in-law’s dogs, and neglect have reduced it to a sea of moss and dandelions.

I keep forgetting how much I enjoy large-scale yard work. It’s a nice counterpoint to the week’s computer work, plus it gives me an excuse to play around with heavy equipment every now and then. How can you argue with that?

Posted by Scott Laird Wed, 25 May 2005 18:16:07 GMT


Traffic

It’s sort of an axiom of programming that features that aren’t continually used or tested won’t actually work. A similar rule holds for system administration–any feature that hasn’t been tested since the last upgrade is probably broken. An obvious corollary suggests that systems get more reliable as their user load increases–more users means more features are used more frequently, and broken features will be spotted sooner. And the corollary to that is that any server wedged under a desk in someone’s home office is probably flakier then hell because it’s probably just sitting there collecting dust and not getting used.

I’m not convinced that that applies to my home gateway box. It’s a busy little beaver:

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes  prot opt in out source           destination     
 234M   75G  all  --  dsl0   *   0.0.0.0/0        0.0.0.0/0       
  47M 1001G  all  --  eth0   *   0.0.0.0/0        0.0.0.0/0       

In the 25.75 days since I last rebooted this system, it’s received over 75 GB via its DSL link and around 1 TB over its main Ethernet link. If my math is right, that’s an average of 3.6 Mbps on the Ethernet link and around 270 kbps over DSL. I wasn’t keeping outgoing traffic stats when I first booted this box, but more recent estimates make it look like there’s almost as much outgoing traffic on dsl0 as there is incoming.

CPU load is similarly heavy–the box has averaged 51.9% idle since it was rebooted. My rule of thumb for years was that any production box that was under 80% idle was due to be upgraded soon, because it was probably pegging the CPU during peak times during the day. If the box was under 70% idle, then it was time to start scrounging for an immediate upgrade. By those metrics, this box is way overdue for a major upgrade. Fortunately for my wallet, those metrics don’t really apply to this box–it’s spending a lot of its CPU time on tasks that aren’t particularly critical. Also, Linux 2.6 made some changes to /proc/stat that procinfo doesn’t seem to have picked up on; once you factor those into the equation, the box is really closer to 75% idle. Subtract off the non-critical usage, and the system is probably only 10% busy. I’ll probably upgrade it later this year if my virtual-server project works out, but that’s more for security and reliability then pure performance.

Posted by Scott Laird Tue, 08 Mar 2005 07:23:21 GMT


Success: My new DSL line is up and running

I can’t believe it. After months and months of trying to upgrade my DSL, everything is finally up and running on my new DSL line. The line wasn’t supposed to go live until Friday evening, but Verizon sent me mail today with my IP address and claimed that it’s up and running. And, indeed, it is.

Unfortunately, I wasn’t quite as prepared as I thought I was; I’d forgotten to change the DNS TTL on scottstuff.net, so people may have a hard time getting through to this site for a couple days. Oops. Other then that, though, everything seems to be up and running perfectly. The sample size is still kind of small, but smokeping suggests that my average ping time has improved drastically–I was seeing numbers from 25 to 800 ms before. Right now, the line looks flat at 30 ms. That’ll probably break a bit once traffic picks back up on the website, but I should be able to keep it under 100 ms, easy.

I’ll post more details later tonight or tomorrow, along with a review of the Sangoma S518 PCI ADSL card. I have to put the kids to bed first, though.

Posted by Scott Laird Thu, 10 Feb 2005 05:08:58 GMT


DSL: 6 days and counting

I have a date now for my new DSL line: February 11th. My new DSL PCI card is in UPS’s hands, too. I’m starting to believe that my months-long DSL upgrade quest is nearly complete.

Posted by Scott Laird Sun, 06 Feb 2005 00:43:01 GMT


One step closer to faster DSL

My DSL upgrade saga took another step towards completion today, when Verizon installed a new phone line at home. Amazingly enough, it’s already in their DSL database, and Verizon’s DSL sales website allowed me to order DSL. I don’t have an install date for the DSL service yet, but I’m two steps closer to the finish line.

I was surprised that Verizon’s business DSL department gives you a choice of buying a new DSL modem ($99) or providing your own. I took the opportunity to order a Sangoma S518 PCI DSL card instead of an external box. Supposedly, their Linux drivers are solid (Sangoma has been involved with Linux practically forever), and there are a couple big advantages to native DSL interfaces, rather then DSL-to-Ethernet bridges. The biggest advantage is buffering–right now, my DSL modem has at least a couple seconds worth of buffers on it. If I send outbound traffic as fast as I can, I rapidly get to the point where nothing makes it out in under 2 seconds. So, even if I use extreme care in setting up QoS prioritization rules for VoIP traffic, the VoIP packets will still end up stuck in the DSL modem’s buffers. To combat this, I’ve been forced to rate-limit my outbound traffic to about 75% of the theoretical limit; even then it can really suck at times. Several people on Asterisk mailing lists have commented that their S518 has really made their VoIP performance shine.

In addition, since the S518 is directly talking to the phone company, it can tell exactly what speed I’m currently provisioned at and can log problems via syslog. My current nasty DSL box can’t do anything but blink lights at me when there’s a problem.

All in all, it looks like a decent improvement, especially since it’s only $115 or so online.

Posted by Scott Laird Thu, 03 Feb 2005 00:36:10 GMT