It all started on Wednesday night, when I discovered that my Asterisk VoIP server couldn’t make long-distance phone calls any more. I could dial, and the phone would ring, but as soon as the other end answered, the line went dead and the logs started filling up with error messages:
Feb 2 20:24:03 WARNING[-1250968656]: Huh? An ilbc frame that isn't a multiple of 50 bytes long from IAX2 (16)?
That’s kind of weird looking. Google wasn’t particularly helpful, so I decided that this was probably a VoIP provider problem and sent mail to NuFone’s support people. Historically, NuFone’s support department has a reputation of being a bit spotty–either they handle your problem right off the bat, or you never hear from them again, but that was probably a combination of broken email servers and staffing shortages due to rapid growth. In this case, I got a response back from NuFone’s president first thing in the morning. We’ve exchanged a bit of mail in the past, and in this case he agreed that it was a weird error, and the first thing to try was upgrading. I was running Asterisk 1.0.2, while the current version is 1.0.5. He suggested that I might be happier with checking a development version of out CVS–they’ve been more stable for him, anyway, and there are a ton of new features and bug fixes that aren’t in 1.0.5.
What the heck, I said. Sounds like fun.
So, I checked a copy of Asterisk and several of its supporting libraries and drivers out of CVS, built them, installed them, and tried restarting Asterisk. First problem: I’d built my kernel without support for module unloading, somehow. So, I had to reboot to upgrade the drivers for my two PCI VoIP cards. Grumble, grumble. Oh well, I’d been meaning to shut it down and add more RAM anyway; the RAM was sitting right there, and it shouldn’t take long to slide the server out from under its desk and install the RAM. Five minutes later, the box was booting back up with the new drivers and twice as much memory.
Problem 2: As soon as I started Asterisk, I discovered that the X101P PCI card that handles my regular POTS line was reporting a ‘red alarm’ on the line, suggesting that it wasn’t connected right. I crawled under the desk, unplugged the phone line, wiggled connectors, and tested it with a $10 phone that I keep on hand for situations like this. I got a dial tone just fine with the phone, so I plugged the line back into the PC and crawled back out from under the desk to discover…
Problem 3: the server locked up while I was fiddling with the phone line. So I rebooted it, only to discover…
Problem 4: it couldn’t load the drivers for the PCI card anymore. The driver spit out a number of weird errors:
kernel: wcfxo: Out of space to write register 06 with e0 kernel: wcfxo: Out of space to write register 0f with 00
That’s ugly. I developed a new theory: the card wasn’t getting reset right on reboot. So, I powered the box completely down, pulled the power cord for a minute, and then tried again. It worked this time; everything came up right, and I was able to verify that everything was working correctly. Incoming calls worked right, and outgoing calls worked, via both POTS and VoIP.
So fast-forward a few hours. I try to call out, and nothing happens–I just get a dead line. Checking Asterisk’s console, it looks like it dialed out on the POTS line, but I wasn’t hearing ringing or anything. The driver for the X101P PCI card had apparently choked again. I tried stopping and restarting Asterisk, but it didn’t help. I didn’t worry too much, though–incoming calls on my POTS line get forwarded to VoIP after about 15 seconds worth of ringing. Somewhere in the middle of this, a couple calls came in from family members, but I wasn’t close enough to a phone to answer them, so I let them go to voice mail. Except neither call left a message. That seemed strange (my family is big on leaving messages), so I called back, and they said that the line had went dead right after it answered–they never heard the “leave a message” message.
Problem 5: incoming VoIP calls weren’t working, either. The connection died in the middle of a
Playback instruction. I tried changing codecs, restarting Asterisk, and even rebooting, all without success. My frustration level was rising. Quickly. At this point, almost nothing was working right–POTS was dead, and incoming VoIP calls only worked right if I answered the phone. I took a break, put the kids to bed, and resisted the urge to scream. I mean, I debug problems like this for a living, which means that I really hate to do it at home.
So I spent a minute thinking–it was really weird that rebooting didn’t fix the VoIP Voicemail problem. I’d tested it earlier in the day, and it had worked after the upgrade to CVS Asterisk. And I hadn’t changed any other software since then. I rebooted again, and this time the X101P PCI VoIP card disappeared completely. Asterisk wouldn’t even start now–my config files are sprinkled with references to this card, and Asterisk was aborting with “can’t find card” errors. So I thought a bit more and came up with a new theory–when I’d pulled the box out to install more RAM, maybe I’d pulled on the phone cable that goes into the X101P a bit too hard, and the card had popped partway out of the slot. It’s easy to test, so I shut the server down again, slid it out from under the desk, popped the PCI card out, checked it for obvious problems, and put it back in firmly. I then plugged everything back in, rebooted, and discovered that everything was working perfectly. POTS worked, incoming voicemail worked via POTS or VoIP, outgoing VoIP calls worked.
I checked again this morning, and everything’s still working fine.
So, did I really pull the card halfway out, or was yesterday’s Asterisk CVS tree just really unstable? And what sort of person goes through this just to save $15 per month on phone calls? And speaking of that–if I’d had a second VoIP provider set up, then I could have shunted calls to them when outgoing calls to NuFone failed, and I would have been able to avoid part of the mess. Hey, yeah, and I still need to finish the Asterisk-caller-ID-on-MythTV project, so I can see who’s calling on the TV while I’m watching movies. Oohh, and my shiny new internal PCI ADSL card is supposed to ship soon. And I need to mod an Xbox so I can run MythTV on it downstairs…
I guess that should answer the “what sort of person” question. A person with incurable gadget lust. I’m getting better. Really.