Posted by Scott Laird
Wed, 27 Jul 2005 04:07:57 GMT
I was complaining to someone yesterday about my DSL bill, since I’m paying $80/month for 1.5/384 service. Out of curiosity, I took a look on Verizon’s business DSL website and noticed that they don’t sell 1.5/384 in my area anymore; the default is now 3.0/768 for the same price as before. So I called them up this morning and they’re doubling my line speed for free. It’ll take a day or two before it takes effect, but in a day or two I’ll either be:
- Happy with my new, fast service.
- Yelling at Verizon for completely screwing up my upgrade and knocking me off the net.
Anyone want to take bets on which one it’ll be?
Update: Well, I got my answer. I found this in the logs this morning:
Jul 27 00:22:35 guam kernel: wanpipe1: ADSL starting training 0x2 ...
Jul 27 00:22:35 guam kernel: wanpipe1: GP_LINK_DOWN, Training State
Jul 27 00:22:51 guam kernel: wanpipe1: Cell Delination successful
Jul 27 00:22:51 guam kernel: wanpipe1: GP_LINK_UP, State Trained
Jul 27 00:22:51 guam kernel: wanpipe1: ADSL Link connected (Down 3360 kbps, Up 704 kbps)
Jul 27 00:22:59 guam kernel: wanpipe1: Link connected!
There was also a “congratulations on your upgrade” email in my inbox. So it looks like the speed upgrade went through without a hitch. I have to say, this is the first really nice experience I’ve had with Verizon DSL in years. I’m feeling a lot better about them then I used to–six months ago I was getting 768/128, now I’m getting 3.0/768 for the same price. I’m not feeling ripped-off any more.
A quick download test verifies that the speed really is faster–I’m getting 298 KB/sec from ftp-mirror.internap.com. Smokeping shows that my link’s lower-bound latency dropped overnight, and this morning’s latency graph is much less noisy then it was yesterday. Both of those are good for VoIP.
Update: I did a quick comparison with my old speeds. I was getting 1792/442, now I’m getting 3360/704. That’s an 88% increase in downloads and a 59% increase in uploads. Not quite the 2x boost that you’d expect, but still quite nice.
Posted in Computer Networking | Tags business, dsl, speed, upgrade, verizon | 11 comments
Posted by Scott Laird
Fri, 29 Apr 2005 00:15:59 GMT
I’ve spent a fair bit of effort getting QoS on my home DSL link working right, so VoIP isn’t overwhelmed by downloads or by people hitting my web server. At this point, I’m down to one remaining problem–when Google and friends fire up their web crawlers and find a new directory full of JPEGs, they can slow other HTTP traffic to a crawl.
If I could tell Apache (2.0) to change the IP ToS flags associated with HTTP web crawler traffic, then my network’s QoS config would do the right thing and send user-driven HTTP traffic ahead of web crawler traffic. Unfortunately, I don’t see any obvious way to do this. I’d rather filter based on HTTP User-Agent, not network block, and that means either using a really smart packet filter or having my web server do the work on its own. And, as far as I can see, Apache 2 doesn’t have a ToS-setting module available. Dean Gaudet wrote mod_iptos for Apache 1.3, but it hasn’t been ported to Apache 2, and I’m not very eager to do it myself.
Does anyone have any suggestions?
Posted in Web stuff, Computer Networking | Tags apache, apache2, qos, tos | 1 comment
Posted by Scott Laird
Wed, 27 Apr 2005 18:56:11 GMT
This isn’t exactly new news, but Cisco bought Sipura yesterday. Sipura makes a number of VoIP products, including the SPA-841 phone that I’ve been using for the past few weeks. They’re generally considered to have the best SIP implementation of any of the cheap vendors, and they make good, solid products for low prices. It’s a nice combination. Cisco has been licensing Sipura’s technology and using it in Linksys’s cheap VoIP hardware for around nine months now. Linksys has had to jump through a number of hoops to keep Sipura happy recently; apparently Sipura didn’t like customers buying the unlocked Linksys PAP2-NA instead of the more expensive Sipura SPA-2000. Now that Cisco owns both companies, I suspect that they’ll work out their differences.
Hopefully Cisco won’t gut Sipura to keep them from competing with Cisco’s more expensive products. The jury is still out on Cisco’s Linksys acquisition–they haven’t released many exciting new products since Cisco bought them, but they haven’t killed off any of their interesting product lines or tried to stop the flood of alternate Linux firmware distributions for the WRT54G family either.
One thing that’s interesting about this acquisition is that Sipura was formed by a bunch of ex-Cisco people. After Cisco bought Komodo in 2000, a bunch of the Komodo people left Cisco to go form Sipura. Now they’re back at Cisco again. This seems to be how Cisco does R&D these days–it spins employees off to work on their own products and then acquires them if they accomplish anything interesting. I’m not convinced that it’s a bad way to deal with R&D risk in a huge company–it shields Cisco from the cost of failure and promotes risk-taking by R&D engineers, but it doesn’t do anything to help unify Cisco’s massively fractured product lineup.
Posted in Phones, Asterisk, Computer Networking | Tags cisco, sipura, voip | no comments
Posted by Scott Laird
Mon, 18 Apr 2005 18:25:32 GMT
It looks like Cisco is finally starting to push their new, modular IOS code down from their uber-expensive CRS-1 router into their merely amazingly expensive routers. Network World is reporting that they’re almost ready to release a version of IOS XR for Cisco 12000-series routers. So now you’ll be able to run a semi-modern operating system that implements things like memory protection between processes on routers that cost under $500k. When they get under $100k, this might start to be interesting.
Cisco’s also going to release a line of “shared port adapters” that can be used in routers from the 7300, 7600, 12000, and CRS-1 product families. You need a SPA Interface Processor card for your router type, and then you plus SPAs into the SIP card. According to their picture, the SIP for the 7600 family can hold 4 SPAs, which means that the SPAs themselves must be fairly small–almost certainly smaller then the PAs that 7200/7400/7500 routers use. There are a pile of different SPAs on their list, from 8x cT1 to 10x GigE to 1x 10GigE, to OC 192.
Of course, in typical Cisco fashion, the “shared port adapters” aren’t really all that shared. There are 3 different SPA carrier cards for the 7600 series; one model is only good with the VPN SPA, and the other two overlap a bit–both are good with 2x or 4x OC-3 SPAs, but one supports 1x OC-12 while the other supports T1 and T3 SPAs. None of the three models support the GigE SPAs. Ther 12000-series has 2 different SIP models; one is good for T1/T3 use, while the other one is good for GigE and OC-192s. The CRS-1 SIP is even more fun–it supports POS OC-3s, POS OC-192, and GigE. No OC-12 or OC-48 support, apparently.
So, even though the cards are called “shared port adapters,” there are some real limits on which chassis will work with which cards. The DS3 SPA will apparently only work on the 7600 and 12000, and not on the CRS-1 or 7304. I suspect that a lot of this is mostly a driver support problem, but it shows how screwed up and fractured Cisco’s product lineup is.
Posted in Computer Networking | Tags bfr, cisco, crs1, ios, networking, router | no comments
Posted by Scott Laird
Wed, 02 Mar 2005 21:16:21 GMT
As mentioned earlier, Intel has been making noises about improving network I/O on PC servers. Today, at IDF, they released a few details on their plans. Apparently the presentation itself was good, but their web documentation is slim on details. Lennert Buytenhek summarized the important details, centering on the threading improvements:
[…] Rather than providing multiple hardware contexts in a
processor like Hyper-Threading (HT) Technology from Intel, a
single hardware context contains the network stack with
multiple software-controlled threads. When a packet
thread triggers a memory event a scheduler within the network
stack selects an alternate packet thread and loads the CPU
execution pipeline. Porcessing continues in the shadow of a
memory access. […] Stall conditions, triggered by requests
to slow memory devices, are nearly eliminated.
This isn’t exactly like the IXP2800, but there are some distinct similarities. In essence, it looks like Intel wants to provide the OS with the ability to task-switch on cache misses. I’m not sure that current OSes can switch threads much faster then the CPU can handle a cache miss, so this will be interesting to follow. I suspect that you could switch fast enough if you don’t touch the TLB or most of the CPU mode bits.
Intel also points out that with 10 GbE, just mitigating the effect of cache misses by processing multiple packets in parallel isn’t enough–packets actually arrive faster then the computer can fetch data from main memory–with 64 byte packets at 10 Gbps, a new packet arrives every 51.2 ns, which isn’t even long enough for a single main-memory access. According to Intel, normal packet processing requires 5 main memory reads. Intel’s fix for this is to add the ability to DMA directly into the CPU’s cache, and then add support for offloading memory copies onto the memory controller itself.
While Intel is aiming at improving network performance, I suspect that other types of processing may see big improvements from the planned changes. Video compression, for instance, can have horrible cache performance; I saw a study a while back that showed P4s running a MPEG-2 codec were averaging one instruction every 5 cycles during part of the processing, or way under 10% of what the CPU is capable of. A video codec that could compress several macroblocks at once, switching between them on cache misses, could easily see big speed boosts.
Posted in Computer Networking | Tags hardware, intel, ioacceleration, sysadmin, threading | no comments
Posted by Scott Laird
Wed, 23 Feb 2005 03:08:59 GMT
There’s an interesting thread going on right now on the Linux netdev mailing list, speculating about the network accelerator technology that Intel’s been talking about recently. No one’s quite sure what Intel is planning on adding, but for the past several years “network accelerator” has usually meant TCP offload engines (ToE), and Linux’s core networking guys are almost famously anti-ToE. Even though no one really knows what Intel’s up to, there’s a feeling that it’s not just ToE this time.
Several people have pointed out other technologies that can make a huge difference without requiring the sorts of compromises that ToE needs to work. For instance, this post by Lennert Buytenhek suggests that PCI and memory system latency is a big problem, but fixing it can have huge payoffs:
The reason a 1.4GHz IXP2800 processes 15Mpps while a high-end PC hardly
does 1Mpps is exactly because the PC spends all of its cycles stalling on
memory and PCI reads (i.e. ‘latency’), and the IXP2800 has various ways
of mitigating this cost that the PC doesn’t have. First of all, the IXP
has 16 cores which are 8-way ‘hyperthreaded’ each (128 threads total.)
I haven’t paid much attention to Intel’s IXP network processor family in the past, and that may be a mistake–from the description here, the IXP2800 sounds like a cross between Tera’s multithreaded CPU and IBM’s new Cell processor. Tera’s CPU, which was designed to support tons of threads, automatically switches between threads whenever one thread blocked due to I/O or memory access. The goal with Tera was to be able to remain efficient while the gap between CPU and memory speeds continued to grow. The IXP2800 isn’t as ambitious as the Tera, but the fundamental concept looks similar–support lots of threads in hardware, and switch when latency gets in the way. The IXP2800’s threaded CPUs aren’t full-blown processors, though–like the Cell, the IXP2800 contains one main CPU and a cluster of smaller domain-specific processors that are specialized for one specific task.
It’s unlikely that Intel will roll something like this into their Xeon CPUs anytime soon, though. It’s certainly not a quick fix–it’d require major changes in any OS that wanted to make use of it, and would probably take 3-6 years before it was really fully utilized.
Massively-multithreaded CPUs aren’t the only approach that has paid off for dedicated network processors, though. Some of FreeScale and Broadcom’s chips know how to pre-populate the CPU’s cache with headers from recently-received packets. This drastically cuts latency, but it seems to require that the CPU and network interface be very tightly coupled. Reducing the overhead needed to talk to the NIC can help, too–apparently some of Intel’s 865 and 875 motherboards use a version of their GigE chip that is connected directly to the north bridge, bypassing the PCI bus entirely, and some benchmarks show substantial improvements.
Reading the thread suggests that most of the effort going into Linux network optimization in the next few years will be happening on the receive end of things. Over the past several years, most higher-end NICs have added limited support for checksum generation and TCP segmentation offloading (TSO), where the CPU can hand the NIC a block of data and a TCP header template, and then have the NIC produce a stream of TCP packets without requiring the CPU to touch the data at all. Relatively little has happened on the receive side, but this seems to be changing. For example, Neterion’s newest card can separate headers from data, and is nearly able to re-assemble TCP streams on its own, sort of the inverse of transmit-time TSO. It’s not clear how many streams the card can handle at a time, though–even my little web server at home is currently maintaining 384 simultaneous TCP connections, and a busy system could easily have tens or hundreds of thousands of open streams. Odds are, throwing 100,000 steams at the card would run it out of RAM and completely negate any benefit that receive offloading would have. Unless it’s bright enough to be able to handle the 1,000 or so fastest streams and then let the main CPU handle the 99,000 that are dribbling data at 28k modem speeds.
This is a fascinating topic, and I can’t wait to see how this will turn out.
Posted in Linux, Computer Networking | Tags intel, linux, networking, sysadmin, toe | no comments
Posted by Scott Laird
Fri, 11 Feb 2005 21:52:02 GMT
As regular readers know, I recently turned up a new DSL circuit at home, replacing an older, slower line that Verizon had refused to upgrade for months. As part of the upgrade process, I needed to buy a new DSL modem. Instead of using an external DSL modem (DSL-Ethernet bridge would probably be more accurate, but “modem” seems to have stuck), I decided to buy a Sangoma S518 PCI ADSL modem. I had two main reasons for preferring this internal modem to a generic external model:
- Better control over upstream buffering, for better VoIP QoS.
- Better visibility into the modem’s state, so I can syslog minor outages and notice things like speed changes.
I chose the Sangoma model instead of a cheap, generic card because the manufacturer strongly supports its use with Linux, and a number of people on the Asterisk-Users mailing list have recommended it. I paid $115 plus shipping from BSD Mall.
Read more...
Posted in Computer Networking | Tags adsl, dsl, linux, neetworking, review, s518, sangoma | 6 comments
Posted by Scott Laird
Thu, 10 Feb 2005 05:08:58 GMT
I can’t believe it. After months and months of trying to upgrade my DSL, everything is finally up and running on my new DSL line. The line wasn’t supposed to go live until Friday evening, but Verizon sent me mail today with my IP address and claimed that it’s up and running. And, indeed, it is.
Unfortunately, I wasn’t quite as prepared as I thought I was; I’d forgotten to change the DNS TTL on scottstuff.net, so people may have a hard time getting through to this site for a couple days. Oops. Other then that, though, everything seems to be up and running perfectly. The sample size is still kind of small, but smokeping suggests that my average ping time has improved drastically–I was seeing numbers from 25 to 800 ms before. Right now, the line looks flat at 30 ms. That’ll probably break a bit once traffic picks back up on the website, but I should be able to keep it under 100 ms, easy.
I’ll post more details later tonight or tomorrow, along with a review of the Sangoma S518 PCI ADSL card. I have to put the kids to bed first, though.
Posted in Computer Networking | Tags dsl, home, networking, verizon | no comments
Posted by Scott Laird
Wed, 09 Feb 2005 23:21:15 GMT
My DSL modem showed up yesterday, so I dropped it into my gateway box and fired it up. It immediately reported that it was unable to train; there was nothing to talk to on the other end of the phone line yet. Since my official install day is still a couple days out, that didn’t surprise me. Then this morning, I saw this in the logs:
Feb 9 08:19:22 guam kernel: wanpipe1: ADSL Link connected (Down 1792 kbps, Up 448 kbps)
Feb 9 08:19:30 guam kernel: wanpipe1: Link connected!
Feb 9 08:41:03 guam kernel: klogd 1.4.1#11, log source = /proc/kmsg started.
The gap between the second and third lines is the problem–the box went down, hard, right after the DSL line came up. On the other hand, it looks like I’m provisioned above 1.5/384 on the ATM side. Assuming a 20% cell tax, this gives me a usable connection of around 1430 kbps down and 360 kbps up, which isn’t too bad. Now I just have to keep the thing from crashing. I’m rolling my ADSL drivers back from the beta version that I’d started with to the most recent release; hopefully that’ll be good enough to fix my problem.
Posted in Computer Networking | Tags atm, dsl, linux, sangoma | no comments
Posted by Scott Laird
Sun, 06 Feb 2005 00:43:01 GMT
I have a date now for my new DSL line: February 11th. My new DSL PCI card is in UPS’s hands, too. I’m starting to believe that my months-long DSL upgrade quest is nearly complete.
Posted in Computer Networking | Tags dsl, home, networking | no comments
Posted by Scott Laird
Thu, 03 Feb 2005 00:36:10 GMT
My DSL upgrade saga took another step towards completion today, when Verizon installed a new phone line at home. Amazingly enough, it’s already in their DSL database, and Verizon’s DSL sales website allowed me to order DSL. I don’t have an install date for the DSL service yet, but I’m two steps closer to the finish line.
I was surprised that Verizon’s business DSL department gives you a choice of buying a new DSL modem ($99) or providing your own. I took the opportunity to order a Sangoma S518 PCI DSL card instead of an external box. Supposedly, their Linux drivers are solid (Sangoma has been involved with Linux practically forever), and there are a couple big advantages to native DSL interfaces, rather then DSL-to-Ethernet bridges. The biggest advantage is buffering–right now, my DSL modem has at least a couple seconds worth of buffers on it. If I send outbound traffic as fast as I can, I rapidly get to the point where nothing makes it out in under 2 seconds. So, even if I use extreme care in setting up QoS prioritization rules for VoIP traffic, the VoIP packets will still end up stuck in the DSL modem’s buffers. To combat this, I’ve been forced to rate-limit my outbound traffic to about 75% of the theoretical limit; even then it can really suck at times. Several people on Asterisk mailing lists have commented that their S518 has really made their VoIP performance shine.
In addition, since the S518 is directly talking to the phone company, it can tell exactly what speed I’m currently provisioned at and can log problems via syslog. My current nasty DSL box can’t do anything but blink lights at me when there’s a problem.
All in all, it looks like a decent improvement, especially since it’s only $115 or so online.
Posted in Computer Networking | Tags dsl, home, verizon | no comments
Posted by Scott Laird
Tue, 25 Jan 2005 17:14:23 GMT
The Register says that AOL is dropping their support for Usenet news, referring the handful of people who still care to Google Groups. I’m not completely sure how I feel about this–I clearly remember the massive invasion of cluelessness that happened when AOL first invaded Usenet. On the other hand, we’re going to see a new wave of “Usenet is dying” stories, now that AOL is gone.
Posted in Computer Networking | Tags aol, usenet | no comments
Posted by Scott Laird
Sat, 22 Jan 2005 16:22:18 GMT
I feel like I’m finally entering the third act of my DSL upgrade drama. This started over a year ago when I realized that I really wanted faster service then the 768/128 link that I’m paying $80/month for right now. Last June, I asked Verizon to turn up my DSL’s speed, with predictable results–they ran around in circles for over a month, with different departments giving me different answers, ranging from “we already turned it up” to “we lost the order” to “we can’t do that, you need to cancel DSL, wait for it to go dead, and then re-order.”
Amazingly enough, the “you need to cancel” camp was correct–Verizon is unable to increase the speed of my current DSL setup. I played as many cards as I could, pulled the few strings that I have inside of Verizon, had off-the-record conversations with installers, and concluded that I had three choices:
- Put up with my current service, as slow and expensive as it it.
- Cancel DSL, wait two weeks, and re-order.
- Order a second phone line, wait for them to install it, then order DSL on it, then cancel the old line and DSL.
I looked into cable modems, but there’s no way to get a static IP address out of Comcast around here, and I need to run a number of servers. I considered moving my mail, web, and Asterisk servers off onto a hosted system somewhere–that’d let me use Comcast with a dynamic IP, but the cost and complexity of it all just makes it impractical.
So, yesterday, I finally decided to go with plan #3. I ordered a new phone line. It ends up costing me $29 to get it installed and $20/month. Hopefully I won’t have to carry both lines for more then a month. It’s supposed to be up on February 2nd; as soon as that happens, I’m ordering new 1.5/384 DSL service on the line. I’ll cancel the old DSL the same day that the new one comes up–I just need to swing DNS over to the new IP and then wait for a few short timeouts. So, hopefully, this whole saga won’t cost me more then $100 up front. The nice thing is that it’ll end up saving me a few bucks in the long term–with 3x the upstream bandwidth, I can move more phone services over to VoIP, so I can turn off more features on the phone line. That could save me almost $10/month. It’s not a lot of money, but every bit helps sometimes. Besides, it’s mostly the principle of the thing.
Posted in Computer Networking | Tags dsl, home, upgrade, verizon | no comments
Posted by Scott Laird
Tue, 09 Nov 2004 17:42:27 GMT
Slashdot has an article this morning on the OpenBSD people’s new BGP daemon, OpenBGPD. In essence, the OpenBSD people did the same thing that they’ve done repeatedly before, and taken a protocol that didn’t have an open, secure implementation and provided a clean, minimalistic, BSD-licensed tool.
Personally, I find OpenBGPD kind of fascinating, because I’ve worked with router jockeys for years, and I get dragged into “can we run a BGP daemon on this PC” discussions with surprising frequency.
OpenBGPD’s stated goals include this fun little snippet:
Provide a lean implementation, sufficient for a majority. Don’t try to support each and every obscure usage case, but cover the typical ones
And that’s where my problem lies. I don’t think I’ve ever been asked for a “lean implementation” of BGP. Every time I’ve been dragged into a BGP discussion, it’s been because network engineers have been trying to do something bizarre and creative with BGP, and the tools that they’re used to using aren’t sufficient. For instance, at Internap, we wanted to add per-prefix, per-peer prepending for a huge number of prefixes, and we wanted to change the path selection algorithm to include a bunch of extra information that we had on reachability and performance. In other cases, I’ve been asked for simulators and BGP loggers that could feed BGP prefix reachability information into a database. Inevitably, every time someone needed just a “lean implementation,” they’d already have a Cisco box handy and they’d use it instead of monkeying with BGP on a PC.
That’s not to say the PCs make lousy routers or anything like that–the price/performance is impossible to match with anything from Cisco–but that the totals costs involved in any BGP peering that I’ve seen make the cost of the router little more then noise in the equation. If you’re paying tens of thousands of dollars per month for multiple pipes to providers, then what does saving $20k on a router buy you, besides maintenance and reliability headaches and a hard time finding network engineers familiar with your setup? Most of the time, it’s cheaper to spend $20k on hardware and make it up on productivity and reduced downtimes.
So, while OpenBGPD is cool, I’m not sure how useful it really is outside of test labs and maybe small ISPs, if there are any of them left. On the other hand, I’d love a good OpenBGPD-ish OSPF implementation. I’ve played with Zebra, and the whole design of the thing just rubs me wrong (although Quagga might be better). I need to remember to actually give Xorp a try, too. OSPF is more useful inside of existing networks, and it makes a lot more sense on a LAN then BGP does.
When it gets down to it, I suppose my real point is this: it’s largely pointless to scale PC-based routers up to make them compete toe-to-toe against Cisco’s big WAN routers, because the network
costs and the maintenance costs of doing one-off routers works against us. It’s also really hard to get reliable, well-tested WAN interface cards for anything faster then a T1. Try finding a PCI OC-12 POS card with Linux drivers sometime.
On the other hand, other alternatives make a huge amount of sense:
- Scale them down. You can build a cheap Linux router for almost no money these days–look at the Linksys WRT54G.
- Scale them out. Imagine a medium sized company replacing all of their assorted branch office routers with PCs talking to DSL and providing QoS, routing, firewalling, VPNs, VoIP, etc. It’s expensive to do it once, but you can replicate the work onto a hundred devices for very little additional cost.
- Push them into niches. There are cases where the fantastic flexibility of PCs can make them much more useful then an equivalent Cisco. Linux, for example, has no problem running multiple routing tables and a fantastic number of firewall rules. You can do amazingly creative things with just the stock tools, if you can figure out how to use them.
Posted in Computer Networking | Tags bgp, cisco, linux, openbgpd, openbsd, router | 1 comment
Posted by Scott Laird
Mon, 11 Oct 2004 20:03:37 GMT
So, suppose you’re setting up a test environment, and you want a server to be able to handle some improbably large number of IP addresses, like a /16 or even larger. You could just write a script to add them all one at a time, or you could use this little shortcut and add the entire netblock at once:
# ip route add local 10.0.0.0/8 dev eth0 proto kernel scope host table local
That’ll make the entire 10/8 block local to the box. It’ll answer pings sent to any address in the 10.x.x.x space, and you can bind to any address in the netblock and use it as the source address for packets. For all intents and purposes, it’ll act just like the addresses were added with ip addr add or ifconfig.
I used this last week when I needed to simulate a network with 4,000+ hosts all sending UDP traffic to a specific server. It was a piece of cake.
I’ve seen this a few places online, but it’s never been very easy to find. To the best of my knowledge, it’s worked with Linux back to at least 2.0 or so; I’ve used it with 2.4 and 2.6 recently.
Posted in Linux, Computer Networking | Tags howto, linux, router | no comments
