There’s been a fair bit of discussion online about “Horus”, Newisys’s glue chip for building 8–32 processor Opteron systems. The Horus architecture glues 4-processor clumps of CPUs into a larger system. This lets them overcome the Opteron’s 8-processor scaling limit and build really big boxes.
Fortunately, one of Horus’s designers has posted some details to comp.arch. Here’s a quick summary: * Each Horus chip has 7 links; 1 to each local CPU and 3 to other Horus chips. * The inter-Horus link is a proprietary version of HyperTransport, modified to work better over cables. * While they can scale to 32 processors, going past 16 costs extra latency, because Horus only has 3 inter-Horus links. The sweet spot is 8–16 CPUs. * Their reference design uses 2 HT links per quad for I/O. This means that some intra-quad IPC has to go through an intermediary. * They designed for a NUMA factor of 3–remote memory costs 3x what local memory costs. * The Horus architecture can support up to 64 MB of “remote cache.” It’s still unclear if that’s 64 MB total spread across 8 Horus chips, 64 MB per chip, or 64 MB on a dedicated “Horus Cache Chip” that replaces a CPU quad in the design. * The inter-Horus links can be reconfigured and reset on the fly; this will allow for hot-swapping and partitioning.
We can expect to see Horus systems show up late next year. Since AMD’s dual-core Opterons are due in about the same timeframe, we should see some fascinatingly huge PCs by Christmas 2004. The basic design should be similar to Sun’s Enterprise Server xxx systems–a bunch of plug-in slots that take 4 CPUs and a bunch of RAM each. Since Horus clearly wants I/O to be local to each quad, it’s unclear exactly how networking and disk I/O will work–will there be FC and GigE controllers on each quad, and the system routes them through to the backplane? Will each card have I/O on the front panel, even though that makes swapping CPU cards a royal pain? Will the manufacturers include an “I/O node” on the motherboard with its own Horus and a bunch of HT-to-PCI-X bridge chips? Hopefully, we’ll see a few of each design and the market can sort it out.