Crossing the Gigabit barrier
Recently, I’ve been charged with investigating into faster-than-gigabit networking, in an effort to switch our VM hosts away from local storage to an NFS-based NAS system. There are a few reasons for doing this; the greatest of which is Sun’s ZFS file system.
ZFS, for those of you who aren’t familiar, has really shaken-up the world of file systems recently, as it changes almost everything that we perceive about a modern-day file system. On top of these fundamental changes (which I won’t go into detail about here) the ZFS developers have added some really neat features, such-as zero-cost snaphosts, replication between machines, RAID-Z, and quite a lot more.
It’s the promise of these features that has prompted our change over to a NAS-based storage system. Given that we can completely replace our current system of identical live/backup hosts, with slow backup scripts and drbd mirroring, it’s quite promising to think what we can achieve.
The problem is transport. And keeping fast transport. Given the extra overheads of IP/NFS that NAS brings (weighted against the benefits given ZFS over the more efficient use of raw disks in a SAN) it’s been deemed that a single gigabit link just won’t be up to the demanding task. The problem is that once you decide to cross the gigabit ‘barrier’, your costing simply spirals uncontrollably skyward. 
There are a few options available to achieve a decent throughput:
- Multiple, bonded (802.3ad) gigabit links – cheap-ish, but some multiport adapters really aren’t cheap.
- 4Gbit FibreChannel – readily available Solaris support, but over-shadowed by 10GigE/Infiniband and requires costly HBAs with extremely expensive XFP/SFP+ modules.
- Infiniband (SDR 4x, 10Gbit) – really, really cool, but there’s a huge lack of support in Solaris.
- 10Gigabit Ethernet – very new, and switches are extremely expensive (laughably so, think $20,000 for a 24-port switch + Gbics!) mainly due to the lack of 10GBase-T support (meaning we need 10Base-CX4 or some Fiber-based solution.)
So what’s the answer? We’re not a Fortune 500 company, so most of this is still out of reach. On top of it all, we need to rely on Solaris for ZFS – an operating system which seems to have very little manufacturer support, despite its presence in the cluster and virtualisation markets. Sun’s Hardware-Compatibility List is almost devoid of recent Infiniband/10GBase-T adapters, particularly in PCI-E interconnect guises.
It wouldn’t be so bad if some manufacturer had thought to release a small-scale, 8-10 port 10GigE, 10GBase-T switch. They just don’t exist.. At present, it’s quite likely that we’ll have to dump the idea of a switched fabric altogether, opting instead of multiple point-to-point links.
It seems we’re either just a few years ahead of ourselves, or really, really out of our depth.