As I was having another (un-successful) crack at getting a disk controller (other than the motherboard’s IDE controller) to work with Nexenta Core v2, I’d again forgotten how I was meant to discover the disks as-probed by the OpenSolaris kernel.

Of course, Nexenta includes Ubuntu Hardy’s userland tools, but anything kernel/device-related is still very different to what I’m used to.

I finally found a particularly well-written post by Pascal Gienger, whom notes that:

First we will try to look up the disks accessible by our system:

# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0d0
/pci@0,0/pci-ide@1f,1/ide@0/cmdk@0,0
1. c1d0
/pci@0,0/pci-ide@1f,1/ide@1/cmdk@0,0
Specify disk (enter its number): ^C

Type CTRL-C to quit “format”.

If your disks do not show up, use devfsadm:

# devfsadm
# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0d0
/pci@0,0/pci-ide@1f,1/ide@0/cmdk@0,0
1. c0d1
/pci@0,0/pci-ide@1f,1/ide@0/cmdk@1,0
2. c1d0
/pci@0,0/pci-ide@1f,1/ide@1/cmdk@0,0
3. c1d1
/pci@0,0/pci-ide@1f,1/ide@1/cmdk@1,0
Specify disk (enter its number): ^C

You’ll notice that the virtual disks are mapped as IDE/ATA drives, so the disk device names don’t have a target specification “t”.

Which has helped me to finally find out that my second-hand (i.e. ‘borrowed’ from an old work machine) Adaptec RAID card, doesn’t work with Nexenta Core v2. Still, Core v3 will be out in a few months – maybe I’ll try again then.

Also worth noting, as it may be useful, iostat -En prints out similar information useful when searching for disks to use with ZFS.

Plenty of drive bays there (24 to be exact).

Here you can see how neat it is. Partly because of the good design of the case, and partly because of the tight integration with Supermicro’s own boards. The shroud that ducts air over the CPU also works wonders.

As you can imagine, it sounds like a jet taking-off when it’s going at full pelt. I wonder if co-los typically have an ‘upper noise limit’?

I’ll put more detail down about what I’m doing with it a bit later… I’m currently testing all manner of Solaris-based distributions (a learning experience in its own right) with some funky zpool configurations. More to come!

ZFS, for those of you who aren’t familiar, has really shaken-up the world of file systems recently, as it changes almost everything that we perceive about a modern-day file system. On top of these fundamental changes (which I won’t go into detail about here) the ZFS developers have added some really neat features, such-as zero-cost snaphosts, replication between machines, RAID-Z, and quite a lot more.

It’s the promise of these features that has prompted our change over to a NAS-based storage system. Given that we can completely replace our current system of identical live/backup hosts, with slow backup scripts and drbd mirroring, it’s quite promising to think what we can achieve.

The problem is transport. And keeping fast transport. Given the extra overheads of IP/NFS that NAS brings (weighted against the benefits given ZFS over the more efficient use of raw disks in a SAN) it’s been deemed that a single gigabit link just won’t be up to the demanding task. The problem is that once you decide to cross the gigabit ‘barrier’, your costing simply spirals uncontrollably skyward.

There are a few options available to achieve a decent throughput:

• Multiple, bonded (802.3ad) gigabit links – cheap-ish, but some multiport adapters really aren’t cheap.
• 4Gbit FibreChannel – readily available Solaris support, but over-shadowed by 10GigE/Infiniband and requires costly HBAs with extremely expensive XFP/SFP+ modules.
• Infiniband (SDR 4x, 10Gbit) – really, really cool, but there’s a huge lack of support in Solaris.
• 10Gigabit Ethernet – very new, and switches are extremely expensive (laughably so, think $20,000 for a 24-port switch + Gbics!) mainly due to the lack of 10GBase-T support (meaning we need 10Base-CX4 or some Fiber-based solution.)

So what’s the answer? We’re not a Fortune 500 company, so most of this is still out of reach. On top of it all, we need to rely on Solaris for ZFS – an operating system which seems to have very little manufacturer support, despite its presence in the cluster and virtualisation markets. Sun’s Hardware-Compatibility List is almost devoid of recent Infiniband/10GBase-T adapters, particularly in PCI-E interconnect guises.

It wouldn’t be so bad if some manufacturer had thought to release a small-scale, 8-10 port 10GigE, 10GBase-T switch. They just don’t exist.. At present, it’s quite likely that we’ll have to dump the idea of a switched fabric altogether, opting instead of multiple point-to-point links.

It seems we’re either just a few years ahead of ourselves, or really, really out of our depth.