Archive for the ‘Storage’ Category.

Pretending to be a Solaris admin

I’m always, always forgetting how to discover the available disks on a Solaris/OpenSolaris machine.

As I was having another (un-successful) crack at getting a disk controller (other than the motherboard’s IDE controller) to work with Nexenta Core v2, I’d again forgotten how I was meant to discover the disks as-probed by the OpenSolaris kernel.

Of course, Nexenta includes Ubuntu Hardy’s userland tools, but anything kernel/device-related is still very different to what I’m used to.

I finally found a particularly well-written post by Pascal Gienger, whom notes that:

First we will try to look up the disks accessible by our system:

# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0d0
/pci@0,0/pci-ide@1f,1/ide@0/cmdk@0,0
1. c1d0

/pci@0,0/pci-ide@1f,1/ide@1/cmdk@0,0
Specify disk (enter its number): ^C

Type CTRL-C to quit “format”.

If your disks do not show up, use devfsadm:

# devfsadm
# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0d0
/pci@0,0/pci-ide@1f,1/ide@0/cmdk@0,0
1. c0d1

/pci@0,0/pci-ide@1f,1/ide@0/cmdk@1,0
2. c1d0

/pci@0,0/pci-ide@1f,1/ide@1/cmdk@0,0
3. c1d1

/pci@0,0/pci-ide@1f,1/ide@1/cmdk@1,0
Specify disk (enter its number): ^C

You’ll notice that the virtual disks are mapped as IDE/ATA drives, so the disk device names don’t have a target specification “t”.

Which has helped me to finally find out that my second-hand (i.e. ‘borrowed’ from an old work machine) Adaptec RAID card, doesn’t work with Nexenta Core v2. Still, Core v3 will be out in a few months – maybe I’ll try again then.

Also worth noting, as it may be useful, iostat -En prints out similar information useful when searching for disks to use with ZFS.

Optical drive firmware updating in Linux

I recently needed to burn a copy of Windows 7 Pro but realisd that I’d unfortunately run out of blank DVD-Rs long ago. Fear not, for I live near an Aldi supermarket, whom sell everything dirt cheap. DVD-R’s a DVD-R, right?

Wrong. I tried at least three of the twenty I purchased (for a few quid) and none of them would even begin writing. Brasero/K3B both complained about incompatible media types.

Remembering that my DVD drive, a trusty NEC 3500A, was designed, built and purchased somewhere between 2004 and 2005 (4-5 years ago at this point) and that I hadn’t ever updated the firmware, I set about researching ways and means into doing this.

I came across this website, run by a pair of firmware hackers named Liggy and Dee whom have (between them) released, and continue to host, many firmware releases (both official and unofficial) for a wide variety of NEC optical drives.

What’s more, their binflash (or ‘necflash’) utility was even released as a Linux binary and it even provides compatibility for reading the official NEC .exe firmware releases! I was sceptical that it would work under Ubuntu 9.10 at first, but much to my delight it worked perfectly. With a little reading, I was able to dump my current firmware (2.16) to file and subsequently flash two different firmware releases: 2.58 (an OEM firmware release) and the latest, official NEC firmware 2.1A release.

The full output of my escapades for anyone curious:


~$ sudo ./necflash -flash -v -s Desktop/NECND350_v21A.exe /dev/sg2
Binflash - NEC version - (C) by Liggy and Herrie
Visit http://binflash.cdfreaks.com

Identified drive: 4 - 3031
Detected drive from Firmware: 4

You are about to flash your drive with the following firmware:

Vendor: _NEC
Identification: DVD_RW ND-3500AG
Version: 2.1A

Remember no one can be held responsible for any kind of failure!
Are you sure you want to proceed? (y/n) y

Entering safe mode
Sending firmware to drive at 0x006000
Sending firmware to drive at 0x00e000
Sending firmware to drive at 0x016000
Sending firmware to drive at 0x01e000
Sending firmware to drive at 0x026000
Sending firmware to drive at 0x02e000
Sending firmware to drive at 0x036000
Sending firmware to drive at 0x03e000
Sending firmware to drive at 0x046000
Sending firmware to drive at 0x04e000
Sending firmware to drive at 0x056000
Sending firmware to drive at 0x05e000
Sending firmware to drive at 0x066000
Sending firmware to drive at 0x06e000
Sending firmware to drive at 0x076000
Sending firmware to drive at 0x07e000
Sending firmware to drive at 0x086000
Sending firmware to drive at 0x08e000
Sending firmware to drive at 0x096000
Sending firmware to drive at 0x09e000
Sending firmware to drive at 0x0a6000
Sending firmware to drive at 0x0ae000
Sending firmware to drive at 0x0b6000
Sending firmware to drive at 0x0be000
Sending firmware to drive at 0x0c6000
Sending firmware to drive at 0x0ce000
Sending firmware to drive at 0x0d6000
Sending firmware to drive at 0x0de000
Sending firmware to drive at 0x0e6000
Sending firmware to drive at 0x0ee000
Sending firmware to drive at 0x0f6000
Sending firmware to drive at 0x0fe000
Sending checksum to drive
Erasing flash block 2
Erasing flash block 3
Erasing flash block 4
Erasing flash block 5
Erasing flash block 6
Erasing flash block 7
Erasing flash block 8
Erasing flash block 9
Erasing flash block 10
Erasing flash block 11
Erasing flash block 12
Erasing flash block 13
Erasing flash block 14
Erasing flash block 15
Erasing flash block 16
Erasing flash block 17
Erasing flash block 18
Writing flash block 2
Writing flash block 3
Writing flash block 4
Writing flash block 5
Writing flash block 6
Writing flash block 7
Writing flash block 8
Writing flash block 9
Writing flash block 10
Writing flash block 11
Writing flash block 12
Writing flash block 13
Writing flash block 14
Writing flash block 15
Writing flash block 16
Writing flash block 17
Writing flash block 18
Leaving safe mode

Whilst the 2.58 OEM release didn’t fix my problems, 2.1A did and I now have a freshly-burnt copy of Windows 7 Pro to go and play games with. Nice one, Liggy & Dee. :)

I’m spoilt these days…

I’ve just had to setup Windows on a physical machine (shudder) to control and monitor the IOMeter disk benchmarks that are needed for my final year project. I didn’t try to run it in Wine, but I suppose I should’ve. Needless to say, I do require it to be perfect in order to maintain the fairness of my testing, so Windows was unfortunately my first choice.

Due to the age of the hardware I had lying around; an old Athlon XP-M system with an Abit NF7-S 2.0 and 512MB of ‘borrowed’ memory (thanks Ian), it was safe to say that it wouldn’t be any good installing Vista on it. Therefore I downloaded and burnt an XP ISO from my MSDN account and set about installing XP to the 200GB SATA drive I had (thanks Neillans, actually!)

The Abit NF7-S range of boards (particularly the V2.0) were highly-regarded during their hay-day: a testament to Abit’s awesome legacy. Not least for their inclusion of SATA ports way back in 2002, when Serial ATA was a relatively new feature on desktop boards. It even included basic RAID functions across the twin ports, courtesy of the Sil3112r chipset, which is still sold today if you look hard enough. When this was my main motherboard I actually ran a pair of 36GB WD Raptors in RAID-0 (scarily the same pair I use as my root drive now! I’m poor, OK?) and everything worked extremely well.. I never had a single problem with it.

But fast-forward to installing XP onto a SATA single disk, and I was stumped for a little while. Aside from the faff in convincing my floppy drive to work with the board (I’d previously disabled it via three, separate options in the BIOS — nightmare) I then had XP’s installation looping continuously, instead of booting from the HDD to continue with the second phase of the installation. It was almost as if XP was failing to write NTLDR into the MBR, somehow.

Now by convention on modern motherboards, SATA ports can typically be set to three modes: RAID, AHCI, and IDE. The latter of which is used purely for compatibility with older operating systems. However, the ‘RAID’ mode typically prevents that particular disk from being presented as a possible boot disk by marking it for use within RAID arrays only. It’s all fairly self-explanatory, however.

However, within the NF7-S’s BIOS, there are no such options. You can either enable/disable the SATA chipset, and optionally enable/disable the ‘SATA RAID ROM’, which you would believe would be only required if creating RAID arrays. I didn’t wish to use the RAID features and therefore I didn’t intend on ear-marking the disk as a RAID disk, as I wanted to boot from it. Sounds sensible, right?

Sadly, unless this ROM option is actually enabled, regardless of whether or not I wished to use any of the RAID features; the disks will not be presented as boot disks. Quite why there is even an option in the first place is beyond me! Because of this, the XP installation CD was failing to find a suitable boot disk and was therefore intent on looping endlessly through the first phase of the installation process. Fun times…

It has since occurred to me just how far SATA adoption and usability has actually come in the last 5-6 years. With most chipsets now natively including anything from two to eight AHCI SATA ports, as well as incorporating much better integration into the BIOS menus. Similarly, with natively AHCI-aware operating systems such as Linux, Solaris (and friends), many BSDs, Vista (and Windows 7!) now becoming largely common-place, there are few reasons for any of the IDE-compatibility options any longer.

That is, unless you’ve only got a single-core processor, 512MB of memory and an old, awkward (but great) motherboard. I just wish the IOMeter devs would consider creating a GTK+/QT4 front-end for dynamo! :)

Classic XKCD

For those of you who don’t follow XKCD, you really are missing out. It’s just genius, and today’s comic really did tickle me.. (Click for a larger image!)

This really is a true story, and she doesn't know I put it in my comic because her wifi hasn't worked for weeks.
This really is a true story, and she doesn’t know I put it in my comic because her wifi hasn’t worked for weeks.

Well, I laughed at least. And it reminds me of a famous euphemism, too!

Nick: “Jon, why have you locked your door?
Jon: “I’m re-compiling my kernel!

It’s alive!

I’ve recently been building servers again. Aside from the usual 2U stuff, I thought I’d show a few pictures of the current project I’m working on. This 4U Supermicro chassis is destined to be used as our backup/storage server at the co-lo facility. VM backups, database backups, general file store, etc. etc.

ZFS Server: 24 hotswap bays

Plenty of drive bays there (24 to be exact).

ZFS Server: a view from above

Here you can see how neat it is. Partly because of the good design of the case, and partly because of the tight integration with Supermicro’s own boards. The shroud that ducts air over the CPU also works wonders.

ZFS Server: very, very loud fans.

As you can imagine, it sounds like a jet taking-off when it’s going at full pelt. I wonder if co-los typically have an ‘upper noise limit’? :D

I’ll put more detail down about what I’m doing with it a bit later… I’m currently testing all manner of Solaris-based distributions (a learning experience in its own right) with some funky zpool configurations. More to come!

Crossing the Gigabit barrier

Recently, I’ve been charged with investigating into faster-than-gigabit networking, in an effort to switch our VM hosts away from local storage to an NFS-based NAS system. There are a few reasons for doing this; the greatest of which is Sun’s ZFS file system.

ZFS, for those of you who aren’t familiar, has really shaken-up the world of file systems recently, as it changes almost everything that we perceive about a modern-day file system. On top of these fundamental changes (which I won’t go into detail about here) the ZFS developers have added some really neat features, such-as zero-cost snaphosts, replication between machines, RAID-Z, and quite a lot more.

It’s the promise of these features that has prompted our change over to a NAS-based storage system. Given that we can completely replace our current system of identical live/backup hosts, with slow backup scripts and drbd mirroring, it’s quite promising to think what we can achieve.

The problem is transport. And keeping fast transport. Given the extra overheads of IP/NFS that NAS brings (weighted against the benefits given ZFS over the more efficient use of raw disks in a SAN) it’s been deemed that a single gigabit link just won’t be up to the demanding task. The problem is that once you decide to cross the gigabit ‘barrier’, your costing simply spirals uncontrollably skyward. :(

There are a few options available to achieve a decent throughput:

  • Multiple, bonded (802.3ad) gigabit links – cheap-ish, but some multiport adapters really aren’t cheap.
  • 4Gbit FibreChannel – readily available Solaris support, but over-shadowed by 10GigE/Infiniband and requires costly HBAs with extremely expensive XFP/SFP+ modules.
  • Infiniband (SDR 4x, 10Gbit) – really, really cool, but there’s a huge lack of support in Solaris.
  • 10Gigabit Ethernet – very new, and switches are extremely expensive (laughably so, think $20,000 for a 24-port switch + Gbics!) mainly due to the lack of 10GBase-T support (meaning we need 10Base-CX4 or some Fiber-based solution.)

So what’s the answer? We’re not a Fortune 500 company, so most of this is still out of reach. On top of it all, we need to rely on Solaris for ZFS – an operating system which seems to have very little manufacturer support, despite its presence in the cluster and virtualisation markets. Sun’s Hardware-Compatibility List is almost devoid of recent Infiniband/10GBase-T adapters, particularly in PCI-E interconnect guises.

It wouldn’t be so bad if some manufacturer had thought to release a small-scale, 8-10 port 10GigE, 10GBase-T switch. They just don’t exist.. At present, it’s quite likely that we’ll have to dump the idea of a switched fabric altogether, opting instead of multiple point-to-point links.

It seems we’re either just a few years ahead of ourselves, or really, really out of our depth.