Professional Geekism

Recently, the company I work for has been looking for a way to simulate poor or slow connections to the server component of their next software product. I’m sure you’d all agree that it’s important to know how well your server compensates for those with less than ideal Internet connectivity, so we decided to try out the traffic shaping facilities provided by m0n0wall.

Now, of course we have Cisco routers and the like, but setting them up for something like this could take ages. With m0n0wall, it’s all web interface, and therefore is far more intuitive. So far I’ve not got around to setting up the actual connectivity, but I thought I would make a post covering the nuances I experienced whilst installing it in a virtual environment.

Now I find working with VMs is fantastic in many respects, though it can sometimes be infuriating to not have physical hardware to work with when something isn’t right. Thankfully, I’ve recently learnt that diagnosing network faults/complexities surrounding VMs and their bridges, is as simple as checking out the contents of /proc/vmnet/ (under Linux 2.6.x – don’t ask me about Windows!) and digesting the information contained there-in.

As it turns out, I thought my host’s bridge configuration was wrong. After many different tests, and lots of cat’ing files within /proc/vmnet, I realised that I wasn’t doing anything wrong (well, maybe not directly.. See below) with my host configuration. The ‘default’ NIC chipset used by VMWare Server is covered by the AMD PCNET32 driver. Under Gentoo/Windows this works great, however, it wouldn’t bring up the correct interfaces within m0n0wall. Cue my helpful colleague, who suggested that adding a EthernetN.virtualDev = "e1000" line to the .vmx file; one for each NIC you have configured. In my example, this was simply a case of replicating the line for both ‘Ethernet0′ and ‘Ethernet1′ (but you may very well require a third NIC.)

Once m0n0wall was booted with the e1000 (A.K.A. Intel Gigabit) NICs, each one was described as ‘up’ immediately. But unfortunately, it was apparent that something else was also going wrong with the way I had chosen to setup the VM itself.

What really threw me was the inability of m0n0wall to intelligently use the hardware I provided it with. I configured a fairly generous machine (given the slim system requirements for m0n0wall) which included 96MB RAM and a 512MB IDE virtual drive. I mounted the CD ISO, and provided added a Floppy Disk drive with a blank disk.

However, even with a blank virtual floppy inserted, there was no provision for formatting this – and no complaint when m0n0wall attempted to save it’s configuration and couldn’t. On top of that, why was it preferring the FDD over the HDD? Granted; neither was formatted, but I didn’t think it would be too hard to provide some warning or message regarding your choice of non-volatile storage.

So, I needed a formatted virtual floppy image. I couldn’t help think how annoying the following options would be:

Power off a (live) Windows guest to add an FDD
Re-compile a Gentoo VM for FDD support, and add the FDD whilst it’s off
Install VMWare on my workstation so I could install XP/Gentoo, just to format an FDD

How about no.

In the end I opted for using a Gentoo (minimal) live CD on the VM I had already prepared for m0n0wall. Using links (yep, the bridging worked just fine) to obtain the latest CF card .img file, I issued the command:

gunzip -c generic-pc-x.xxx.img | dd of=/dev/hda bs=16k

Which unpacks the image and copies it to the block device (no need to format, or partition.)

After that, I abandoned the idea of the CDROM/FDD combo, removed their devices from the VM configuration and started it up again. Of course this time it booted straight from the image on the HDD, and coupled with the ‘new’ NICs, everything began working as it should. I had a web interface in no time at all.

My words of wisdom? Forget using the CD ROM ISO image! Another little tip I could offer, would be to edit the scsi0.present = "TRUE" attribute in the .vmx file to read "FALSE". You don’t need it, but for some reason you can’t disable it from the VMWare console. It helped to speed up m0n0wall’s boot time, due to removing the need to ‘wait at least 15 seconds for SCSI devices to settle’.

Over-all, I’m quite liking it so far. I may write again soon with some reflections. In the mean time, where did I put that WRAP box?

Had a slightly embarrassing problem yesterday, although I’m not entirely sure it’s my (our) fault…

One of our servers, for historical reasons, is actually hosted in the same rack at UKS as four more; all owned by one of our larger customers. We happen to manage all of these servers, too.

Anyway, sometime around 4pm yesterday (just as I’m about to leave – typical) one of the redundant PSU modules in their primary VM host, and as a horrible, unexpected side-effect, it blew the fuse in the rack’s power bar! Which really shouldn’t happen, but.. Well.. It did!

UKS inspected the issue as soon as we noticed, and promptly replaced the fuse. Of course, [almost] all of the machines are set to ‘power-on after AC power loss’ in their BIOS settings. As they all do, the resultant surge in power draw once-again trips the fuse.

We were blamed for not using staggered start-up on our APC PDU. But they don’t all support it? Why should the fuse blow at the sight of four >2A servers starting up? It’s not like the rack was purchased on a fixed ampage – the servers are all paid on single-server hosting packages!

Anyway, as I’m actually off on holiday today, I’ve had to convince the admin girl to drop a spare PSU down to Redditch.. No big deal, as UKS are more than capable of swapping the PSU out.

The bigger problem was the extended downtime of our main web server. It hasn’t got a working LED, so it was “quite difficult” for the on-site engineer to determine if it was up or not. I suppose checking the airflow through the back of the machine, or the multitude of drive lights weren’t obvious options…

After a night of down-time, it’s actually up again. The backup VM didn’t go quite to plan, which I think we’ll need to investigate in the future. And I suppose I’ve got a LED to fix when I’m next in the data centre!