It’s been two weeks (roughly) since I started my placement, and there has been quite a bit to learn.
My first week has given me time to get to grips with my new environment, and also gave me a steep introduction into the systems that I’m going to be responsible for in the forseeable future. That is, a large amount of Gentoo co-located servers (some running VMs, which in turn run the services, and some caring for VM back-ups via DRBD) and an Asterisk PBX, which itself is actually hosted on a Gentoo server.
Due to.. Something… Wonderful, there isn’t a network/server map in sight. So all of this was pretty confusing to start with, and ust trying to envision the layout of the entire system from one person’s explanation was very difficult indeed. Thankfully now, I’m pretty well-versed, but there are still some aspects I’m yet to grasp. Though it shouldn’t take long now..
What is particularly interesting is the variance of the work. In the early days I was doing rudimentry work; changing passwords on the 3Ware RAID card web interfaces to something new. This was fine until I realised that the passwords had an 8-character limit, and thus the passwords I had picked were far too long. This wasn’t noticed until after I’d been through almost all of the RAID cards… Even the MD was laughing from his office when that hit the e-mails.
On the other hand, last thursday myself and Jon had to visit UK Solutions to replace some hard drives in a server, and remove the aging RedHat install on that machine, along with another, in favour of shiny new Gentoo installs. Unfortunately the phrase ‘Shiny Gentoo install’ is somewhat of an oxymoron; it’s anything but shiny, glossy, or any other descriptive word for ‘pretty’. We’re talking about a purely CLI installation. Any harder and it’d be Linux From Scratch…
But if it wasn’t hard, it’d be boring! And it’s definitely a nice change to get out of the office and into the .. frying pan. Anyone who’s had to sit between the arse-end of two server racks will understand my synonym. Cool, it most-definitely is not. But I had a job to do, and that was to install Gentoo onto the new RAID-5 array of recently-purchased 500GB Seagate drives.
The basic installation of Gentoo is by-far the most confusing – there’s a reason the install manual has probably been re-written 5001 times in order to make it simpler for first-time users, and anyone who remembers their first attempt at a Gentoo install will be nodding violently right about now. Thankfully once that’s done (and we head back to HQ in search of a more comfortable, ssh-supported server configuration environment) things do get easier. You get used to simply typing ’emerge application-name‘, editing the necessary config file (or stealing it with scp from a working server 😉 ) and using the necessary init.d script to restart the daemon.
Sounds groovy, but let me take you back to the RAID-5 configuration that I was working on in those two fun-packed days. RAID-5, for the unitiated, provides a fall-back in server environments in the event that one of your drives fails. For instance, in a three-drive RAID5 array of 500GB disks you will have a striped array of 1000GB in total, with one third of each drive taken up by parity data. The downside being that you don’t get the full 1500GB to use. The upside is if any one of the drives in the array fails, you can connect a new drive, and the data on the lost drive can be ‘re-constructed’ from the parity data contained on the remaining two drives. Our nice 3Ware RAID cards also have support for ‘hot spare’ drives, so if one drive fails – the controller can bring in the fourth drive and re-construct the failed drives’ data without anyone having to physically visit the machine.
There are two problems with RAID-5:
- It’s slow. Having to write parity data with each write is hugely taxing on write speed. Reads are generally similar to that of RAID-0 though.
- If two drives fail at once, you’re screwed. There’s not enough parity data contained on the remaining drive to construct two failed drives. RAID-6 was implemented to cover this, but isn’t widely used. There’s two lots of parity to write, and THEN you need more disks (4) for only the same volume size.
Anyway, what happens just as I’ve finished configuring my first Gentoo server? Yep, one of the drives drops out of the array! The RAID card attempted to replenish the array from the hot spare, but low and behold … THAT FAILED TOO! 🙁
TYPICAL.. So those drives shall be going back under RMA, and we’re spending a bit more on some Seagate enterprise-class drives (which we should’ve had in the first place really, but then even desktop drives shouldn’t have failed that prematurely!) which shall then require another trip to UKSolutions in order to install them! Oh, and ANOTHER Gentoo install. Plus a few more. And we have to re-wire the internal LAN switch, which is going to be a massive job due to the lovely job of cable tidying that UKS has done.
I think I’m having fun! 🙂 </geek>
More another time, I think. 🙂