Older blog entries for Boris (starting at number 114)

A kernel ghost story...

I just discovered a real nasty thing with grub, raid and RedHat 7.3. It goes like this:

I just spent the last 2 days building a server. I have RedHat 7.3 as the os, 2 ide drives running software raid 1, and these are using ext3 as the filesystem.

7.3 has been out a while and a lot of the packages are out of date, so I do a update all RPMs to bring the server up to spec. This includes updating the kernel to 2.4.18-5. It shouldn't be a problem.

I check /boot, and it's all been updated to the correct kernel. I check grub, and these files are pointing to vmlinuz 2.4.18-5. I'm happy and I reboot.

Grub loads. It prompts with "Red Hat Linux (2.4.18-3)". I think "What the...?" I try booting it to see what happens, expecting obscure error codes, kernel panics, and the end of civilisation as we know it.

It starts booting, then it loads vmlinuz-2.4.18-3, alogn with the rest of the OS. Of course all the device drivers fail because the symbols are wrong. I'm amazed, astounded and astonished. I search for signs of 2.4.18-3. It's not there. Totally absent. Not a 2.4.18-3 anywhere on the server. Totally eradicated. I check two different ways just to be sure. I am totally confused, confounded and chastened. The server just semi-successfully booted a ghost; I had thought I felt a chill enter the room. I wrote that off to the AC coming on, maybe it hadn't. I think "Buh?"

I search the internet for answers - it knows everything - this has to have happened before. This will be fixed in two minutes. No problem.

I find lots of questions relating to this. No answers. Nothing. Not a clue. Not a hint.

This is a problem.

I wonder if maybe civilisation did end when I booted a kernel that didn't exist and could only have been a ghost. I idly wonder if Dr. Egon Spengler would have any ideas. He seems like the type that would know about a ghost kernel.

I assume grub is storing the config file elsewhere. It doesn't explain how it's loading the previous kernel though; it can't be storing that much data in the MBR. I try running grub-install. It won't run because it says "/dev/md0 does not have any BIOS drive". Of course not. I try forcing it to do something. Anything. Because I have everything on a raid device it refuses. I try everything short of trying to find a sharp stick and poking it, but it just won't work. By now I'm looking at losing 2 days of a server install because I have no way to get all my config files off the machine (no devices are working, remember?). By now, I'm beginning to drool and idly wondering if it will stain my shirt.

I decide that the system is pretty irreprable, and elect to try desperate measures.

init 1
...
umount /dev/md0
mount -t ext3 /dev/hda1 /tmp
cd /tmp/boot
ls

Suddenly I see the old boot directory. vmlinuz-2.4.18-3 is there, along with the old grub files that it won't change. I wipe the drool off my chin, and type the following:

mkdir old
mv * old
cp -r /boot/* .
ls

All new files are there, I double check permissions and links. I type: sync; init 6

Grub appears on the screen. It happily prompts me to boot "Red Hat Linux (2.4.18-5)". The new kernel boots and all is right with the world again. I've exorcised the ghost of 2.4.18-3 and I can go back to breaking MySQL again.

Wow. It's so cold in here, I think my PC is running faster

It's build a borg week. Here's the recipe:

Take 4 SCA drives from some Dell servers, 2 IDE drives from a couple of other PC's. Grab a SCSI card from yet another PC, use the motherboard from an old server, and the case from another server.

Upgrade the CPU, and add some extra memory.

Fit the motherboard into the case and attach cables as required.

Add an adaptor so that the SCA drives will work with standard scsi, and fit into drivebays

Fit the SCSI card, and cable this to the SCA disks

Fit ide drives, and cable to the motherboard

Configure IDE drives as a Raid 1 linux array, and install Red Hat; configure as a bug tracking server

Cable SCSI drives to a Windows NT cluster

You're done bulding a borg...

Actually I think I'm going to have to remove the SCSI card from the setup; it seems to be confusing the NT raid card, and is probably unnessecary.

Now to finish configuring the cluster, and testing failover...

Man, I've been having some wierd problems with Borland C++. I can't get librarys to link, io.h won't #include (well it includes, it just won't recognise anything that io.h should provide...) It's being a royal pain. I wish I knew what was up with it.

Hmm. It's been a while. Lets see: I took off to visit my folks in Ireland. Did that for about 3 weeks came back and started looking for work.

Just finished debugging the christmas tree lights and now getting onto the decoration part.

I need to find out what is contained in the Philips 7.12 DVD firmware upgrade.

And thats about that...

12 Oct 2001 (updated 12 Oct 2001 at 06:27 UTC) »

So now I'm unemployed again. My contract ended on Monday. This came as a surprise to all involved, as everyone else thought it was at least Friday. Contract Services notified security to revoke my pass. They didn't notify me, my firm, or my manager about this, so nobody else knew. This came as quite a surprise to me on Tuesday morning when I turned up for work and was refused entry into the building. I had to get my manager to sign me in as a visitor so I could at least clear out my desk, and finish up stuff.

Quite reasonably all parties involoved were more than a little annoyed about the way my contract ended, especially me. I'd been after my manager for weeks to try and get a definite finish date, but he couldn't get any information about it.

Whats even more stupid is that although they need people in the department, they can't hire anyone. They can hire contractors, but only for a year (because legal is scared of the permatemp problem Microsoft had). So they let us go, then hire more contractors.

My firm told me that they had a position available at Microsoft in Redmond. There were a few problems with this:

  • I don't like Microsoft, and I have ethical problems with working for them
  • Redmond is on the other side of Seattle, and it's about a 3 hour commute given Seattle traffic
  • Microsoft don't pay contractors that well.

So here I am updating my resume, and I've decided to go back to Ireland for 3 weeks and visit family and friends.

...and then there was one. Probably in 2 weeks there will be none. I've not been officially told yet, so I guess I'll keep coming in until they stop paying me...

Scott finished on Friday. Management have told me that they can't renew me or hire me after the year, but they can replace me with another contractor. How stupid is that?

Clinton didn't get called up (yet). There are a few linux guys still here, but none who are doing 32 bit validation. I guess one of them will have to learn.

18 Sep 2001 (updated 18 Sep 2001 at 21:08 UTC) »

So, now I'm the last contractor in our department. It's a bad situation too. The second last contractor, Scott, is a Novell guy. During the last round of layoffs they fired a Novell engineer, leaving just Scott and Clinton. Clinton is currently back in the army on reserve duty. Theres no idea when he will be back. This leaves us with zero Novell people. Actually half a person. One of the 64 bit linux guys will be taking over Novell duties, but he will be also have to do 64 bit stuff...

No word on my job, but I've only got a couple more weeks before I'll find out... Most likely I'll be looking for a new job soon...

It's a bad hardware day for me.

First fibre channel card I try to use isn't tagged, so it can't be logged in the database. I put it in the pile for tagging.

I pull another card to use, and I find that the insertion point (the shiny metal part) is loose and has broken the connections from the card to the socket.

Third one I pull is fine. So I try to insert a Adapetec 3950 into the system only to find that some idiot had screwed the insertion point in backwards...

I hate UnixWare.
I HATE UnixWare.
I HATE UNIXWARE.

I'd much rather be spending time with my wife than trying to get this damn unixware install going.

Ahem. Thank you. Nothing to see here. Move along...

105 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!