Advogato blog for scoop

1 Apr 2002

Mon, 1 Apr 2002 12:30:07 GMT

No, this ain't no April fools, it's the uncensored truth. NGI, my DSL provider that I only signed up with last November, claims 98% availability of their lines per month, 8 hours response time upon a line failure and 24/7 reachability of the hotline.

That said, it does not apply to what I'm suffering from for the past 5 or so days. My DSL line crapped out on Friday at 11am. Friday was a national holiday and, although they claim for 24/7 support, their hotline was not staffed. They don't even have any answering machine!

When I finally reached one of their support guys the line had already been unavailable for 24 hours. It took them another 6 hours to get it fixed.

As if that wasn't enough this same error occured to me Sunday night at midnight when, who am I kidding, again noone was answering the hotline number. Since (I am sooooo lucky this weekend is the easter weekend) Monday (today) is a national holiday as well, I won't get ahold of anyone there before 9am tomorrow (Tuesday) morning and that makes 31 hours of unavailability not counting the time they actually need to fix it after I was able to tell them about my problems. That means I've been stuck for 72 hours of outage for one weekend.

All in all, this renders me totally pissed and I guess I really need a new DSL provider Real Soon Now.

30 Jan 2002

Wed, 30 Jan 2002 19:29:10 GMT

I'm getting married!

As the countdown on poocs.net indicates, I'm getting married on May 25th, 2002. I know my girlfriend for more than 8 years now and we've been engaged for the past half year. Now it's about time we're getting married after all :)

We haven't outlined any of the ceremony of after-show party yet, but I'm sure it'll be fun and worthwhile. I think we'll be in need of a decent location now. Next steps probably include spec'ing out the guestlist and some sort of menu... (help!)

26 Jan 2002

Sat, 26 Jan 2002 10:17:28 GMT

freshmeat's quad database servers finally booting again
or: The Mylex saga

On Wednesday, Jan 16th freshmeat suffered from a major database server outage. Both quad CPU database servers crashed hard, one after the other. As we knew we had a swap problem on these machines anyway, we upgraded them from Red Hat 7.1 to 7.2. Since 7.2 shipped with a 2.4.7 kernel we were trying to upgrade it to 2.4.9 from the Red Hat updates repository. Sad but true, this kernel didn't like our DAC960 Mylex controller and panic'ed on boot. The 2.4.7 kernel didn't survive the load MySQL was putting on it so freshmeat went down for 6 hours straight.

When we finally got it up and running again, it was in a very flakey state, since we had to disable searches to make the machines survive with kernel 2.4.7 as we didn't find a single kernel besides that one from the 7.2 install CDs that would boot on these machines. Going back to 2.2 wasn't an option either, since there's a 2GB memory limit and the machines are equipped with 4GB each.

Repeated attempts to self-compile kernels failed also, even with an updated DAC960 driver we found on the net. Therefore we decided to replace the quad machines by dual CPU machines without that dubious DAC960 controller.

After two days of testing with three dual CPU machines with 2GB memory each, we decided to switch the site to exclusively use these machines the coming day.

When I woke up that day I noticed a security advisory from Red Hat in my inbox with a cute little note appended to it saying "* Updated DAC960 driver". I contacted OSDN's netop staff to talk them into another kernel upgrade attempt with the quad machines, which happened just yesterday. And.. who would've guessed, they're booting!

We threw away the dual CPU plans and brought the site back up on the quad machines after resynching the replicated databases and that whole mess seems to have come to an end now. *phew*.

Kudos go out to Karl and Yazz and the rest of OSDNs netop staff for spending several days in the OSDN cage at Exodus (days without a fleece in a wind tunnel like environment!) without sleep, without food, without TV, only with my poor self trying to keep them awake. Good job!