Older blog entries for mx (starting at number 77)

Failures Galore - Our hardware is starting to show its age at work. That, and something fishy is going on.

We've had several system failures in the last few months, some of them obviously related to our ancient systems (6+ now), and some just plainly weird. As there have been so many failures, the net effect is much worse than we'd like.

For example, our backup system has been intermittently failing (a loose wire in a drive caddy), which has thinned our 2-week backup set (a few backups are missing). One of our primary webservers has had some nic instability, which also reduces the hit-rate of the backup system, and we're running low on off-site drives and have missed a few off-sites recently. It's a recipe for disaster ... one that came to a head this week.

So we have a flakey/shallow backup, some failing NICs (still don't know why), and then we start to see some strange drive failures. Two systems this week showed some flakey behaviour, and we scanned the systems and swapped out the suspect hardware. On reboot the systems are dead; the raid-arrays have no mbr or partition table, and the rest of the data on the drives is severly borked.

The logs and clam scans for the systems are clean, and the custom tripwire scripts didn't detect an intrusion, but it's still the likely cause. The only clue that would hint otherwise is that the systems are so old, leaving so many possible points of failure ... except that it happened to two systems in two weeks. I couldn't find enough data on the drives to prove anything either way, and none of our scanners caught anythign obvious. But we're looking for rootkits on the rest of the servers, and we've cycled passwords/locks/keycodes for the entire building.

The kicker, of course, is that we didn't have recent backups for either system, as the backup system has its own problems. We had the primary partitions imaged (dd + netcat rule), but a lot of data has to be built from crumbs around the network. Nothing critical was lost, but it's still pain.

What are we going to change?

1. Failed backups will result in someone making the backup by hand. Ignoring a broken backup will result in failure, eventually.

2. We're moving our main production sites to a managed farm. We write software, we don't manage servers. I've been pulling this IT group out of a hole that it may never get out of. Too few people, too little experience, too little time to fix it all.

3. Buying better hardware does not guarantee success. The previous incarnations of this IT group believed that spending more money meant better uptime. It just isn't true anymore ... these servers are dual-cpu, multi-nic, multi-raid array machines ($10k CDN), and they fail every 1-3 years. And the failures are often hard, despite the raid arrays. I'd kill for cheaper hardware, where I could swap in a new machine at will. Instead we're stuck troubleshooting old, expensive hardware. Replacing drives costs us more than makes sense, as there is an obligation to continue using the high-end SCSI stuff, despite it not making sense anymore.

One reality I'm learning is that legacy is always a problem. The principle I take from that is that decisions need to be as orthogonal as possible, to make changes in the future easier. Smaller, simpler, fewer, cheeper.

bender Lives - My pet blogging tool is shaping up. I've written a lot of requirements, and worked on some design, and am part way through converting my site data (my site is the prototype). I've written most of the components now, at least in basic form, and am testing various pieces of functionality.

The next release will likely only contain a few of the UI-CGIs, so I can get my new site up. I'm really tired of the current backend uses textpattern, which looked good at first ... but doesn't look like it's intended to remain free forever. I used to use blosxom, which was good -- but I found it had too few built-ins (absolutely everything was a plugin). Bender will be a lot like blosxom, but will contain the essentials as part of the core, things like configuration/meta, a text-backend, some auto-markup stuff, and a basic webmin interface (as well as a command-line tools).

And this is one of the reasons why I think diversity in software is good. A great project like blosxom gets people thinking, dreaming of how it could be better. The result is better software, as it's the stuff of our dreams.

zeenix - It was a weird epiphany for me that day, based on a whole bunch of reading that came to a head. It's one of those things that should be obvious, but I've been dense and naive. The "So" was a cheep trick, or my inability with the language. But thanks for the kudos.

So capitolism is a sort of organized greed, where freedom is found in financial independence. When the number of principles in a financial undertaking is limited, and there are real people in control, the system works well. As the number principles grows, and the endevor is an amoral quasi-citizen (a corporation), the system optimises for greed efficiently. Corporations do not value society, except as a means of production and consumption.

Corporations grow by nature, and only have a conscience when it fuels growth -- at least on average. There are exceptions to the norm, as in nature, but the good companies stand little chance against well-optimised money making machines in the long run.

Business is applied greed, though we're generally fine with it as long as we gain from it. As long as the majority gains minimally, the whole system works. Alternatively, as long as the majority is enslaved (even if under marginal freedom), the whole system works. Our world is generally a balance of these two approaches. And we're generally fine with it if we have our computers, big-screen TVs, and other middle-class wealth.

Just like me writing this on my stupid laptop, thinking about how globalisation is killing the little software company I'm at, and how we should really be fucking happy to be eating ... rather than worried that we won't be able to buy our yuppie shit.

prozac : I agree about the basic anti-office approach for documentation. I've been using the text/style/processor approach for several years now, publishing to web, postscript, or pdf using several different tools. None, of course, are easy enough for normal people to use, but it's the right general approach. One of my projects is related to this, though work seems to trump its progress regularly. Good commentary though, thanks.

I started watching The Corporation last night. It's a fairly balanced Canadian documentary about how corporations came to be the odd entity that they are. The history itself is worth it, to see how the current mess of off-shore slavery was caused through the imancipation of local slaves a bit more than a century ago. Well, it wasn't just the 14th amendment, there may have been some optimisation on the greed axis too.

And it's officially summer here: it hit 36C (>100F) last week, and us igloo-bearing canucks aren't quite used to it. Few people here have air-conditioning, as things are generally reasonable around here. Damned heat. I walk to work daily, about 3.5km, so I'm nice and warm after both trips.

My pet project is stalled due to the heat and post double-time burnout. I'm letting my brain rest a bit before I push too hard on the next release. What was once a link logger is becomming more of a Blosxom replacement (somewhere between Blosxom and Wordpress). I want something with a simple backend, a GPL or Artistic License, and some way to deal with very different types of pages (to fit how I want my sites). It's a good diversion from my usual device driver/protocol development.

Things I know today:

I should not feel guilty for taking time to think at work. (The worker-bee ethic is a pile of rat droppings)

perldoc is really cool.

Gnome/Gnu/Linux are a full replacement for Win32/Pain.

Working for other people can be difficult -- especially when those people are working for other people. These days I've been doing a lot of contract work, some of it for a company that does contract work. I enjoy most of what I do, but sometimes clients can be difficult, especially when they're the livelyhood of a large group of co-workers.

We should really turn down one of our current contracts. It's going to be close to impossible to succeed on the contract, as the customer has found a pattern of changing direction (and forcing their contracts to comply). The problem is that we *can't* say no, or a bunch of people don't work. Unfortunately, I think that I'm going to have to say no myself, as I don't believe in working under those conditions, where failure is likely, and being part of playing the bitch. Not my cup of tea.

So it's back to the drawing board, again.

I missed the daily readof the Advo-diaries, which are always worth the time.

Work is interesting: writing requirements for one contract, and developing serial drivers for the other, taking me from half time to double time. Gotta take the work when it comes.

Non-work stuff is fun too. I've been neglecting the weblog a bit, but have a new layout and backend ready to go. Now I just need the time.

Part of the new site back-end is a set of tools that tie javascript bookmarklets and cgis together. You can really do some neat stuff, and it helps to have survived on half-baked tools for many years. I've been stewing on usability points for a long time, and now am finding really simple ways of making web apps easier to use. I'll post more about how that stuff is working soon.

Weblogging freedom - I rebuilt my site last month using Textpattern, a fairly decent set of weblogging scripts. I didn't really take the time to understand its license though, and it seems like it might be drifting toward a split commercial/BSD license at some point. Some of the sources state a BSD license, but the site is really unclear, and there are hints that they might charge for it once it's out of gamma. It doesn't matter though, there are lots of choices.

I took a second look at the Textpattern licensing and general development progress after noticing the release of Wordpress 1.2. I installed it on my laptop (later on my testing site) and was impressed ... the progress has been significant since the last version, and the license fits my personal preference. The development group is active and open, it represents more freedom, and a better tool. It's funny too, as a developer I have another freedom that others may not: I can code my data out of a tool that I no longer want to use. This is one of the reasons I've been teaching programming to a group of friends (Freedom for free beer).

For anyone who hasn't used Wordpress, it has a few really impressive features:

• The editing window is large, and resizes with the browser window. You can also configure the height of it (as a text-area can't be scaled automatically in that direction). This is a killer feature, IMO.
• The templates are real Php files. Most weblog tools have their own simple template language, which is almost always limited. Wordpress exposes a Php API that you can use from fully-functional Php pages.
• The plugin system (new to 1.2) provides multiple hook points, and can be activated/deactivated from the web interface. I've been frustrated by other tools that didn't provde hooks for munging article data.
• The install is trivial, the interface is sensible, and it performs reasonably well.

Forensics - We picked up a small contract at work to find some data for legal proceedings. I'm impressed by the Free tools available for Gnu/Linux, especially Sleuthkit/Autopsy (and both are available from DAG's Yum repositories). I have several drives to analyse, and the UNIX way is well-suited to this sort of analysis. I was able to image the drives over the network, to an array of SCSI drives with a few simple tools (netcat, dd), and from there can analyse the vfat and ntfs filesystems easily. The same activity on Windows require a few more steps, and more licenses. I wasn't able to get around the reboots (for swaping drives), as we didn't have any unused hot-swappable bays (that would have been nice).

Visualization - We've also been investigating visualization toolkits for another potential project. We looked at Coin3d, which is a full implementation of SGI's OpenInventor API (way ahead of its time). Our other Architect has some amazing stories about the history of SGI and OpenInventor, making this job so worth the cut in pay. I'm finding that working with incredible people, and living closer to family means more than money (should be obvious eh).

We're also looking at Java3d, as the client has a general preference for Java. We could wrap something like Coin3d, but we'd prefer to skip the complexity if possible, as usability is generally a function of available time. Java3d performs reasonably too, and our group is quite productive in Java. Java has certainly come a long way in the last few years.

Unixisms - I've also been managing IT at the new job. It was a bit of a stretch for me, or so I thought. It actually ended up quite a lot of fun, and I'm learning things I needed to know.

This company has churned through a few IT managers in the last 10 years, eaching bringing their own direction (Windows, Linux, Windows, Linux), so the IT grunts are obviously disgruntled. One of the first things I've done is to set a long-term direction, and put the decision to change in the hands of the team; this is more of a defacto policy change, but the IT guys believe it, and the CEO prefers it too (it costs a lot to change so often). I've also been working on training these guys, who have a windows background (and some *nix traing courses). They've showed some fear of the existing setup, so they now have a test lab to play with, and to build/test new servers. It's working well.

I've also been training myself on sendmail, ldap, imapd, and other bits. I've used most of these tools for years, but not at the enterprise scale, which is really very different. Spam is much more of a problem too, and the dynamics of the solution change fairly quickly. We were drowning in joe-jobs (bounce spam), which I'd never heard of before (quite a clever/evil approach too). The tools are good, and their actively developed, so at least we have a chance of keeping things running smoothly without drowning in license fees.

Netbore - Slashdot is getting really annoying again, and I think I might have outgrown it. While there is the odd nugget in the comments, most of it is unbalanced and damned predictable (whine, bitch, and complain). On the other side, I've been reading some MS blogs, where there seems to be a lot of a similar imbalance. Somewhere in the middle, there is truth (but where?). Imbalance seems to be how sites get noticed these days.

Speaking of imbalance, Eric Idle has a new song that fingers the FCC. I don't usually talk politics, but man Freedoms are fleeting down south. I fear it will happen in time here too.

I don't think the nearest-book game as silly, it's a curious social experiment:

While the story of Edison's preferences for pencils is understandibly not so oft-repeated as that of Edison and the electric light bulb filament, the history of the led pencil itself does involve quests so memorable. -- Henry Petroski (The Pencil)

I am Canadian