Older blog entries for apenwarr (starting at number 6)

14 Apr 2003 (updated 14 Apr 2003 at 03:21 UTC) »

Woo hoo, my UniConf and Sharvil's ExchangeIT talks got accepted for Guadec. In half-hour timeslots instead of hour ones, which is unexpected but not really a problem. (Were you supposed to request something, or something?)

Now all I need is a passport, a plane ticket, and a place to stay. Eek!

Amazing Discovery

Today I learned that according to Statistics Canada, there are more married women in Canada than married men. (Statscan is truly awesome, so I suppose there's some explanation for this.)

Document Correlation

This weekend I wrote a page correlator using ideas I blatantly stole (badly) from Paul Graham's essay on spam detection using Bayesian Filtering. His math makes way more sense than my cheesier version, but it mine just a hack so that's okay for now.

Quick summary of the theory: the interesting aspects of a document are characterized by the locally most common words among the set of globally least common words. That is, if I say the word splahooie a lot, but nobody else does, that makes "splahooie" an interesting characteristic of my document. But if everyone else says splahooie, *or* I don't say splahooie very much, it's not a keyword.

So anyway, that works pretty well with some refinement. But using this technique and a cheesy "keyword correlation" algorithm in perl+mysql, and using our internal company wiki (900+ pages) as a data source, I made it so you can get a list of "interesting pages" related to your current page.

The results were... interesting. The algorithm, though simple, is surprisingly good. What it does, though, is a bit weird, because of the way we define "interesting." We only care about globally uncommon stuff. The result is a system that tells you not exactly, "What's related to this page?" but rather, "What's unusual about this page?" If I ask about pphaneuf, it mentions XPLC but not Net Integrator, even though I (as an evil manager) make sure he works more on the latter than the former. But other people do too, so XPLC is more interesting as far as the algorithm is concerned. It's kind of like the anti-google.

Hmm, an abnormality detector. I bet I could sell this to school boards in Kansas.

Cascading Failures

Well, I promised not to discuss my life in here, but since I'm about to use it as an example of generalized system failure modes, I figure it's okay.

The goal: from Waterloo on Thursday morning, travel to Montreal by Friday evening at 8 pm. By car, this is a 7 hour drive. No problem, right?

Well, we were going to rent a car and drive on Thursday afternoon, but it starting snowing/raining/sleeting so we decided not to drive after all; instead, let's take the train, which is safer in bad weather. Since the night train leaves you kind of tired, we decided to take the Friday morning (9:30 am) train from Toronto.

Friday morning, the weather still sucked, but that's okay. We called a taxi at 6:20 am to take us to the bus station in Waterloo. At 7 am, it finally arrived - delayed by bad weather, of course. So we missed the 7am bus to Toronto. No problem, there's an 8 am bus that should still make it to Toronto in time. Unfortunately, the 8 am bus showed up at 8:30 (bad weather), departed shortly afterwards, and got to Toronto at 10:00 (extra late - bad weather). No problem, though; we rescheduled our train tickets from the 9:30 to the 11:30 train. We changed the reservation by cell phone from the bus, luckily, because by the time we arrived all the trains for the day were fully booked. Turns out all the airports were closed (bad weather) and the people taking flights had all switched to the train.

As we were picking up the tickets, they made an announcement that the 11:30 train would be leaving at 12:30 instead - bad weather. No problem: the 11:30 is supposed to get to Montreal at 4:45, so an hour later is 5:45, and even with additional weather delays I should *certainly* be in Montreal by 8 pm. So, we have some time, let's go for lunch.

At 12:20, we came back and found out that the train had left at 12:07, having been re-re-scheduled while we were gone. In fact, they had made the new announcement before we left the station, but because of a ridiculously loud random music performance (something about the Juno awards) in the middle of the station at the time, all the public announcements were inaudible.

Feeling guilty, they asked us to wait while they figured out what they'd do to get us to Montreal. The result: at 1pm or so, we found out that they could squeeze us on the 3:30 train (arrives around 9:30; useless) or a special 2:30 shuttle bus (could arrive at 8:30 in *good* weather; useless). So Via Rail wasn't going to be able to help.

Last chance: rent a car after all (there's a rental place at the train station) and drive it to Montreal. That takes at least 6 hours in good weather. By 1:30 we had almost finished filling out the rental forms, meaning that we *could* be in Montreal by 7:30 on a good day. Sadly, it wasn't a good day. (Interestingly, if we had known at 11:30 that we would miss the train, the rental would have saved us.)

I mentioned above that the airports were closed too (bad weather).

The Moral of the Story

Despite a metric tonne of backup plans (an extra day; an extra bus; an extra train; backup train should still arrive early; could rent a car if the train was cancelled) and slippage, we *still* didn't get to Montreal on time.

In management, we call this "slippage." In clustering, we call this "cascading failures."

The lesson to learn here is that if you're going to add redundancy (like the extra buses, trains, time, etc) you'd best make sure that the same root cause can't screw up *all* of your backup plans at the same time. That means don't put a five-station Oracle database cluster on the same power circuit, don't write software that shuts down and expects the cluster to take over if it gets confused (because what if *all* the nodes get confused by the same thing?), and don't plug all your backup servers into the same Internet connection. For that matter, don't store them all in the same nuclear bunker in the Swiss Alps. If exactly the wrong thing happens, you'll be in trouble.

31 Mar 2003 (updated 13 Feb 2008 at 17:31 UTC) »
Current Status

Offically Not Dead.

Branch Constraint Theory

Oh yeah, baby, I just invented a series of simple mathematical formulas that tells you why your project is taking so long to release, has so many bugs, and seems to show no signs of getting better. It might explain the weird many-way-branching of the Linux kernel and the annoying ever-slowing Debian release schedule, as well as (more importantly) some of the phenomena we've seen at work.

It's 28 pages of pure fun. I'll see about publicly releasing it if I can trim out the NITI-internal bits.

The most brilliant insight, which seems obvious once you read it but is quite non-obvious until then, is:

If you have two branches, a stable and an unstable branch, and you spend more time in code freeze than in feature addition, then each release will take longer than the last. That's because you start stabilizing the current "unstable" branch no sooner than you release the previous "stable" branch, so every feature addition phase is longer than the previous bug fixing phase. And if, within a release, your bug fixing phase is longer than your feature addition phase, well... it flies out of control, and you're doomed.

Doomed, that is, until you create MORE SIMULTANEOUS BRANCHES! And then you're differently doomed! Woo hoo!

Corrollary: simply adding more testing time at the end of a release cycle will not save you. And I'm now certain of it.

Oh, another interesting insight is that if you add features more slowly you can make more frequent releases. It's a fundamental law of software engineering and I proved it more-or-less mathematically. Go figure.

Update 2008/02/13: The link the Advogato article no longer works. You can find the Branch Constraint Theory PDF here instead.

22 Mar 2003 (updated 22 Mar 2003 at 20:27 UTC) »

Okay, so seeing as it's All My Fault, I suppose I should say something interesting.


Right, so Doc Searls was visiting our office (NITI-Toronto) last week and he said some interesting stuff. A lot of this is straight out of the ClueTrain, but since it was personalized for us, it seemed to hit home that much harder. As time goes on we'll be fiddling with stuff and downplaying the brochureware on our main web site, since (as he correctly points out), nobody cares about or even believes brochureware. Hear hear.

But what nobody mentioned is that our OpenSource site is also brochureware, albeit on less expensive paper. My plan is threefold:

a) Wiki-ize the OpenSource site so people internally will actually keep it up to date.

b) Finally get public anonymous CVS working, so people can find out about and download and maybe contribute to all the new free stuff: WvIPsec (AKA "Thank God it's not FreeS/WAN"), WvStreams, UniConf, ExchangeIT, SSoya, WvTFTP, and a bunch of things I probably already forgot. Note to self: consider making advogato projects for all these so I can add myself to them. Hmm.

c) Start blogging. Note: I hate the word "blog" and all its derivatives, but unfortunately that's all we've got. I'll call this my advogato diary nonetheless, and if you're lucky this is the last instance of the word "blog" you will hear from my virtual lips. Also, while some people claim that the interesting thing about a bl- err, diary, is that you can read all about people's personal lives, I basically have no interest in yours and expect you to have no interest in mine. So I'll stick to the technical stuff wherever convenient.

By the way, although it's All My Fault, choosing advogato as the target was All pphaneuf's fault.


Down with it!

(You will hear more from me on this topic.)

Trust Certifications

Since a bunch of NITI people have joined advogato, I felt as if I should certify them.

I think the whole web-of-trust thing (pgp, advogato - it's all the same) is amazingly cool, although nobody has ever taken proper advantage of it. Come on, let me define my own trust roots and then apply the trust calculation to arbitrary things - now we're getting somewhere.

Now, for advogato in particular, I feel rather silly certifying someone like slajoie, who is, franky, a genius with real-life experience, at Apprentice level just because most of his work has been on non-free software. Someone with years of coding experience who's not a big name in free software is just plain not an apprentice. We need a different word. But I'll follow the rules and certify him that way anyway, because luckily it doesn't matter.

Also upgraded my own self-certification to Master. "Modesty will get you nowhere," that's my new motto. Hence the bl- err, diary.


No comment.

14 Mar 2003 (updated 15 Mar 2003 at 00:07 UTC) »

Testing XMLRPC interface. Please ignore.

Ignore this also.

Added another diary entry.

God, my life is exciting.

10 Mar 2000 (updated 15 Nov 2000 at 05:01 UTC) »

Added a diary entry.

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!