Older blog entries for apenwarr (starting at number 165)

(No, I still haven't finished transcribing my handwritten notes from when I was laptopless in early September. Here's another entry. Note to self: I know it's where the term "copy and paste" came from, but yeesh, handwritten notes are a really inferior medium for actually doing it.)

Delirium 3: Specs and Constraints

So let's say you managed to assemble a team of obsessive, maniacal geniuses who are all "signed up" and ready to get the job done. Great! So, uh... what job, exactly?

There's more than one reason people don't run teams like this too often. The most basic reason is obvious enough: the rules are so strange, undocumented, counterintuitive, and completely opposite from normal (successful!) engineering practice that people just don't try to assemble the "right" conditions very often. But if, after takeoff, the results tended to work out spectacularly... well, people would get over all that other stuff, right? ... Right? ... Actually, yes, I think so.

But life isn't so simple. There's a much more serious problem with the "obsessive genius" technique. The problem is that the results are entirely out of control. You'll almost certainly get something cool - probably more cool stuff than you know what to do with. But you won't know in advance exactly what it'll be. And that, my friends, makes it hard to run a business.

Some companies do it anyway. They start a "research lab" or "skunkworks," put the wheels of obsession in motion, and pray like crazy that something good comes out.

Well, research labs aren't for me. I'm not going to beg "the establishment" for their precious resources, with nothing more to offer than a random splash from my really, really big shiny innovation firehose. When all I have to choose from is total randomness in large quantities or total determinism in disappointing quantities, I'm in a classic compromise situation. The non-compromise condition is massive, sustainable, directed ingenious output. And my theory is that it's a lot easier to direct output that's already massive, sustainable, and ingenious than it is to take small deterministic output and make it bigger and more ingenious. It's easier to learn to aim a big firehose than to put out a big fire with a really accurate squirt gun. Mind you, no matter how well you aim, some tangential stuff is still going to get a little wet. But it'll dry off eventually.

Which all finally brings us around to my point. It is possible to direct the output of a team of obsessives - sort of. You just have to be careful about what exactly you direct.

Determinists love specs - clear descriptions of the ins and outs, which must be captured exactly. Then you leave your geniuses to do any design they want, as long as it implements the spec precisely.

I tried that method. I really did. I sliced and diced specs vs. designs in every way I could think of. UI design isn't specification, it's design!, I declared once.

But with each and every attempt at this technique, I failed in one of two ways. Either the spec was too dumb and restrictive - or the result was too crazy and random. Either the product was overspecified or it was underspecified. And now I realize that it always will be. Trying to "right-size" your spec is just a tradeoff - a compromise. The right answer doesn't even lie on that axis.

But even though I've never really managed to separate the spec from the design (for management purposes at least - it works for post-documentation, or what we sometimes call a "retrospec"), I have successfully produced non-random, ingenious results - and I've worked with other teams, like NVS and GourD, that produced them. What's the secret?

The secret is "end-to-end." The problem with our spec-design separation is actually very simple: it draws a line between the real, deterministic world - the spec - and the world of genius - the design. But that's nonsense. How can your product be ingenious if its workings were defined by someone deterministic? How can it be useful if the workings are defined by someone crazy? The spec, too, must be written by someone in genius mode. Those were the real successes I've seen - where the spec, the design, and the product were all done in genius mode, "proper processes" be darned. And I've been fooled like everyone else: GourD had a Requirements doc, then a Spec, then a Design, then an implementation. Deterministic, right? The model of good engineering!

Yeah, sure. The author of all those parts was the same person, and while they were written down in sequence - which is a very fair, respectable, methodical approach applicable to geniuses as well as anyone else - I know for sure that the documents were "non-causal." The contents of the design changed the requirements long before anyone wrote down the design. I was there. You can't fool me. Just try to tell me the requirements would have been exactly the same if Mediawiki hadn't existed as a model. It was all a good exercise in in thinking clearly. But it wasn't a "deterministic engineering process." It was a sham. It was an ingenious bunch of work that couldn't help but come out because all the right pieces, including the right people with the right motivations, were in place.

So the question is, then, how did it get to be the right product? Where did the genius stop and the real world begin?

The answer to that question is the key to everything.

18 Oct 2006 (updated 18 Oct 2006 at 13:04 UTC) »
ISO-9001

My apartment in Montreal is near an ISO-9001 certified parking lot. I didn't even know that was possible. Or necessary. Or even something that would have occurred to anybody in a million years. But now... I'm just not sure anymore. Is it really safe to deal with those other, unprofessional, "fly by the seat of your pants" parking people? I mean, what if I'm parked there and they just go out of business, or the pavement falls off, or worse? Will they have all the requisite forms?

Writing stuff down

    Before you read this, please understand that I never wanted to write this document, being grossly under-qualified, but I always wanted to read it, and this was the only way.
    -- Paul "Rusty" Russell, Kernel Hacker

apm wonders if an expressed idea is more valuable than an unexpressed one. You can spin in circles for hours wondering what "valuable" means and what's the meaning of life and whether there's a God, but let's think of it this way instead: why exactly is it so hard to write things down?

Well, it is.

I think the reason I write stuff down, at least, is because writing it down forces me to make it better. It's so easy, once something is written down, to see that it completely doesn't make sense and you'd better start over. (Perhaps this is why salespeople don't write stuff down.) So even if you don't care whether writing your story will improve the lives of anyone else, maybe you can justify all that work by thinking that maybe you'll be improved in the telling.

Speaking of which, several people have asked me why exactly I write this journal, since it's pretty obvious that I mostly blather on about nothing important, there's no particular unifying theme, most people don't understand what the heck I'm talking about most of the time, I'm certainly never going to get rich doing it, and so on. I've made up several answers in the past. But here's another possibility: maybe writing it down helps me to clarify things in my own mind. And maybe pretending to write for a "real" audience keeps my standards a bit higher, so that I'm forced to clarify things even more. But in the end, maybe I just write because I enjoy reading what I write. And yes, I always laugh at my own jokes. Always. Even if it's just out of sympathy.

Err, my condolences if you actually enjoy reading this crap. Feel free to continue. But you'll be sorry:

High-level intellectual discussion

Okay, I just have to get this out of my system. The other day I was cleaning up a spill - as a purely scientific experiment done in the most hygienic way imaginable (given the circumstances) - with my tongue. And I noticed, to my amazement, just how effective it was. Now, we all know that smearing around a spill with your finger just smears it around and makes it worse. And paper towels sometimes help, particularly the extra-absorbant kind.

But as far as I know, my tongue isn't extra-absorbant. It's not like I lick stuff up, then pull it back in and squeeze it out so I can lick up some more. You just lick stuff up, and it goes in, and then you can lick more stuff up. Why can't my fingers do that? Isn't it amazing? Wow.

Today, a collection of random and mostly unrelated thoughts. Oh, how I do prattle on.

Selfishness

Today, while I was considering the sad state of the Quebec construction industry, it occurred to me that unionization, and socialism in general (ie. Canada and especially Quebec) is not as straightforward as it seems.

The idealized American system is a rather liberal everyone-for-himself economic free-for-all, in which "the system" is intended to make it so that if everyone acts "selfishly," it'll all work out for the public good. In Canada, we don't trust that theory so much, so the government gets involved more frequently, and in general people take a somewhat more "moralistic" outlook even in business decisions.

Both systems have their good and bad sides, but the resulting style is very different, which I will explain in terms of my experience at Subway (the food chain) in Canada and in Seattle. In Canada, you go to Subway, and they make you a sandwich according to your specifications. In Seattle, the person who made my sandwich was totally insane and made a perfect sandwich according to my specifications faster than I've ever seen anyone make a sandwich in my life. You could barely even see his hands, he was so fast. I'm not making this up.

Now, I don't go to the U.S. very often, so I don't know if this is typical or not. But imagine it is: the idea is that, because everyone is being selfish, they become hypercompetitive: not only do store managers have to outdo the other restaurant chains, but maybe they try to outdo other Subway outlets as well. If you can get your sandwich twice as fast here as at the place two blocks away, maybe you'll choose their store instead of the other one. So they focus on customer service. Figuring it out was all a bit circular, but in the end it's not indirect at all: being selfish equates to being as good to the customer as possible. That's selfish?

In Canada, people are more laid back and they don't worry about competition as much. The result is nobody particularly cares how fast they serve me my sandwich. In fact, if the Sandwich Ace from Seattle showed up and tried to work here, people would probably look at him funny: he makes the other employees look bad. The unselfish thing for him to do would be to slow down and not rock the boat, resulting in worse customer service overall. That's unselfish?

Which system is more selfish then, really? Is it the one we thought it was?

BarCamp, Advogato, and Self-selection

I went to BarCampWaterloo a couple of weeks ago and it was quite entertaining: a small, interesting group of people, just like I like.

The problem is, that doesn't make any sense.

In general, communities that are good and interesting and widely applicable tend to start off well, and then explode into hypergrowth until they're no longer manageable and most of the people there are just annoying and all the fun is gone. Take the Linux kernel developers (it was possible to follow their mailing list, once, even if it wasn't your full time job), or Debian, or actually even This Whole Internet Thing. Impromptu communities (I wanted to say "online communities", but BarCamp isn't, exactly) tend to be in one of two states: growing or dying, and growing typically leads eventually to hypergrowth. The best you generally hope for is either slow explosion or slow death, so that it can be fun while it lasts.

The typical way to slow your community's growth sufficiently is to limit your topic area, so fewer people are interested. Forget the linux-kernel or debian-devel mailing lists; try linux-fsdevel or debian-apache instead.

What's weird about this is that I have two as-yet-unexplained counterexamples: Advogato, which claims to be a web site connecting "free software developers" (how restrictive is that?) and BarCamp, which generally claims to be "about Web 2.0" (nobody even knows what that is!) but in which anybody can show up and present about anything even remotely relevant.

Why, then, did I find that the majority of stuff produced by both communities was interesting to me? Certainly I'm weird, because the majority of people wouldn't have found them interesting at all. But that's the point. The communities created are self selecting and quite restrictive, but it's not selected by topic area. It's something else. Perhaps BarCamp simply selects for people who don't think deliberately failing to plan your conference in advance is a stupid idea. And that's a pretty small group of people - and they're pretty compatible with each other.

(This concept of people connecting better based more on style than content relates to my earlier comments on literacy.)

NaNoWriMo

    If my doctor told me I had only six minutes to live, I wouldn't brood. I'd type a little faster.
    -- Isaac Asimov

This is a pre-announcement of my intention to join this year in National Novel Writing Month, in which each foolish participant attempts to write a crappy 50,000 word novel in 30 days. Normally I wouldn't bore you with my plans for such things, but apparently one of the keys to success is setting yourself up to get teased a lot if you slack off (ie. mutual motivation). In my case, this is actually by far the most likely situation since I don't actually have any free time to write a novel in.

But I mean... impossible deadlines and absolutely no quality standards? Hello, count me in! Anybody wanna race?

Making banking fun, round 1

It turns out that other than so-called "business logic," there isn't much to business software. It's all the same stuff: a database connection, forms, fields, buttons, some daily/weekly/monthly batch operations, and the dreaded Reports.

Over and over again.

What's interesting - and interesting is the first step on our way to fun - is the process of converting fluffy human requirements into usable software. Many programmers doing business software aren't quite up to the level of programmers doing other kinds of software, but also interestingly, you don't get a lot of cross pollination between groups. Things just start getting done a particular way, and they keep getting done that way, and nobody really thinks twice until and you get these funny "programmer cliques" that believe totally different things and don't really talk to each other. You know, the Oracle types don't have much respect for the MySQL types, and vice versa, and they'll never resolve their differences because, well, they don't really care enough to bother.

Now I've gone and switched cliques - from operating systems to financial software, of all things - and what I learned is that both groups have plenty they can learn from the others.

One thing I learned is that there's a huge, tangible, visible difference between a company that understands business (and software is secondary) and a company that understands software (and business is secondary). Software companies do an awful lot of silly things that just don't make basic business sense. I don't mean any particular company here - they're almost all pretty clueless. Me too, for now. But it can be tough not to be clueless when you don't even know what clueless is, and worse, you've isolated yourself from the people who do know.

Now turn it around. People with business sense tend to be profitable, which is a good start. But they don't know that much about advanced software development, and silly things happen and time gets wasted, and they don't know that those things are totally avoidable. The smart ones, though, are willing to learn. Great software is an amplifier for a great business model, and the two put together is how you can get a really great company. This is straight out of Good to Great, of course. Go figure.

Bankers are literate!

All this is tangentially related to another amazing discovery I made recently, which is that banking-type people are actually much more compatible with programmers than I thought. You would normally not expect this, since banking-type people (kind of an extreme form of accounting-type people) are in many surface ways different from programmers: less energetic, less liberal, less creative, and often less smelly.

But one thing we have in common - and this is a critical thing when you're trying to work together - is that both bankers and programmers are literate.

"Literate" is the word I use to describe the sort of person who processes text efficiently. I hold literacy to a higher standard than, say, the people that tell me 97% of Canadians are literate. Maybe so, but I'm sorry, hardware engineers can't spell. That goes for assembly language programmers too, and I'm grossly overgeneralizing, but my generalization goes almost exactly opposite for good high-level language programmers.

Maybe you noticed the trend too, and thought it was just a coincidence. But it's not.

By my extended definition, many "illiterate" people can read and write... but they'd rather not. It's a last resort. Illiterate people would rather have meetings and "facetime" and phone calls. They also prefer diagrams over long-winded explanations. There's nothing really wrong with "illiterate" people, I suppose - let's call them "visually oriented" or "spacially oriented" people instead, and it suddenly sounds better.

Illiterate people are hard for literate people to get along with. They don't understand why programmers like to have long email flamewars to resolve their problems. They don't see the point of developing documents collaboratively in a wiki. They prefer shorter and more abstract, not longer and more detailed. They can't skim. They can't search. They don't understand irc. Being the vast majority of the population, they agree in these respects with almost everyone, and so they assume the literates are just doing things wrong and our methods are somehow horribly inefficient: "Geez, it must have taken you all morning to type that up! Why didn't you just have a meeting?"

...Illiterates are frequently slow typists.

But bankers are literate. Of course they are - they have to be. Their whole profession is about spending long hours avoiding people, working with long, complicated legal documents, writing things down so they don't get forgotten or misquoted, and checking long columns of numbers, making sure nothing gets lost, because one mistake messes up the whole thing. Hmm. Avoiding people, complicated documents, concentrating, columns of confusing numbers, one tiny mistake screws everything up. Does that sound like anyone you know?

Programmers are totally capable of dealing with this sort of people, but they let differences of external style - mostly the conservatism, I suppose - stand in their way. But little things like that are less important to collaboration than the fundamental communication channel. If you can talk to people effectively, you can deal with them. Or so I claim.

Hmm, have you ever seen an accountant using a wiki?

Smallness, responsiveness, feedback

One thing I've discovered about myself over the years is that I prefer driving small, responsive cars with clear feedback over big, fancy cars that try to isolate you from the real world.

I like small cars because they fit in places where big cars won't. I try not to have a lot of luggage, so that doesn't bother me. But going where other people can't go is valuable in itself: supply and demand. Okay, yes, people with Land Rovers can also go places where my Pontiac Pursuit with Pulsating Performance can't, but statistically speaking, more people want to pay me to park downtown than to go driving through swamps. You know. Statistically speaking.

I like responsive cars because I'm short-term indecisive and my sense of direction is complete crap. I'm always impressed by people who actually know where they're going and how to get there, but I'm not one of them. So sometimes I need to change my mind and turn a corner I wasn't planning to turn. And sometimes I need to make a U-turn. And sometimes I need to stop or start quickly because the person in front of me or behind me was even less decisive than I am. Responsiveness is good when you need to deal with surprises, and perhaps out of necessity, my specialty is dealing with surprises.

Feedback is a more interesting question that a lot of people don't even think about: in fact, they almost think of feedback as a negative. The most expensive cars are the ones that make you feel like the outside world doesn't impact your driving. That is, they're soundproofed so you can't hear outside; the steering wheel never resists your turning; they have cruise control so you set-and-forget the speed; they have anti-lock brakes and fancy auto-traction control to prevent skidding when you stop or turn. The idea here is that you should be able to concentrate on setting the speed and course - the strategy - while the "system" handles the rest of the details for you.

Well, I don't trust the system. Maybe it's because I'm an programmer and I know better than to trust programs, or maybe I'm just paranoid. But luxury cars like that drive me crazy. When I drive one downtown, I often open the window a crack, defeating the expensive sound isolation, so I can hear the street noise, like someone yelling or tires screeching. I like steering feedback, because they let you feel whether you're turning too fast, mostly eliminating the need for traction control on turns, and anti-lock brakes actually make braking distance worse on dirt roads. There's no substitute for just driving carefully, and isolating yourself from the real world discourages that. If all you have to do is set the speed and course, what's to stop you from setting a just slightly faster speed and a just slightly bumpier course?

In a company, the equivalent is the executive who isolates himself in the corner office and reads reports all day rather than visiting the front lines. I can't stand to do that - and if you're an executive, it's not even your fault, people try to set you up that way. People will see it as their job to make problems go away between the front lines and you; after all, if they can't deal with these little problems, why did you hire them in the first place?

That's absolutely true, of course. But it's also true that if you as an executive never see the problems, you'll set stupid strategies that arbitrarily increase the problems for no reason. You'll take the bumpy dirt road instead of the superhighway, but you'll try to drive 120 km/h anyway, because what's the difference? The bumpy road is a more direct route than the superhighway, and the speed is just a dial that always seems to work fine, and it all looks the same from in here.

Company Size

Okay, so how does this relate to my preferred company style, then? Well, I prefer small and responsive companies and teams to big and stable ones. Small companies can afford to do projects that big companies can't justify. And the way you run your company - my way is controlled instability - can make your company more responsive than others, so when you make the inevitable mistakes, you can recover.

I'm interested in ways of making stable companies more responsive without making them unstable. I'm not sure you can easily make big companies small, but it's easy to start a new small company anyway. As for isolation vs. feedback, well, just try to isolate yourself from a customer when your whole team is only five people. It forces you to do a great job, or you'll die of exhaustion.

Code Size

And while we're here, I can tie all this directly back to programming, as usual: small, concise code vs. big, unwieldy code has exactly the same characteristics as cars and companies.

Big, ultrastable, isolationist companies, like typical big banks, program in Java and use UML and huge specs and "Software Engineering" and five-year plans and code monkeys. I'm not a fan of that whole development style, but I'll tell you one thing for sure: it works just fine for them. Check out one or two of their balance sheets and tell me I'm wrong.

But big code is hard to change, so you have to plan it in advance. The isolationist executives at giant, stable companies get very good at this, and the stability of such a company is totally compatible with this, which is why it works. But in a small, responsive company, you need small, responsive code. And sufficiently smart people can do the same code in a smaller, faster, more responsive way, allowing them - and thus, the program itself - to respond more quickly to feedback.

That's my kind of team, my kind of company, and my kind of code. I got the car. Now I'll have to work on the rest :)

6 Oct 2006 (updated 6 Oct 2006 at 13:20 UTC) »
Conservation of Life Force

dwiseman mentioned that some people are polite and civilized, but because of the way they act, may be unknowingly "extracting positive life force from those who will let them."

I think that's an interesting way to look at things. Many years ago when I was first starting NITI, someone I know referred to such people as "hangers-on" and warned me to avoid them - the people who are friendly and supportive, but don't really add anything to what you're doing. They just want to be pulled along with your momentum.

There are lots of people like that. The question is really: do you really have a limited amount of positive energy to share? Does sharing it with someone really mean you have less of it left over? If someone can be happier because they can bask in the warm glow left by other people around them, is that a problem?

I decided it's not a problem for me. People who get in my way - they're a problem. People who just play along and enjoy the show - at worst, they can be safely ignored.

But if you let them get to you - if you start to feel like these harmless people are "stealing" energy from you - that's when you start to lose your energy. But it's not them, it's the way you think about them, and thinking that way is what demotivates you. The good news is the way you think is in your control, not theirs.

I think of it this way instead: how can making someone else's life better possibly be a bad thing?

(Must... resist... urge... to compare... life energy... to MP3s...)

RSS Housekeeping

To appease the whiners I have updated my copy of NITLog to produce valid RSS 2.0, including If-Modified-Since and a GUID field. That was interesting: I've never implemented If-Modified-Since in a script before. The auto-title-sucking algorithm has also changed again to rip all the subtitles, not just the first one. I hope that doesn't confuse your RSS reader too much.

Cat is the anti-gap

My new car has an mp3 player that is very nearly not stupid, but fails to even remotely attempt any sort of gapless playback. This is fine for most things, but really, really annoying for CDs that have a continuous mix format.

Well, after reading various long-winded tirades about how incredibly difficult it is to make mp3 players gapless, hardware limitations, blah blah, I solved my problem by simply concatenating most of my Tiesto collection together (one album per file, of course) and writing a new CD. Of course, when I ripped the CDs in question in the first place, I used LAME's nogap option, or it wouldn't have been quite right.

The bonus is that I can now seek from one album to another rather rapidly. It's a good thing I have absolutely no desire to seek to specific tracks, since that totally defeats the point anyway.

So, advanced audio enhancement technology or ancient 1970's Unix? You decide!

BarCampWaterloo, this Friday

Okay, it's official. I will be attending the official trial edition of BarCampWaterloo this Friday, September 29, from 2pm to 7pm. If I get a timeslot, I guess I'll probably do a presentation about either GourD or Schedulator.

The conference seems a little funny, but it should be interesting. Note that "bar" refers to "foo", not alcohol.

26 Sep 2006 (updated 26 Sep 2006 at 00:02 UTC) »
Delirium 2: Imbalance

Imagine you're in a speedboat testing an experimental new engine with 5x the propulsive power of any engine that came before. Moreover, the lead engineer for the new engine, who is also an expert technician, is there in the boat right next to you, excited, tuning the engine as it runs. "Egads!" he exclaims. "I just found a way to double the power one more time! This thing is really going to go now!"

Impressed, and listening to the scream of his supercharged nuclear turbine as it spins at full speed, you yell out the only obvious question: "So why aren't we moving?"

The engineer, an expert at engines but not much at boating, looks around in dismay. "I guess... it's still not good enough." He turns back to his engine with a determined look on his face. "We just need more power!"

Hours pass. The engine roars louder and louder. The churn beind your still-stationary boat is getting out of control. The engineer, apparently rather motivated, has doubled, tripled, quintupled the power. By now a crowd has gathered around the dock to watch. They all agree that this engine is the most powerful one they've ever seen at this size and weight - by a long shot. Your friend the engineer has discovered at least two new laws of thermodynamics right there on the spot, and people are sneaking up behind him with honourary PhD's. But still - the boat's not moving.

Suddenly a thought occurs to you. "Hey, are we still tied to the dock?" you yell. It's hard to hear you over the wailing engine, but he stops, looks at you, and then turns in the direction you're looking, to see the lines still holding the boat in place. Then he slowly sits back, stunned and silent. He throttles back the engine, and the world gets a bit quieter. You look at your audience, still there watching from the dock. "Why didn't someone say something?" you ask.

"We thought you knew," someone shrugs, not meeting your eyes.

Epilogue

By now the sun is already setting in the evening sky, but it's not too late. You unlash the boat from the dock. "Let's do it," you say quietly, and take the steering wheel. The two of you begin to edge away from land at last.

But before the engine his even 10% throttle, it's already threatening to flip the boat. "It's too fast!" you exclaim. "I can't steer! Slow down!"

Your friend looks back at you, then looks away and begins to throttle back to a normal speed. "What's the point then?" he asks.

"I think we'll need a better boat, and we'll have to find a better driver. An engine this good must be useful... this boat just can't handle it. But the engine is way too good to waste!"

"You don't understand," he replies. "This engine was tuned for this boat. The concepts can all be generalized, but it's going to be a huge amount of work... I guess this particular engine is never going anywhere."

The two of you make your way back to shore. The engineer, looking exhausted and frustrated, gets out and starts to walk away. "That's it. I'm too tired. I'm done here."

"That's it?" you ask. "What about the engine?"

"You can keep it," he says. "I left my notes. If you can make it run half as well as it did today, it'll probably make you rich."

Just don't tie it down.

Explanation

It's such an odd problem - if all you have is a hammer, everything looks like a nail. If all you have is duct tape, everything looks like a duct. If all you have is an engineering team, every problem looks like an engineering problem. After all, it... has to be. How can we deal with it otherwise?

You know, if we had kept that engineer working long and hard enough, eventually he might have fixed his problem using pure power. Maybe he would have snapped the lines, or torn the dock right off the shore. But the results would still have been out of control, out of proportion, and even more disastrous and disappointing in the end.

Even though this is a sad story though, the point I'm trying to bring up is actually a happy one.

A great "engine" got built, based on new "technology" and "laws of thermodynamics" that we discovered out of necessity. The bad news is that the engine was unnecessary - a merely passable one would have worked just as well, if what you want is a really fast boat. But the innovations themselves were still useful and can be reapplied elsewhere, on a new boat equipped to withstand that kind of propulsion. Someone just has to build the boat.

Those innovations would never have been made at all if it weren't for the original grotesque error in judgement. The engineer will keep those innovations - along with a determination never to make that same mistake - with him, ready to be applied when needed, for the rest of his life.

And if you were in the boat along with him, or even just in the crowd watching, maybe you learned something too.

Delirium 1: The Genius of Obsession

My friends dmg and apm recently lent me a book called The Soul of a New Machine, by Tracy Kidder. It's a work of long form, non-fictional journalism that describes the process of designing a new minicomputer at Data General around 1980. It's much more about the people and their social issues than about the computer itself.

What an interesting book to recommend to me right now. I certainly recognize the story: team of 30 or so crazy people in larger company isolate themselves, gel as a team, build impossible product on crazy timeline, live through pressure, team reorgs, smart and stupid managers, and have way too little real life experience. Product gets completed (a bit late, of course), team leader moves on, and team essentially disintegrates and carries its experience elsewhere, person by person, while parent company lives on, continuing to sell the resulting product and ones based on it. I guess we can be proud - we held together the culture longer than they did.

But the things that really struck me about the story weren't so much the similarities as the differences. They did a few things "right" in that story - for some definition of "right" - that I never quite managed. Their team lead, Tom West, managed to bring out of everyone on his team something that I've only ever managed to bring out in myself personally - total obsession with the end goal.

And obsession, as I learned for sure in the last few months of not being obsessed, is where it's at. Obsession is the only way to really get into The Zone. It gives you energy you didn't know you had. It turns you from a normal mortal into something more. I've done it for myself a few times. I used to wonder if you could even do it for other people, because I never had much luck. But after reading this book, I have a few clues. Not surprisingly, they echo the environment I create naturally for myself - but I have to change my style if I'm going to create the same conditions for others.

Mutual Motivation. You can't obsess alone for long. The key thing isn't so much having help as being needed. You have to not let people down. That fear is what makes stopping harder than going on.

"Signing up." It has to be voluntary. You have to know what you're getting into on day 1, and decide it's worth it. You have to decide it out loud, and promise it to others. You have to be visibly, publicly in the game, so any failure will be an obvious one.

No way out. There can be no escape clause. If you fail, we're all dead. Nobody will save you. If you're late, we're all late with you. There's no superhero who will do it better and bail you out. You're going to have to grow up and be the superhero. If you really have to do it, maybe then you really can do it.

Too stupid to know it's impossible. It has to be harder than you thought it would be, not easier, so the pressure builds instead of dissipates near the end, when it's easiest to lose interest. And so experience kills - you'll never be able to obsess the same way twice. That's why younger people, especially students, are better at it than oldsters. (As someone on the verge of being an oldster myself - in the book, 30 years is the magic age - I already have to find crazier and crazier projects each time, always outside my existing experience, or I simply can't obsess enough.)

Too blind to see it doesn't really matter. In real life, almost nothing you do makes a serious, long-term difference. Realizing this will kill your fun. You need blinders - and if you're lucky, a really good reality distortion field. Again, older people have trouble here, because once you've been disillusioned by dedicating a year to an ultimately pointless project, it's hard to fall for that again. Except that falling for it is fun.

Too restricted to take the easy way out. Smart, sensible ways of solving problems will knock you out of The Zone. Sensible schedules, flexible and forgiving bosses, 8-hour work days, good nutrition, the newest equipment, a social life (outside work), even using low-cost direct mail services instead of composing your own poetry and stuffing your own envelopes - all these things make it too easy. That means less pressure. Less pressure means less obsession. There's a good reason why entrepreneurial companies are responsible for most of the wildest innovation in the world: because they're so helpless, disorganized, and resource-constrained that nothing you do there isn't a challenge. And that makes your brain work harder, nonstop, without any chance to cool down and drop back from hyperspace into normal space.

When I've done the best work in my life, those are the rules I followed. I turned my load factor down to levels I knew perfectly well were too low to be safe. I set things up so I had no escape and nobody could possibly bail me out. I bet my reputation that I would be done on time. And because it's my special skill, I always, always, failed to realize how impossible and unimportant the job was - at least until afterwards.

And that, let me tell you right now, is the magic combination that makes life worth living. From now on, I'm going to be a very different kind of manager. You should be glad you don't work for me anymore!

156 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!