Older blog entries for apenwarr (starting at number 140)

Brooks' Singularity

In The Mythical Man-Month, Fred Brooks explains that the amount of communication necessary for a project time is n(n-1)/2, where n is the number of people on the team. Eventually, the communications effort begins to overshadow the actual productive effort. This is the primary factor limiting team sizes, which is why people start breaking into multiple teams, hierarchies, etc beyond a certain size - hierarchies may not be all that efficient, but at least it reduces the need for huge amounts of communication.

Now, this efficiency decrease is some kind of decaying function. If I add 5 people to a 5-person project, it'll be pretty hard on efficiency. If I add 5 people to 100-person project, I probably won't notice the difference. People tend to focus on the 100-person part of the curve (Brooks did, since he was talking about a massive project at IBM), which is why they summarize Brooks' Law as "Adding manpower to a late software project makes it later."

The converse, of course, is not necessarily true: removing manpower from a late software project doesn't necessarily make it earlier. Wouldn't that be nice!

But what if you tried? Eventually you would get back to the early part of the curve, where n is some small number. What if n=1: you're working all by yourself? Then n(n-1)/2 = 0, with zero communications effort! Awesome! I can get infinite work done in zero time!

Well, no, of course not. When n is large, communications efficiency becomes the controlling factor in your overall efficiency. If you take the simplified formula towards n=0, it will give you a "singularity," that is, a place where the number explodes because you're dividing by zero.

In real life, as the number of people on the project decreases, effects other than communications become the limiting factors. This means that if you're going to try to produce more work with fewer people than anyone else, you will have different problems than everyone else. (Of course, this makes it an inherently interesting question.)

So what are those limiting factors? Well, as a person who has worked on quite a few solo projects, I can give the top two that I've found: first, working totally alone, there's no chance for mutual motivation. Second, you run a greater risk of groupthink effects, where you don't have enough diverse opinions and you might end up going on terrible, inefficient tangents and failing to find anything like an optimal solution. For example, a one-man company run by a programmer will probably lack any business sense, and thus fail to make money even if the programmer is a genius.

What's interesting is that nowadays, smaller teams can produce much more than ever before. That's because something is already mitigating the most serious of these effects. Motivation? Well, if more people can do what they love, motivation is easier. Groupthink? Well, the Internet. QED.

I wonder how far you can take that model?

Side Note

The technological singularity can be expected to follow a similar pattern to what I called "Brooks' Singularity" above. Eventually, we will find out that our formula isn't complex enough and there is a speed-limiting factor on technological progress. Believe it or not, the time between "paradigm shifts" can't actually proceed linearly through zero until it takes negative time to do a shift.

Completely Offtopic

Q: So, how are you feeling about everything that's been happening lately?
A: Um... resigned?

Mouse Angles

prozac writes and Chicago responds about popup menus and the annoying "90 degree problem" with navigating submenus.

This is a good time to mention that Apple, being a bunch of freaking geniuses, actually solved the problem long ago in MacOS. This was demonstrated to me about 6 years ago on MacOS 9, so hopefully the behaviour is retained on MacOS X.

Whether or not the submenu pops up or disappears depends on which direction and how fast you move the mouse. Basically, if it's mostly down and to the right, it sticks on the currently popped-up menu, even if you mouse over another one, because we assume you're trying to get to the submenu (moving to the right, after all!); if it's mostly up or to the left, it pops up the menu you're currently over, instantly, because you're obviously not heading toward the current submenu. And there are never any arbitrary delays.

It's really hard to explain, but it's total genius. The "wait for half a second arbitrarily" technique and the "never delay" technique are massively inferior to this. I have a feeling Apple patented this technique. I wonder if it's expired yet?

Also, when you underline "Gaggle" in MacOS, the gg isn't underlined because it has descenders, but the underline stops a little bit into the gg, so it looks perfect and you can't fake it on other OSes by just asking to underline Ga and le.

That is all.

14 Jul 2006 (updated 15 Jul 2006 at 02:48 UTC) »
Optimizing for Productivity

I was reading in the aforementioned Good to Great about different "great" companies and their way of reducing their vision to a single "hedgehog concept," which has various useful boring attributes that I won't go into right now, as well as one more that I will: an optimization goal, which they call the "key denominator." Basically, to be maximally financially successful, a company needs to optimize its operations for the best ratio of profit to something.

Some companies use things like "most profit per store" or "most profit per product line." They talked about a bank in the U.S., which optimized its operations for "most profit per employee" when banks became deregulated. The most obvious way to do this is to reduce the number of employees and/or otherwise chop your HR costs, but in a world where banks are suddenly not protected monopolies anymore, that might be the only thing you can do.

That particular optimization goal got me thinking. I don't really like big companies very much. Once they get too big, they get annoying because it's hard to keep track of everyone and there are always some stupid people. So optimizing a new company in terms of profit per employee - which isn't nearly so painful in a company with no initial employees as it is when you'd have to lay off a bunch of people - might be quite entertaining.

And then I thought, wait, that's not quite it. You can increase profit per employee by simply making your employees work longer hours and paying them less, which isn't really what I want. Okay, so I work really long hours, but that's just me, and it's still sort of cheating; it makes me look more productive but I'm only producing more, not producing more per hour. If I was really smart, I would be able to produce more per hour than the next guy. If I also work more hours, well, great, but that's just a bonus.

And so that's it. We want the smartest, most productive people around, and the people we get, we want to make even smarter and more productive. So why not maximize profit per man-hour? Why not do it all the way across the company, and why not help your customers do it too?

In other words, laziness, impatience, and hubris. It's not just for programmers anymore!

A few loosely correlated comments this time.

Stubbornness is a Virtue - Sometimes

I've been thinking a lot lately about what makes someone a good high-level leader. There are lots of things to consider, of course, but here's one of the things that has struck me most in the last little while: the best leaders are stubborn and also good listeners.

What, it's possible to be both?

Yes. People who are stubborn persist in a particular way of thinking despite encountering huge resistance. If you're trying to convince someone about a revolutionary new idea that will solve their problems, stubbornness is critical. Meanwhile, people who are good listeners adapt their thinking when someone convinces them that they've made a mistake or were missing some information. If you're wrong - which is frequently, if you have a lot of revolutionary ideas - you have to be a good listener in order to correct your mistakes fast.

The proper combination, though, is important. The best leader will persist in thinking something until someone properly convinces them otherwise. Once they've been convinced, then they change their mind and they're stubborn in the new way. That's not flip flopping; that's just a smart way to learn from your mistakes.

If you're missing one of those two traits, you'll fail in one of two very common ways: you'll persist in believing something stupid even it's been made totally clear and obvious to everyone else that it's wrong. Or else you'll never settle on a single direction, constantly changing your mind whenever anyone raises an objection.

Some people take it a step further and switch back and forth between being too stubborn and being not stubborn enough. One example I've seen a few times is a stubborn belief that "the investors are always right." So whatever it is that the investors want, you'll give them... even if they change their mind day after day. So you're left spinning in circles because of your persistent belief in someone else's idea. Other popular variations of this are "the customer is always right" and, most annoyingly for me at the moment, "the CEO is always right."

So my suggestion: believe in yourself. And then don't be afraid to let other people try to change your mind.

Trust and Respect

On another note, I was thinking about what would be the ideal company culture. This goes back to my earlier notes on membership control for working in a group.

I think a culture based on two primary concepts would be nice: trust and respect. It works like this:

First, you need to trust that people will be honest and upstanding (if you're paranoid, you can check their backgrounds before you admit them to the group). Since you trust them, you don't need annoying controls to make sure they're not trying to screw you, because you're not worried about that. If anyone ever gives evidence that they can't be trusted, remove them immediately; there's nothing more deadly to a social environment than a person who can't be trusted.

Second, you only admit people to the group whom you respect. That is, if you don't think they can do a good job or think clearly or impress customers or whatever, then you don't want them in your group. (In his book Blink, Malcolm Gladwell discusses a researcher who has discovered the #1 indicator of marriage breakups: a spouse who obviously looks down on the other.)

These two factors work together to produce interesting results. For example, if you respect someone, you will automatically become more trustworthy; would you screw over someone that you respect? Would you slack off at work, if it would let them down or hurt the project they're working on? Probably not. And that means they can trust you even more.

Meanwhile, because you respect them and so they trust you, you're actually working harder and producing more. That makes it easier for other people to respect you. And since they do, they won't want to screw you over, so they'll be more trustworthy and work harder. And so on.

This system covers most of the other factors in building a group as special cases. For example, if you're not a good listener, you obviously aren't giving the people the respect they deserve when they talk to you. And if you're not stubborn - if you just keep changing direction randomly - then eventually people will realize that you can't be trusted, because your opinion is based on whichever way the wind is blowing at the moment.

And about that Smart and Gets Things Done rule? Well, if you're not smart, that's going to show up pretty fast, nobody will respect you, and you're out. If you don't get things done, then you can't be trusted in an emergency, and you're out.

This system even deals with interpersonal compatibility in an interesting way: different people respect other people for different reasons. For example, a group of musicians might highly respect someone with a lot of musical talent, regardless of their table manners; a group of aristocrats probably wouldn't. (They can buy musicians for a dime a dozen, but it takes money to become an aristocrat!) This is interesting because the groups are probably fundamentally incompatible. If each group selects its members based on the trust/respect rule, things should rapidly work themselves out into two separate groups.

And so on. I like this rule because it makes the "membership control" system easy. I wouldn't feel the least bit guilty about kicking someone out of my company if I don't respect their abilities or if they can't be trusted. And yet those two things cover all the other important things.

Good to Great

I just finished reading the book Good to Great. It's interesting and makes many believable points, but it does scare me a bit with its selection bias. They tried really hard to avoid sampling errors, but didn't quite make it: the problem is that highly risky behaviours are correlated with both huge success and huge failure, while highly unrisky behaviours are correlated with mediocrity. So when you compare a set of "great" companies with a set of "merely good" companies, you will definitely find some statistically significant differences. Those differences might very well be absolutely critical to becoming great. However, they might also greatly increase your chances of killing yourself.

To name just a couple of examples, the "Level 5 leadership" concept makes perfect sense in terms of growing a company for the long term. But if you're "almost" a level 5 leader, then maybe you're just a quiet person with a vision and nobody really listens to you and so your company goes in random directions. Or the "hedgehog concept" - do one thing, and do it well. This is great, as long as it's the right thing. The book even gives some good advice on how to choose the right thing. But what if it's not quite the right thing? Then you're screwed. Massive diversification may prevent greatness, but at least you can hedge your bets.

Beware of advice that seems too good to be true.

Dynamic Equilibrium

I wrote before about precious instability, the idea that I design my life in an "unstable" fashion, then carefully control it, in order to make it more manoeuverable. For people who know me, some recent events might have put that into a bit more context.

But one thing I realized since then is that this concept is much more widely applicable than just me and my idiosyncratic ways. Modern high-tech business requires this kind of instability, but only when it's in the right places.

Put simply, there are two fundamental ways to manage any project: creatively (ie. a prototype) or using a well-defined process. Well-defined processes are efficient and scalable; creative solutions are resilient and flexible. (I wrote about efficiency vs. resilience before too.) An obvious way to balance the two is what I've been trying to do lately: do it (whatever "it" is, even if it's a business activity) the first time as a creative prototype, then try to generalize what you did into a repeatable process. As with software, making a mostly-right generalized design is usually possible, for a good designer, after you've done it about twice creatively.

But once your process is well-defined, then what? I've experimented with the "hand it off entirely to process-driven people" technique, but that often doesn't work very well, because if your process isn't perfect at the start, process-driven people aren't very good at fixing it. Meanwhile, trying to find some middle ground ("we'll follow the process, but bend the rules whenever necessary") is a disaster, because it's not a combined solution, it's a compromise, and it ends up being neither very efficient nor very creative. Gross.

I don't exactly have a tried-and-true solution to this, but here's what I'm thinking. Joel Spolsky has written about The Development Abstraction Layer, which is a partly-right variant of what I'm going to suggest. What he wants is a programmer's ideal world: the programmer just builds what he wants, which is assumed to be the right thing, and the rest of the company is an "abstraction layer" that gets all the annoying repetitive cruft out of the way. In terms of creativity vs. processes, that means the "developers" are creative, and the rest of the company delivers things based on defined processes (or at least inter-group communication follows defined processes).

The part where I disagree with Joel is the part where I've already specifically screwed up so I figure I can serve as a warning to everyone: developers shouldn't be the ones running the show. Not quite. Smart people who get stuff done (another Joelism) need to be running the show, but it's not about programming and designs and version control and operating systems and XML. It's about business and customer relations and good communication and... being smart, and getting stuff done. It just turns out that Joel's kind of programmer is exactly the right personality type... if they're deliberately caring about the right things, which is more than just the software.

So how about this. Classify your people into two categories: the creative solutions people and the process-driven people. Don't separate them physically or logistically; they have to work together. The creative solutions people design and update business processes. The process-driven people execute those processes - maybe only after the first prototype - and give rapid feedback on results. Don't worry, Joel, both groups should be smart and get stuff done. But they're two different kinds of people. Some very smart people really have a great time fixing bugs and optimizing failure rates and avoiding creativity at all costs. (Seriously, there are lots of people like this!) These aren't my favourite people to work with, but they're extremely useful because they like doing exactly the kind of stuff that I don't.

At the same time, process-driven people risk falling into dangerously stable ruts and getting blindsided when changes become necessary, because they enjoy their stability and will try to retain it at all costs.

So as a business designer, you need to set up the incentives of the creative and process people correctly. My suggestion? Don't let the process people define their own processes. Make them responsible for executing defined processes efficiently and make creative people responsible for changing the processes to make them efficient.

I call this "dynamic equilibrium," after the term from high school science class. It looks pretty stable, because the majority of people are sitting around executing repeatable processes. It's just that it's not really stable, because exceptions will come up, and when they do, weird stuff happens automatically to re-stabilize the system.

Disclaimer: I haven't tried out this advice yet, but it sounds like fun.

Footnote

Crap, I really wanted to work in my example about airlines losing my baggage, but it didn't fit... so here it is anyway.

Airlines process absurd amounts of baggage every day very efficiently, and lose a vanishingly small percentage of it. That's an efficient business process.

But every now and then they lose a bag, and they don't know what to do, because if they knew what to do, it wouldn't be lost, sort of by definition.

You need business process people around to deal with the majority of cases, where the bags aren't lost. And you need creative people around to handle the exceptions, learn from them, and fix the processes so even fewer bags get lost.

28 Jun 2006 (updated 28 Jun 2006 at 03:24 UTC) »
Practical Extraction

Well, that could have been worse. Presumably.

Pathologically Eclectic

Since people keep asking, yes, I am currently interested in projects related to distributed storage, network path optimization, unified configuration (UniConf, GConf3, KDE4 KConfig), and several other loosely related things. Like unified multi-contributor documentation and decent Unix printing and NaNoWriMo and Business Design (hah! See! I didn't make it up!) and and and... so yeah, I'll just pick one of those.

Also, because I'm obviously insane, I have subconsciously arranged to be in Toronto for the entire Toronto Fringe Festival. Ha ha. Very funny, brain. This is not good at all.

Unraveling at the Fringe

Several people have posted reviews of some Fringe Festival shows they've seen, along with their ratings. For me, though, the purpose of the fringe festival isn't so much to see or measure quality entertainment. After all, as I've mentioned, as far as art is concerned, I have no idea what I'm doing and thus it's easier for me to appreciate.

No, the purpose is to gather another year's worth of random input. So without further ado, I bring you: 40 tidbits of randomness, in alphabetical order by show name. (Yes, I have a leaning toward puns. So shoot me.)

Needlephobic spasms. No, no, his house - the Hur house; it's enough to turn gay men straight.

Wah - I love pizza - but I Kant!

People never to break up with; people never to buy dinosaur souvenirs from.

Airport jokes and flamenco made interesting; human relationships the MIT way.

Eat here - get gas for free! Dancing in the rain and the worst audition ever.

Brain sucking ingenious crazy dance! Anorexia personified, and at least one continuously improvised dance scene!

Dinosaurs aren't souvenirs, they're EOLed products. White clothes make great projection screens! Breakdance fusion!

Queuing was stolen from the British; also, There was a young man from Peru, whose limerick stopped at line two. Non-lost innocence and a squirt of toilet humour. Non-toilet-related private moments made public and an encounter with Sweden's orb of literacy.

Speaking in tongues, the musical. Verbal smilies!

The missing minute from my life.

You can't escape a guilty conscience... and eventually, your creations will take on a life of their own (ergh, tell me about it).

Nunavut -- twice. Dames in disdress, vs. a very different form of fake but extremely convincing shyness.

Always have a lawyer look over your contracts, particularly if you make them with the devil. Speaking of which: Stephen Harper and David Suzuki, together at last, along with Terry Fox's right sock.

Octopus dance (I wish I'd had a fork). Vegetable dance (enough to make you dizzy).

Learning English from Entertainment Tonight and a wishing well talks back to cheapskates. Criminals are people too.

Honourable mention, even though it was free and thus doesn't count: 11-second dance parties, 16-second slowdance parties (thanks, Cheer Politeness), 22-second dance parties, and so on. To heck with the popular kids! To heck with respectable theatre!

Peer pressure and mutual motivation

And now for the obligatory Advogato tie-in.

Last year at the Fringe Festival, I saw 26 shows in 10 days, and it was exhausting. When I set out this year, I decided: no sense overdoing it. I'm not going to try to beat last year's insane record. I'll just see as many shows as I want to see, then I'll stop.

So then I saw 40 shows. Oops. What happened?

What happened was peer pressure. None of us really wanted to see that many shows. But when you have a few people working together and encouraging each other, the positive feedback loop can be irresistable. "Come on, you can't be tired yet! I want to see one more!" "Come on, there's a show on - let's leave work early!" "Come on, you already bought the multi-ticket pass, so you might as well use it up!"

I learned a couple of years ago now that you should never let a person work on a project at work alone; it's hard for someone to get motivated that way. This is the idea behind pair programming, but you don't have to go that far; two people can work on parts of the same project, as long as their work is slightly mutually dependent - say, at least one reason to rely on the other person per day. Then if one person falls behind, they're disappointing the other person, and vice versa. This mutual motivation technique can be extremely effective, and it vastly outweighs the cost of having a second person working on the project, because it'll probably get done more than twice as fast.

So now, thanks to all that motivation, I feel like I've accomplished something. It turned out to be something ill-conceived and stupid; who the heck needs to see 40 shows in 10 days?? But I guess that's my point. Motivational techniques work, even if you're trying to get people to do ill-conceived, stupid things.

12 Jun 2006 (updated 12 Jun 2006 at 20:34 UTC) »
Independent thought

    The purpose of a designer is to make happy coincidences happen more frequently.
    -- me

Someone else's independent thought

    So, what benefit am I taking from all the freedoms I have because I am not a duck? ... That's the sort of question you should expect poets to be asking on your behalf.
    -- Jem Rolls
Trying and Being

    Do or do not. There is no "try."
    -- Yoda

The thing I find most interesting thing about yoga classes is the difference between how it feels when you're clueless (like me, usually) and when you know what you're doing (which I do in a few rare cases). When you're clueless, every position is a struggle, and it's really hard to maintain it for more than a few seconds. But when you actually figure out how to do one of the stretches properly, it becomes easy all of a sudden, and it doesn't feel like a stretch at all.

Sadly, none of the instructors have actually brought this up in class. You seem to be left to figure it out by yourself. But once you realize it's true, you stop trying to stretch yourself into a particular position, and you spend more time trying to figure out the trick of getting into that position properly. It's much more productive.

The rest of the world works like that too. And there are various philosophies that suggest that if you're truly enlightened, you can find the right place to stand and the world will revolve around you, working out just like you want.

But there's a catch. As much as it would be nice to be enlightened and just let things be the way you want, if you're not quite so enlightened, all that will accomplish is randomness or stagnation (depending where you land). In yoga class, you have to attain a certain degree of flexibility before you can even really start - and that takes solid effort. Once you get that far, you have to figure out how to do the different positions properly - and that takes effort too, albeit of a different kind. And the further you progress, the more convoluted it becomes.

And yes, I can tie this back to programming. I'm glad you asked.

I did a presentation at work a couple of years ago called "Coding Faster." In it I described the well-known concept of being in "The Zone" and also the opposite, which I called "Ooze." Most programmers aim to spend as much time as possible in the zone, but that's not quite right; the zone is being. You know what you want, and you know how to do it, and really: you get into the zone and it just happens by itself. (If you're a programmer and that's not how it feels for you, then you're doing it wrong. Trust me on this.)

The part people miss, though, is the ooze time leading up to the zone. It takes a lot of hard work to understand the universe well enough to be able to be properly productive in the zone. Ooze time, although it's slow and feels unproductive, is critical to overall productivity. Ooze time is trying.

So I'm afraid I'm going to have to disagree with Yoda on this one. There is definitely a "try." Anyway, he was probably just telling Luke that to keep things simple for the early lessons. But I agree that "try" is really not the same thing as "do," which I think was his real point.

The Curse is Broken

109 days. So there.

I had to change a lot of things about myself and the company in order to break the curse, but the core is intact and finally it's self-consistent.

Albeit in a different way than I was expecting.

131 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!