Older blog entries for apenwarr (starting at number 147)

Taking The Hard Way Out

One thing that has bothered me for a long time was a comment in a lecture by famed Waterloo economics professor Larry Smith. To paraphrase, he said there are a lot of big problem areas in business that need solving that are just ignored. Meanwhile, everyone in the computer industry seems to have a fatal attraction a particular other set of problems, which they solve over and over again.

The example he gave at the time was Windows vs. Linux: who would want to start a company to compete with Microsoft in this day and age? Even if you did manage to win (or even to not die), it would just be so much work! And no, the irony was not lost on me at that point in my entrepreneurial career.

My answer to his question at the time? Well, some projects are just more interesting than others, and I want my career to be interesting. It was abundantly clear to me even then that going into, say, banking software was the way to make loads of money, because the customers are rich and (it was a y2k madness at the time) boy, do they ever have lots of easy problems to solve. But banking software was by no means interesting. So I elected to take the path more travelled, as it were, and go for technical challenge instead of easy megaprofits. In other words, a compromise. I even formed a little continuum in my mind of different kinds of programming projects, in which the most technically challenging ones (hardest I could think of: video games) are the least profitable for the developer. And vice versa: banking software has got to be some of the least technically difficult software in the universe, and attracts some of the very worst programmers, who then get paid a lot to work reasonable hours with very low stress.

But the fact is that I'd be more than pleased to increase my stress levels a bit in exchange for an interesting project. That much isn't a compromise; in fact, too little stress is downright boring. That's why the best developers gravitate toward harder projects instead of easier ones. Video game developers know perfectly well what they're getting into before they start. They're buying into a whole lifestyle.

Well, good for them. I wasn't quite willing to go as far as the insane working conditions in the gaming industry, but I still wanted challenge, and operating systems seemed fun. There was stress, but that's not a compromise, because I wanted stress.

The compromise was the profits. Say what you like, and I do dearly love the free software community, but making massive quantities of money is a great way to make an already-fun company even more fun.

So what's the non-compromise solution in this case? Well, the form of the solution is obvious enough: it must be highly profitable but also challenging and difficult so that it can be fun.

It's pretty hard to take something that's challenging and difficult and not very profitable, and then make it highly profitable. Profits don't just magically appear just because you did something that was difficult. If it was the wrong difficult project, you're still out of luck.

But what about taking something profitable and making it difficult? Why, you can take just about anything and make it difficult if you try hard enough!

Of course, it would be best if the difficulty were actually warranted in some way, for example, to make the product even more profitable, or to also make the world a better place, or whatever. But that's actually only a secondary requirement. Look how much fun we had revamping our knowledgebase, for example.

So what if I could find a useful way to make, say, banking software challenging and fun?

On Wishful Thinking

wlach commented on my INTJ posting to mention that this sort of thing is often just telling you what you want to hear, so of course it sounds believable. Certainly there are plenty of "personality tests" and other things out there (astrology, anyone?) that do this on purpose in order to manipulate you, but the Myers-Briggs and Keirsey personality descriptions are different. They're just too specific to be able to fall for this problem.

Especially in the case of INTJ, sure, it says a lot of nice stuff, but it says a lot of specific stuff that not only fails to apply to almost everyone, but also that most people would find violently opposed to their self-image. "This self-confidence, sometimes mistaken for arrogance by the less-decisive..." and "INTJs have also been known to take it upon themselves to implement critical decisions without consulting their supervisors or co-workers." These are simply not generic statements. They're not true of almost everybody.

Moreover, all the other personality type descriptions I read, which all try to say as many nice things as possible ("accentuate the positive," you could say), simply don't match me nearly as well. In other words, a highly accurate personality description was attached to me automatically by a 76-question multiple choice quiz. That's pretty cool.

But I already know a lot about myself. The really interesting thing is slotting other people into their own types and realizing why you've had trouble dealing with them, and then what you can do about it now that you know.

Hiring People

Peter Norvig at Google writes about hiring only people who are above the average for your company, a strategy I first heard about from my friend at Amazon.com. Unlike Norvig, I didn't do a simulation to see if it made sense, but it did make intuitive sense to me at the time.

They also talk about the "no hiring manager" concept. NITI used to do this sort of subconsciously, but has been drifting away from it lately (letting individual project managers do their own hiring) with exactly the results that Norvig predicts. I didn't do a graph for that either (and indeed, the graph for that one isn't very convincing; the results are too close), and it also didn't seem intuitively obvious, but having seen it for myself, I agree.

Incidentally, the reason we had for doing the "no hiring manager" thing originally is that I didn't believe that project management skills and interview skills were particularly correlated. For example, I never felt that I was a very good interviewer, so I found people who were good at it and had them do it instead. At the very least, random team members are probably better at interviewing and hiring than project managers - as the people who actually do that kind of work, they know exactly what skills the person will need to have!

Okay, so score two more points for Google, I guess.


After some recent discussions with various people, I started looking again into the Keirsey personality profiles (related to the Myers-Briggs "MBTI" tests). Well, I just have to say that those things are a lot more statistically valid than all those memes people keep posting.

In fact, reading the results, I almost feel like I'm just acting out a script that someone laid out for me at birth. This INTJ description just spooked me out completely by its accuracy. And then another one says very interesting things about my style of working, such as, "INTJs are natural leaders, although they usually choose to remain in the background until they see a real need to take over the lead," and "Unlike the INTP, they do not follow an idea as far as they possibly can, seeking only to understand it fully - INTJs are driven to come to conclusions about ideas." Hear that, pphaneuf?

Meanwhile, reading some of the other descriptions has helped me explain why certain other people I've tried to deal with just seem to keep doing things that sound totally insane to me; I guess it's okay, since they're normal and I'm not. INTJ is the least common of all 16 types.

Make Your Own Luck

The main problem I've had in dealing with people, when I've had problems, is what I think comes to the "intuition" vs. "sensing" communication problems (eg. INTJ vs. ISTJ personalities). That is, intuition people tend to jump rapidly from the problem to the solution, skipping all the intermediate steps. Sensing people take things one step at a time, because they don't really see the solution at first, just a process that will eventually lead them to one. Remember that stuff about process-driven vs. results-driven people? Well, there it is again, stated another way.

Both "N" and "S" people can fail badly if they don't realize their weaknesses. I think I've seen both types of failures. When an "N" person fails, it's because they see what the solution has to look like, but they don't bother even figuring out the steps to get there. For example, it's intuitively obvious that the final solution to our modern intellectual property concerns will have to be a new system in which copying most kinds of stuff can be done freely; any other system will simply be too wildly unstable to work. But how will we get from here to there? And what will it look like when we get there? That's something you need the "S" people to deal with. Except that most of the "S" people won't believe you when you say that's the way things have to turn out. They simply can't see that far ahead. And "N" people who can't explain it to them are actually worse than useless, because they're just confusing.

So that's the problem pure "S" people run into: they just can't see too far ahead. And not only that, but sometimes they don't even realize that they can't see too far ahead.

For these "sensing" people, just following logically from one step to the next will always give good results; if they don't, well, you didn't follow the logical steps well enough. Or if you don't follow the logical steps and it works out anyway, maybe you were just lucky.

But it's not usually luck. In fact, it's probably an intuitionist who laid things out in just the right way that doing the logical next steps, one after another, got you where they wanted you to be.

And that's just spooky.

Salary Survey

On a completely unrelated note, I found a really interesting New York salary survey. It's from 2000 right after the bubble burst, but I'm sure salaries haven't changed that much since then. I have two things to say about the results:

1. Wow, talk about supply and demand.

2. It appears that salaries are just all over the map, so the only definition of "fair" is "whatever you can get away with." Sigh. Obviously not something the INTJ in me wanted to hear :)

REaction Engine

I've realized that proactivity is just reactivity with negative latency.

What? Okay, first of all, "be proactive" is sometimes known as common management blabber, and being a one-time manager myself, I'm allowed to use it. But used correctly, "proactive" is actually a useful word. It means solving problems before they happen.

And reactivity is solving problems after they happen. Aha. So proactivity is just reactivity where the effect happens before the cause: negative latency.

Easy, right? Well, no, since because some something or other with the space-time continuum you can't actually do that. But negative latency has been successfully employed in the past.

Here's what I'm thinking. If you operate in a normal "reactive" mode, your simple goal is to react as quickly and accurately as possible. Having given up on preventing problems, you want them to at least go away, if at all possible, before too many people notice. This is a good way to run a startup company, where you can't possibly expect to prevent all problems at first; there are just too many of them to start with.

The better you get at being reactive, the faster you can solve the problem, and latency approaches zero.

Imagine if your boat hits an iceberg and gets a hole in it. You want to start bailing water as soon as possible. If you notice that the hole is too big for that to work, you'll need to start evacuating the ship, and so on. But the better you react, the fewer people who drown.

As latency approaches zero, the amount of work increases exponentially; latency can never really reach zero using a reactive method, but it can come infinitely close with an infinite amount of work. This is a singularity.

Now what about that negative latency? Well, it requires some different techniques, and creative thinking, versus positive latency. Since you can't run time backwards, you have to instead be living in the future, reacting now to what you expect to happen, say, 10 minutes from now. Living in the future is impossible, of course, but you can try your best to predict what it will be like. And predictions are about probability. This is the only real difference between being reactive and being proactive.

Many people don't realize this, and think of the two concepts as requiring a total change in technique: "proactively prevent all problems." This leads people to overanalyze the situation, taking only extremely safe, risk-free approaches that will never have problems - but will also be the slowest possible way to get there. This is sort of like sending all the bytes from all the files just in case the TFTP client might want them someday.

In fact, just like the current problems that you react to, each future problem has a cost associated with its occurrence, and increasing costs associated with not solving the problem once it occurs. Reactivity helps you avoid that increasing cost; proactivity removes the fixed initial cost. But while reactivity has an easy-to-calculate cost-to-benefit ratio (you know the cost of not solving the problem, you know the problem has occurred, and you know how much it'll cost you to solve it), proactivity has an extra element to it: risk. Some potential problems, even though they might occur, won't. For those ones, avoiding them is purely a cost and has no benefit.

So as with all negative latency problems, probability and intelligence are key. But you can outdo anybody who is merely reactive or naively proactive, just by thinking harder and more carefully than they do.

28 Jul 2006 (updated 11 Oct 2006 at 04:25 UTC) »

Random Person: Hey, what's in the garbage bag?

Avery: Stuffed corporate bunnies to accompany my accounting poetry.

Random Person: ...oh.

Grand Old Unified Relational Documentation

I'm pleased that what is probably my last major innovation on behalf of NITI, the new Knowledgebase system codenamed GourD, has finally been opened to the public. Even though the code was rewritten almost from scratch after I last had my dirty fingers in it. Thank chrisk for that - it's much better now.

But I still claim credit for the idea, and even if I don't, it's a cool idea anyhow. Among the features of the new system are:

  • It's based on mediawiki, so its UI is designed for collaboration.
  • We greatly improved the search engine, whose ranking and summarization algorithms are better than any other knowledgebase I've seen (and a major improvement over standard mediawiki, too).
  • It has our whole searchable user manual in it, not just individual knowledgebase articles.
  • It has a really neat AJAX "rate this article" system that's actually slightly less obnoxious than other such systems.
  • You can subscribe to articles or whole categories, and get a nightly notification of which articles have been newly published or changed in a non-minor way: this is less confusing for newbies than RecentChanges type pages.
  • It automatically produces a remarkably accurate and useful "similar articles" list based on a variant of my document correlation algorithm that I wrote about so long ago.

But my favourite part is really:

Separation of style, content, and... structure?

So everybody knows that the big glamourous thing nowadays is HTML4, XML, CSS, and so on, the point of all of which are to be able to separate "style" from "content." That is, someone correctly noticed that most writers are poor layout artists and vice versa, so why not do the jobs separately instead of what computers normally encourage you to do, which is spend more time fiddling with fonts in Word than actually writing?

Okay, so that's sort of worked out for some people, and we get a bit closer to the ideal every time HTML makes another evolutionary step.

But anyway, what if we took it one step further? What if we split out the high-level structure of your documentation, and made it somebody else's job?

What? Okay, this sounds a bit confusing. But think of it this way. The world didn't just lose layout experts when it started forcing writers to format their own documents; the world is also losing professional copy editors as more and more people can publish their work directly and have it look half-decent. That means more and more documents are... kind of crappy.

Think of a big 500-page technical manual (our product has one, for instance). Nowadays, those 500 pages won't be written by a single person; not at all. They're written over the course of many years by several different people, where some of those people are actually joining and leaving the company while the book is evolving. The result? Inconsistent drivel, like almost all modern technical documents have become.

The rare exceptions are documents that are written quickly enough, by a single person with a good sense of style, that they're actually good. Except they rapidly get outdated and can never be very long (ie. complete and thorough) because both of those preclude having a single writer enforce his/her artistic integrity; artistic integrity requires a pass over the entire document, by a single person, in a relatively short time.

Incidentally, all this is why books like the "for dummies" series are so popular. They're written by one or two authors in a relatively short time, edited by professional copy editors, laid out by professional layout artists, and rewritten from scratch every few editions. That produces (comparatively, at least) high quality work that really beats what most companies produce.

So what I observed from all this was simple: most KB systems already abstract style from content; generally, you enter plaintext and they format it like a KB article automatically. Good, so your layout artist isn't the same as your tech writer.

But I've never seen a technical documentation system that separates content from structure. That is, there are generally a variety of writers - which is mostly unavoidable in a modern long-term development project, unless you want to rewrite from scratch all the time, which is horrendously inefficient. But there is no editor. So the writers tend to each contribute a new chunk of prose (if you're lucky, it's a whole chapter), and then some unlucky soul - possibly one of your writers - gets to tack it into the big unwieldy book somewhere, until the big unwieldy book gets so big and unwieldy that nobody could possibly read it from cover to cover, then nobody even reads it at all, then you stop including printed copies with your product and just give them a pdf, and then you kind of stop shipping the pdf and just leave it on your web site, and then eventually people don't read documentation at all and resort to forums and knowledgebase searches.

Sound familiar?

Well, GourD has the solution. Documentation is written in convenient, bite-sized chunks, by anyone who wants to contribute it. These start off life as searchable, categoriezed knowledgebase articles with various writing styles. But then our "artistic integrator" (currently apm) can go through and clean up any stylistic inconsistencies between articles - in articles that matter.

She then has the ability to create "books" out of a collection of similarly-styled articles. Where necessary, she can edit articles to make them read better in the sequence of the book. When the articles are not in book form, these changes don't disrupt anything because any confusion can be resolved by just hyperlinking terminology from one article to another or using the "similar pages" or "categories" features; welcome to the web.

The "structure" that the artistic integrator adds is through the process of weaving together a collection of articles. Imagine you had a big bucket of pearls collected over a period of time. To make a good necklace, someone has to choose pearls of similar size and shape and assemble them in the appropriate sequence, maybe polishing a few here and there. The result is beautiful, but not because the person who made the necklace spent dozens of clam-years growing pearls; no, the pearls were produced by someone else, and the final beauty comes just from combining them the right way.

You can produce a series of 100-page books on useful topics just by mixing and matching and polishing GourD (it's much more than just a "knowledgebase" when used like this) articles - and you can do it very quickly, and the artistic integrator doesn't have to write them all from scratch, so it's highly efficient and high parallelizable.

And your books will be better than any of your competitors'.

Wouldn't this be a great way to write documentation for Open Source projects?

Brooks' Singularity

In The Mythical Man-Month, Fred Brooks explains that the amount of communication necessary for a project time is n(n-1)/2, where n is the number of people on the team. Eventually, the communications effort begins to overshadow the actual productive effort. This is the primary factor limiting team sizes, which is why people start breaking into multiple teams, hierarchies, etc beyond a certain size - hierarchies may not be all that efficient, but at least it reduces the need for huge amounts of communication.

Now, this efficiency decrease is some kind of decaying function. If I add 5 people to a 5-person project, it'll be pretty hard on efficiency. If I add 5 people to 100-person project, I probably won't notice the difference. People tend to focus on the 100-person part of the curve (Brooks did, since he was talking about a massive project at IBM), which is why they summarize Brooks' Law as "Adding manpower to a late software project makes it later."

The converse, of course, is not necessarily true: removing manpower from a late software project doesn't necessarily make it earlier. Wouldn't that be nice!

But what if you tried? Eventually you would get back to the early part of the curve, where n is some small number. What if n=1: you're working all by yourself? Then n(n-1)/2 = 0, with zero communications effort! Awesome! I can get infinite work done in zero time!

Well, no, of course not. When n is large, communications efficiency becomes the controlling factor in your overall efficiency. If you take the simplified formula towards n=0, it will give you a "singularity," that is, a place where the number explodes because you're dividing by zero.

In real life, as the number of people on the project decreases, effects other than communications become the limiting factors. This means that if you're going to try to produce more work with fewer people than anyone else, you will have different problems than everyone else. (Of course, this makes it an inherently interesting question.)

So what are those limiting factors? Well, as a person who has worked on quite a few solo projects, I can give the top two that I've found: first, working totally alone, there's no chance for mutual motivation. Second, you run a greater risk of groupthink effects, where you don't have enough diverse opinions and you might end up going on terrible, inefficient tangents and failing to find anything like an optimal solution. For example, a one-man company run by a programmer will probably lack any business sense, and thus fail to make money even if the programmer is a genius.

What's interesting is that nowadays, smaller teams can produce much more than ever before. That's because something is already mitigating the most serious of these effects. Motivation? Well, if more people can do what they love, motivation is easier. Groupthink? Well, the Internet. QED.

I wonder how far you can take that model?

Side Note

The technological singularity can be expected to follow a similar pattern to what I called "Brooks' Singularity" above. Eventually, we will find out that our formula isn't complex enough and there is a speed-limiting factor on technological progress. Believe it or not, the time between "paradigm shifts" can't actually proceed linearly through zero until it takes negative time to do a shift.

Completely Offtopic

Q: So, how are you feeling about everything that's been happening lately?
A: Um... resigned?

Mouse Angles

prozac writes and Chicago responds about popup menus and the annoying "90 degree problem" with navigating submenus.

This is a good time to mention that Apple, being a bunch of freaking geniuses, actually solved the problem long ago in MacOS. This was demonstrated to me about 6 years ago on MacOS 9, so hopefully the behaviour is retained on MacOS X.

Whether or not the submenu pops up or disappears depends on which direction and how fast you move the mouse. Basically, if it's mostly down and to the right, it sticks on the currently popped-up menu, even if you mouse over another one, because we assume you're trying to get to the submenu (moving to the right, after all!); if it's mostly up or to the left, it pops up the menu you're currently over, instantly, because you're obviously not heading toward the current submenu. And there are never any arbitrary delays.

It's really hard to explain, but it's total genius. The "wait for half a second arbitrarily" technique and the "never delay" technique are massively inferior to this. I have a feeling Apple patented this technique. I wonder if it's expired yet?

Also, when you underline "Gaggle" in MacOS, the gg isn't underlined because it has descenders, but the underline stops a little bit into the gg, so it looks perfect and you can't fake it on other OSes by just asking to underline Ga and le.

That is all.

14 Jul 2006 (updated 15 Jul 2006 at 02:48 UTC) »
Optimizing for Productivity

I was reading in the aforementioned Good to Great about different "great" companies and their way of reducing their vision to a single "hedgehog concept," which has various useful boring attributes that I won't go into right now, as well as one more that I will: an optimization goal, which they call the "key denominator." Basically, to be maximally financially successful, a company needs to optimize its operations for the best ratio of profit to something.

Some companies use things like "most profit per store" or "most profit per product line." They talked about a bank in the U.S., which optimized its operations for "most profit per employee" when banks became deregulated. The most obvious way to do this is to reduce the number of employees and/or otherwise chop your HR costs, but in a world where banks are suddenly not protected monopolies anymore, that might be the only thing you can do.

That particular optimization goal got me thinking. I don't really like big companies very much. Once they get too big, they get annoying because it's hard to keep track of everyone and there are always some stupid people. So optimizing a new company in terms of profit per employee - which isn't nearly so painful in a company with no initial employees as it is when you'd have to lay off a bunch of people - might be quite entertaining.

And then I thought, wait, that's not quite it. You can increase profit per employee by simply making your employees work longer hours and paying them less, which isn't really what I want. Okay, so I work really long hours, but that's just me, and it's still sort of cheating; it makes me look more productive but I'm only producing more, not producing more per hour. If I was really smart, I would be able to produce more per hour than the next guy. If I also work more hours, well, great, but that's just a bonus.

And so that's it. We want the smartest, most productive people around, and the people we get, we want to make even smarter and more productive. So why not maximize profit per man-hour? Why not do it all the way across the company, and why not help your customers do it too?

In other words, laziness, impatience, and hubris. It's not just for programmers anymore!

138 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!