Older blog entries for apenwarr (starting at number 152)

The End Result

Today (Sept 15) I was reminded several times that simple engineering is not exactly the meaning of life. First, an extremely opinionated salesperson at a clothing store offered me some excellent fashion advice (which I've already taken, and it helps!). Secondly, I went and saw a random concert by NEeMA and Paula Toledo, both of whom are (for now) relatively unknown Canadian artists.

There's a difference between art and business, and you can think of it like this: business has a customer. Without a customer, you can't usefully do business. But art is different. It really exists as an end in itself. Of course, it's not that great an end if nobody is there to appreciate it, but maybe the artist him/herself is the only person to appreciate it. If so, that's enough.

I'm not much of an artist, and I'm okay with that. I still think back to Deniz's comments about me in my peer evaluation. I exist primarily to step in, help other people do whatever they're doing a bit more efficiently, and step out again. A person who does that can't really exist in a vacuum; the work I do is pointless by itself.

For me, what I need to do is find someone with some good goals, and help them achieve those goals. I think many people, especially engineers, fall into that category, if you really think about it. Of course there's nothing wrong with that, but beware of trying to accomplish things without understanding why you need to. Maybe, after all, there's no point.

Kidney Vacation

Those who know me know that I don't really like relying on others for help. I don't mind people being helpful, of course, but relying on people has always been uncomfortable for me.

Still, it's really nice to know that there are so many people I can rely on when I need to. Thanks to everyone (really, everyone!) for all your support last week. I'm feeling much better now and there's no longer anything to worry about.

Just one lousy hard boiled egg??

Being away from my computer while I was in the hospital, I had a lot of time to think and no place to publish it right away. So I wrote a few entries into my paper notebook, which I'll now proceed to publish here over the next little while.

They were pumping me full of some pretty cool drugs and filtering my bloodstream and not feeding me very much, so the content is perhaps a bit different than usual. I'll try not to filter it too much, and I'll call it my "delirium series." So without further ado...

Delirium 0: The Mind in the Machine

... Ooh. Uh, that one doesn't even make sense. Maybe I'll save it for later :)

Tune in later for the next exciting episode!

Migration

With rumours that advogato.org will be read-only soon, readers of my lovely drivel may be interested to know that I'm planning to migrate my site to alumnit.ca/~apenwarr instead. Future postings will appear there (and if I can arrange it and advogato sticks around, here too, I guess).

Curses Now Sucks Less

I wrote a long time ago about the lack of liberal input handling in curses. Well, I added it. It was even easier than I expected.

There was one part that was worse than I expected: not every program uses the input-parsing library from curses, so not all of them will benefit from this change. Anything that parses its own input (which isn't really useful nowadays, as far as I can see, although admittedly non-ncurses versions of curses do suck more) will have to be patched to handle the additional keybindings. The good news is that the patches will all be about as short as this one.

Digression on Really Bad Brain Scheduling Algorithms

real	916d6h56m
user	6h21m
sys	32m
16 Aug 2006 (updated 11 Oct 2006 at 04:24 UTC) »
The Unmaker

In his Alvin Maker series, Orson Scott Card creates a character named Alvin. Alvin is a "maker": essentially, someone with a super power that lets him create and change things with his mind. As a classical protagonist, Alvin naturally only uses his powers for good, and because Card keeps his symbolism very nearly under control, it is only slightly too obvious that this is meant to symbolize the ability of everyone to create and improve things in nature. Only more so for Alvin, of course, because after all, it's a super power.

Every superhero needs a super villain, and Alvin's nemesis is the "unmaker," a mysterious force that seems intent on undoing humanity's creations and most especially, because super villains do tend to be a bit obsessive about heros, Alvin himself.

The other day, someone said something that worried me. They said, "Nothing you can do will last forever." Now, while I'm sure this is objectively true, it does seem a bit defeatist to me. Why shouldn't you try to make things last at least as long as you can?

Well, why don't they last in the first place? In the physical world, the answer is fairly clear, and we engineers call it "normal wear and tear." Dust, sunlight, rain, wind, army ants, the annoying kid next door, and other forces conspire to undo everything that you build, and all you can do is build things that last longer than others.

But the physical world is not particularly of interest to me. These are modern times, and I'm a programmer, and I deal with abstractions. And abstractions like software can't rot, can they?

Apparently they can, but for an interesting reason: abstractions can't change, but their relevance to the universe can. Mathematics is "pure" because once you've proven something is true, then it will be true forever... in the appropriate context. When you try to apply a concept, your context is the real, physical world. And the real world keeps on changing.

So to a large extent, "the unmaker" of your abstractions is just other people making things. And the way to make your abstractions last longer is to make ones that fit better, not worse, with other people's future creations. In other words, you need a really, really good design.

But I'm pretty sure that nowadays, if you have that good design, you really can make something that will last forever.

Taking The Hard Way Out

One thing that has bothered me for a long time was a comment in a lecture by famed Waterloo economics professor Larry Smith. To paraphrase, he said there are a lot of big problem areas in business that need solving that are just ignored. Meanwhile, everyone in the computer industry seems to have a fatal attraction a particular other set of problems, which they solve over and over again.

The example he gave at the time was Windows vs. Linux: who would want to start a company to compete with Microsoft in this day and age? Even if you did manage to win (or even to not die), it would just be so much work! And no, the irony was not lost on me at that point in my entrepreneurial career.

My answer to his question at the time? Well, some projects are just more interesting than others, and I want my career to be interesting. It was abundantly clear to me even then that going into, say, banking software was the way to make loads of money, because the customers are rich and (it was a y2k madness at the time) boy, do they ever have lots of easy problems to solve. But banking software was by no means interesting. So I elected to take the path more travelled, as it were, and go for technical challenge instead of easy megaprofits. In other words, a compromise. I even formed a little continuum in my mind of different kinds of programming projects, in which the most technically challenging ones (hardest I could think of: video games) are the least profitable for the developer. And vice versa: banking software has got to be some of the least technically difficult software in the universe, and attracts some of the very worst programmers, who then get paid a lot to work reasonable hours with very low stress.

But the fact is that I'd be more than pleased to increase my stress levels a bit in exchange for an interesting project. That much isn't a compromise; in fact, too little stress is downright boring. That's why the best developers gravitate toward harder projects instead of easier ones. Video game developers know perfectly well what they're getting into before they start. They're buying into a whole lifestyle.

Well, good for them. I wasn't quite willing to go as far as the insane working conditions in the gaming industry, but I still wanted challenge, and operating systems seemed fun. There was stress, but that's not a compromise, because I wanted stress.

The compromise was the profits. Say what you like, and I do dearly love the free software community, but making massive quantities of money is a great way to make an already-fun company even more fun.

So what's the non-compromise solution in this case? Well, the form of the solution is obvious enough: it must be highly profitable but also challenging and difficult so that it can be fun.

It's pretty hard to take something that's challenging and difficult and not very profitable, and then make it highly profitable. Profits don't just magically appear just because you did something that was difficult. If it was the wrong difficult project, you're still out of luck.

But what about taking something profitable and making it difficult? Why, you can take just about anything and make it difficult if you try hard enough!

Of course, it would be best if the difficulty were actually warranted in some way, for example, to make the product even more profitable, or to also make the world a better place, or whatever. But that's actually only a secondary requirement. Look how much fun we had revamping our knowledgebase, for example.

So what if I could find a useful way to make, say, banking software challenging and fun?

On Wishful Thinking

wlach commented on my INTJ posting to mention that this sort of thing is often just telling you what you want to hear, so of course it sounds believable. Certainly there are plenty of "personality tests" and other things out there (astrology, anyone?) that do this on purpose in order to manipulate you, but the Myers-Briggs and Keirsey personality descriptions are different. They're just too specific to be able to fall for this problem.

Especially in the case of INTJ, sure, it says a lot of nice stuff, but it says a lot of specific stuff that not only fails to apply to almost everyone, but also that most people would find violently opposed to their self-image. "This self-confidence, sometimes mistaken for arrogance by the less-decisive..." and "INTJs have also been known to take it upon themselves to implement critical decisions without consulting their supervisors or co-workers." These are simply not generic statements. They're not true of almost everybody.

Moreover, all the other personality type descriptions I read, which all try to say as many nice things as possible ("accentuate the positive," you could say), simply don't match me nearly as well. In other words, a highly accurate personality description was attached to me automatically by a 76-question multiple choice quiz. That's pretty cool.

But I already know a lot about myself. The really interesting thing is slotting other people into their own types and realizing why you've had trouble dealing with them, and then what you can do about it now that you know.

Hiring People

Peter Norvig at Google writes about hiring only people who are above the average for your company, a strategy I first heard about from my friend at Amazon.com. Unlike Norvig, I didn't do a simulation to see if it made sense, but it did make intuitive sense to me at the time.

They also talk about the "no hiring manager" concept. NITI used to do this sort of subconsciously, but has been drifting away from it lately (letting individual project managers do their own hiring) with exactly the results that Norvig predicts. I didn't do a graph for that either (and indeed, the graph for that one isn't very convincing; the results are too close), and it also didn't seem intuitively obvious, but having seen it for myself, I agree.

Incidentally, the reason we had for doing the "no hiring manager" thing originally is that I didn't believe that project management skills and interview skills were particularly correlated. For example, I never felt that I was a very good interviewer, so I found people who were good at it and had them do it instead. At the very least, random team members are probably better at interviewing and hiring than project managers - as the people who actually do that kind of work, they know exactly what skills the person will need to have!

Okay, so score two more points for Google, I guess.

INTJ

After some recent discussions with various people, I started looking again into the Keirsey personality profiles (related to the Myers-Briggs "MBTI" tests). Well, I just have to say that those things are a lot more statistically valid than all those memes people keep posting.

In fact, reading the results, I almost feel like I'm just acting out a script that someone laid out for me at birth. This INTJ description just spooked me out completely by its accuracy. And then another one says very interesting things about my style of working, such as, "INTJs are natural leaders, although they usually choose to remain in the background until they see a real need to take over the lead," and "Unlike the INTP, they do not follow an idea as far as they possibly can, seeking only to understand it fully - INTJs are driven to come to conclusions about ideas." Hear that, pphaneuf?

Meanwhile, reading some of the other descriptions has helped me explain why certain other people I've tried to deal with just seem to keep doing things that sound totally insane to me; I guess it's okay, since they're normal and I'm not. INTJ is the least common of all 16 types.

Make Your Own Luck

The main problem I've had in dealing with people, when I've had problems, is what I think comes to the "intuition" vs. "sensing" communication problems (eg. INTJ vs. ISTJ personalities). That is, intuition people tend to jump rapidly from the problem to the solution, skipping all the intermediate steps. Sensing people take things one step at a time, because they don't really see the solution at first, just a process that will eventually lead them to one. Remember that stuff about process-driven vs. results-driven people? Well, there it is again, stated another way.

Both "N" and "S" people can fail badly if they don't realize their weaknesses. I think I've seen both types of failures. When an "N" person fails, it's because they see what the solution has to look like, but they don't bother even figuring out the steps to get there. For example, it's intuitively obvious that the final solution to our modern intellectual property concerns will have to be a new system in which copying most kinds of stuff can be done freely; any other system will simply be too wildly unstable to work. But how will we get from here to there? And what will it look like when we get there? That's something you need the "S" people to deal with. Except that most of the "S" people won't believe you when you say that's the way things have to turn out. They simply can't see that far ahead. And "N" people who can't explain it to them are actually worse than useless, because they're just confusing.

So that's the problem pure "S" people run into: they just can't see too far ahead. And not only that, but sometimes they don't even realize that they can't see too far ahead.

For these "sensing" people, just following logically from one step to the next will always give good results; if they don't, well, you didn't follow the logical steps well enough. Or if you don't follow the logical steps and it works out anyway, maybe you were just lucky.

But it's not usually luck. In fact, it's probably an intuitionist who laid things out in just the right way that doing the logical next steps, one after another, got you where they wanted you to be.

And that's just spooky.

Salary Survey

On a completely unrelated note, I found a really interesting New York salary survey. It's from 2000 right after the bubble burst, but I'm sure salaries haven't changed that much since then. I have two things to say about the results:

1. Wow, talk about supply and demand.

2. It appears that salaries are just all over the map, so the only definition of "fair" is "whatever you can get away with." Sigh. Obviously not something the INTJ in me wanted to hear :)

REaction Engine

I've realized that proactivity is just reactivity with negative latency.

What? Okay, first of all, "be proactive" is sometimes known as common management blabber, and being a one-time manager myself, I'm allowed to use it. But used correctly, "proactive" is actually a useful word. It means solving problems before they happen.

And reactivity is solving problems after they happen. Aha. So proactivity is just reactivity where the effect happens before the cause: negative latency.

Easy, right? Well, no, since because some something or other with the space-time continuum you can't actually do that. But negative latency has been successfully employed in the past.

Here's what I'm thinking. If you operate in a normal "reactive" mode, your simple goal is to react as quickly and accurately as possible. Having given up on preventing problems, you want them to at least go away, if at all possible, before too many people notice. This is a good way to run a startup company, where you can't possibly expect to prevent all problems at first; there are just too many of them to start with.

The better you get at being reactive, the faster you can solve the problem, and latency approaches zero.

Imagine if your boat hits an iceberg and gets a hole in it. You want to start bailing water as soon as possible. If you notice that the hole is too big for that to work, you'll need to start evacuating the ship, and so on. But the better you react, the fewer people who drown.

As latency approaches zero, the amount of work increases exponentially; latency can never really reach zero using a reactive method, but it can come infinitely close with an infinite amount of work. This is a singularity.

Now what about that negative latency? Well, it requires some different techniques, and creative thinking, versus positive latency. Since you can't run time backwards, you have to instead be living in the future, reacting now to what you expect to happen, say, 10 minutes from now. Living in the future is impossible, of course, but you can try your best to predict what it will be like. And predictions are about probability. This is the only real difference between being reactive and being proactive.

Many people don't realize this, and think of the two concepts as requiring a total change in technique: "proactively prevent all problems." This leads people to overanalyze the situation, taking only extremely safe, risk-free approaches that will never have problems - but will also be the slowest possible way to get there. This is sort of like sending all the bytes from all the files just in case the TFTP client might want them someday.

In fact, just like the current problems that you react to, each future problem has a cost associated with its occurrence, and increasing costs associated with not solving the problem once it occurs. Reactivity helps you avoid that increasing cost; proactivity removes the fixed initial cost. But while reactivity has an easy-to-calculate cost-to-benefit ratio (you know the cost of not solving the problem, you know the problem has occurred, and you know how much it'll cost you to solve it), proactivity has an extra element to it: risk. Some potential problems, even though they might occur, won't. For those ones, avoiding them is purely a cost and has no benefit.

So as with all negative latency problems, probability and intelligence are key. But you can outdo anybody who is merely reactive or naively proactive, just by thinking harder and more carefully than they do.

143 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!