Older blog entries for apenwarr (starting at number 21)

After reading a few semi-depressing books and articles about the way standard-economics-as-taught-in-school doesn't actually work (because almost all markets are inherently imperfect), and mostly agreeing with them, I realized that right now in Montreal I'm surrounded by examples of perfect free-market capitalism: restaurants.

Some background: some major reasons capitalist markets fail are incomparable goods and high cost of entry. For example, I can buy cotton from just about anybody with cotton, but CorelDraw just isn't the same as Adobe Illustrator. They're similar, but I can't only compare them on price; I have to decide which one I want more. Furthermore, some guy off the street can't just walk in, make a better graphics package, and sell it to you: he has to spend a lot of money (the "cost of entry") first. One big reason Microsoft can maintain their monopoly is that the cost of just getting started competing with them can (and does, regularly) put just about anyone out of business.

So basically, the software market is depressing, because I only took Economics 101 and it can't be explained by that. In my continuing attempts to simplify my model of the universe, however, I moved to Montreal, which brings me to my point: restaurants. Useless trivia: Montreal is second only to New York City in North America for "number of restaurants per capita", ie. the number of steps you have to walk down the street before stumbling into some food.

There are all imaginable prices of restaurant (and at varying quality, of course), and lots of each type, so they have to compete on price. You can buy as large or small a quantity of food as you want. You can start a cheap restaurant and quickly move up in the world, so the cost of entry is relatively low. And the market, in action, is amazing: just watch the consumers get sucked dry! No matter how much money you have, you can spend every last penny, no more, no less, on food, and still feel good about yourself, as this example shows. If you normally buy groceries, then eating out is a treat. If you normally eat at Burger King, then eating out anywhere else is a treat. If you normally eat at cheap diners, then eating at a nicer cafe/bistro is a treat. If you normally eat at nice bistros, then eating at a fancy restaurant is a treat. And if you always eat at fancy restaurants, well, Canada has a lovely progressive income tax system.

Yay, after all these years, my popularity-contest has made it to slashdot. Of course, most people seem to have assumed the code is totally dumb; they missed the fact that it only votes for packages that you actually *use*, not just the ones you install, so it's not inherently wrong. There are also lots of silly comments about how you shouldn't choose your packages based on popularity... well, maybe that's not how you should decide what to install (and maybe it is), but it makes perfect sense for the most popular packages to be on the first (and maybe only) disc!

I have to say, the people who took over popcon from me (since I'm lazy) do seem to have put a fair amount of good work into it. At least, they have cute graphs of the results.

Update 2004/03/29: Italicized "inherently." It just needed it.

I'm in Germany right now at Chemnitzer Linux-tag. Chemnitz is a town, and Linux-tag means "Linux day," which is slightly wrong because it's actually two days. The going theory for the misname is that the correct, pluralized name would be "Linux-tage", but we stupid anglophones would probably mispronounce "tage" much worse than we mispronounce "tag", so they leave it the way it is. Plus it might have been only one day, once upon a time.

Speaking of stupid anglophones, there are only two of us here (me and dcoombs, who is doing a talk on WvStreams), in a conference that some estimate has about 1800 German-speaking attendants. This is strange. In fact, I was getting really worried at first about the fact that we spent a lot of money to fly us here and neither of us understands, well, any of the talks. However, pphaneuf was totally right: the point of a technical conference isn't the presentations at all. It's the conversations you have with people outside of the presentations. And I've learned a lot by talking to people:

IPv6, Itanium, Dylan, Subversion, Non-sucky X Compression!

It turns out that IPv6 sucks much less than I thought, and Intel's Itanium sucks much more. Both of these revelations came to me from Matthias of symlink.ch, who showed me seamless, painless, easy IPv6-over-IPv4 and told me about his actual experiences programming efficient Itanium assembly language. So in both cases, he ought to know.

I also had an interesting discussion about the Dylan programming language. Tim Pritlove, a big proponent of the language, told me why it was theoretically so great, and I bounced him some tough theoretical criticisms (thanks, slajoie and pphaneuf, for providing me with most of these) that he mostly answered satisfactorily. Of course, nothing proves a language like actual real-world use, and, well... it doesn't have any. Still, very educational. I might try it out and see if the only reason it's not popular is that people like me don't try it out.

Other bits of wisdom: Subversion seems really good, except your whole repository is in one big, probably corrupted bdb database. Nomachine has now GPLed the important parts of their very excellent (I tried it months ago) NX X/VNC/RDP/etc protocol compressor. If you've tried other X protocol compressors before (eg. the worse-than-useless LBX), forget them; this is the real deal. And now you can get away without paying for it!

But most importantly of all:

German Bathroom Construction

German bathrooms are totally awesome. And I mean that not in the, "Yo, d00d, that awesome stuff totally r0x!" kind of way, but rather that when you experience one of these bathrooms, you are actually in awe. It's the kind of awe that makes you think, "You know, if me and this bathroom got in a fight, the bathroom would definitely win, big time."

In Canada, bathroom stalls simply don't lock properly. Don't get me wrong - they all have locks - but the locks, when they aren't actually broken, almost never line up with their sockets. This can't be just shoddy construction - it's shoddy construction combined with some kind of fundamental misdesign.

All German bathrooms, on the other hand (and let's be frank: I've tried a lot of them now, so I oughta know) are built like tanks. Not coincidentally, the Germans also invented tanks. The doors are super-solid, always line up perfectly, and have double locks. "No, I don't think that's locked enough, let's turn it one more time so it's extra locked!" And you're just not going to get out until you unlock - twice. I know, because I'm locked in my hotel bathroom as I write this.

In fact, all German construction seems to be this solid. You get the distinct feeling that anything built by a German is simply not ever going to fall down. And then you realize that we - yes, we, the people with the vastly inferior bathrooms - were actually at war with these people, twice, and somehow they lost. It's hard to believe.

So, rather than doing real work, I was talking to pphaneuf at work about RSS and NNTP the other day after reading Simon Law's comments. pphaneuf convinced me that Simon was essentially correct; RSS is nothing new, and most RSS "aggregators" are pretty sucky, and usenet newsreader technology is mature, established, and generally pretty good, so why not just make an RSS-to-news gateway and read your newsfeeds that way? Great!

So I did. Well, that was easy. Add noffle, 52 lines of perl (yes, I did do serious XML "parsing" in there using some really dumb regexes - what's it to you?), 15 lines of shell, and a few automated wgets, and there we have it - instant RSS-to-news gateway. (Anyone wanting my crappy script is welcome to it - just email me.)

The next step is to go track down one of those mature, excellent, efficient, user-friendly newsreaders and enter RSS heaven. Right? Right? But wait! I forgot! Newsreaders suck too! In the last hour or so, I've tried knode, gnus, mozilla-mailnews, pan, slrn, a horrible mutt-nntp hack, Outlook Express, and manually telnetting to the NNRP port. I have to say, the telnet is looking pretty good right now.

The all-time winner for worst newsreader UI ever is of course gnus, but mozilla-mailnews comes a close second. At least the others make some attempt at making sense; mozilla just kept on trying to connect to a host named 'news', which I didn't configure anywhere. slrn actually worked like I wanted, except it couldn't display html (which all blogs are written in, of course). Outlook Express, while annoying, was the sanest one of all, and it even has a cool "offline" mode that I don't need. Could someone just go and clone Outlook Express News for Unix, please? Hint: when I go to news://hostname, why not pop up a list of that server's newsgroups, and stop asking stupid questions about my "identity"?

Anyway, I think this finally explains to me the massive growth of so-called "RSS aggregator" apps compared to newsreaders lately. It's very simple: all newsreaders, like all RSS aggregators, are horrendous, and therefore people keep rewriting them. Maybe someday they'll get it right.

For my next adventure, perhaps I'll try an RSS-to-IMAP gateway. At least I know some reasonable mail clients exist.

More Curses

I realized that my ESC-is-a-key-and-a-sequence problem can be solved simply by making my own ESC key send something other than ASCII 27; after all, if I'm doing the "liberal" thing with input, I can have both ASCII 27 and my new ESC be treated as KEY_ESC. The 27 has the stupid delay, and my new key won't. Essentially no apps other than curses ones use the ESC key as input anyway, so this wouldn't be so bad.

As for people with their DEL key mapped to 127, well, they can just suffer with having their DEL key act like backspace. They probably won't notice anyway.

Okay, so I finally did it. As part of a project I was working on at work for the last few days, I decided to ignore curses entirely (for various reasons, most of them bad; leave me alone), and in the process, the library I wrote solved two problems:

- regardless of your TERM setting, it displays correctly (and I mean perfectly, modulo the lack of colour in win9x telnet) in the Linux console, xterm, rxvt, putty, minicom, Win2k telnet.exe, and - yes, really! - in Win9x telnet.exe.

- regardless of your TERM setting, in all of the above programs, my HOME, END, PGUP, PGDN, and INSERT keys work as they should (except for programs which happy refuse to send those codes at all - notably the Win2k/Win9x telnet programs).

How did I do it? I did what curses and ncurses never did. I followed the first and most important law of successful communication, attributed to Jon Postel: Be liberal in what you accept, and conservative in what you send.

My program sends only the most basic VT100 codes: gotoxy, change colours, change-to-wacko-line-drawing-font. We could be fancier, but then my output wouldn't work everywhere. Be conservative!

On the other hand, it accepts *any* of several possible codes for HOME, END, etc. Every stupid bloody terminal does it differently, and I don't care; I'll take them all. Nobody who says ESC[7~ doesn't mean HOME, even if not everybody who means HOME says ESC[7~. Be liberal!

Of course, I don't support non-basically-vt100-compatible terminals. Now first of all, I don't care, because (hello, join the 1980's!) there aren't any. Secondly, nothing stops curses from doing my basic-vt100 thing by default, and different things if you *do* set your TERM specifically. You're the weirdo, you go suffer. Unfortunately, curses stupidly tries to do something optimal by default. Well, this is to cut down on wasted output, you'll say. Remember 2400 baud users, you'll say. I want my email reader app to be legible in this crappy terminal emulator without spending three hours trying to guess which of the 5 million 'xterm-*' terminfo settings is the right one! ARGH!, I'll say. This isn't so hard. Be conservative by default, and if I'm a weirdo with a 2400 baud modem, I can set TERM to something more efficient. Easy. The "output conservativeness" problem is only a fault of the people who write terminfo databases, so technically we won't blame curses for that.

Unfortunately, for input, curses made a fatal mistake: the terminfo format *itself* has a one-to-one mapping between escape sequences and input codes. There cannot be more than one HOME. That means, basically, there is no way to "be liberal in what you accept". This is a fundamental design flaw in the terminfo file format, and AFAIK you can't fix it in a backwards-compatible way. But you can still fix ncurses. I'd be more than happy if someone would just finally do so.

There are two flaws, however, that I haven't solved: the ridiculous "ESC is a key and also an automatic sequence", bug, that means pressing ESC to cancel a dialog is essentially never going to work right. And there's the ridiculous "nobody knows what code backspace is" bug, that originally (ie. in a VT10x/VT220) was never a problem, but eventually someone (I think it was the X Consortium) mangled completely by sending ASCII 127 for the keypad DEL key. There's no saving people with that keyboard mapping (backspace->8, DEL->127), and unfortunately there are a lot of those people. But I can save everyone else, because CTRL-H is backspace (shut up, emacs users), 127 is backspace, and several things like ESC[3~ are DEL.

There. I'm glad I got that off my chest. (I think I followed pphaneuf's rules for flaming because I went and implemented something better before I flamed the crappy library we poor losers have been suffering with for decades.)

I've finally half-joined the ranks of the "elite" who seem to all not only have cameras, but also now have digital cameras that cost twice as much. I say "half-joined" because I actually saved myself about 33% of the overall cost by skipping the "normal" camera phase entirely, buying only the overpriced digital kind.

dcoombs is doing a fine job of posting lots of culture-inducing photos in his NitLog, so I won't do that. I just bring it up because I finally figured out, after being pressured into buying one, what's so great about digital cameras: instant gratification. Perversely, it's the same reason people still buy books and CDs at stores instead of online: because you can get your book or CD now instead of waiting for it to arrive. Similarly, with a digital camera, you can have your photo now instead of waiting for it to be developed.

Moreover, you can learn a lot faster. In normal photography, you would have to try a lot of experiments at once, go get them developed, learn where you screwed up, try again, develop them again, and so on, usually with (at least!) a day's lapse in between. It's like Rapid Prototyping for wannabe photographers!

Cole Slaw and One-size-fits-all

The little-known-outside-Quebec-yet-popular chicken roasting chain St-Hubert (now with Business Class!) is the only restaurant I've ever been to that offers both creamy-based and vinegary-based cole slaw. The problem with cole slaw is that there are these two kinds, and for each kind, something approaching 50% of people like that kind and detest the other. (Actually, there's a third kind, KFC radioactive-green-based, but nobody at all seems to like that kind.)

Anyway, most restaurants serve only one kind of cole slaw or the other - people are so sure that their preferred kind of cole slaw is the best that they only serve that one kind, and they don't even label on the menu which kind that is. And yet, half the restaurants in the world still serve the other kind, so you'd think they'd notice. As it is, you have to ask the waiter before ordering coleslaw, "Is it the creamy kind?" and order or not order based on that... or, like most people, simply don't order it at all.

At St-Hubert, unlimited cole slaw is included in *every* meal for free - and they offer both kinds, and people like it and eat it. The only way to have people eat cole slaw was to offer both options.

In this sentence, I was going to tie all that into software development and one-size-fits-all user interfaces and explain how sometimes taking away an option that's "the same for everyone anyway" might be a bad thing, but I think you can probably see where I was going, so I guess there's no need to insult your intelligence. If you have any.

pphaneuf pointed me at a comparison of successful/unsuccessful technologies based on various attributes. The supposed winning method, although the survey is grossly statistically invalid, was that projects developed using the 80/20 rule are generally more successful (Tim Bray).

This reminded me of some other discussion of the 80/20 rule, notably Bloatware and the 80/20 Myth (Joel Spolsky) which is (from the title) obviously of the opposite opinion. And there's a related opinion regarding Java and its flood of APIs.

So that got me thinking: how valid is the 80/20 rule? When NITI started, it was just a small number of people, even fewer developers, and a brilliant (IMHO) idea - and we built and sold a few boxes, but we never suddenly started swimming in money. Ever since then, we've been adding developers and doing more work, but less and less of the work has been brilliant ideas. More and more of it has been normal, day-to-day stuff. And with each release, incorporating more normal, day-to-day stuff, we're able to sell more and more software; exponentially more. The 80/20 rule got us started, but it's the 99% rule (soon to become the 99.9% rule, if we get too popular) that gets you big.

This actually explains a few things. OpenSource is described, in Tim Bray's first article above, as tending to "just copy what works", regardless of whether it's very 80/20 or not. But this isn't the whole story. I think OpenSource is "bloatware", as described in the second article, from Joel Spolsky. It followed 80/20 (Tim Bray's version) to get started, but now anybody can extend it to get whatever they want - and people do. Linux now gets used everywhere you can imagine, precisely because someone was able to easily change it to do what they wanted, not because it really did a lot of stuff or was particularly good at something else.

The alternative approach is something like Windows, which isn't so easily extended (it's not that hard) but is already near-complete - it can do more stuff out-of-the-box than Linux can, and so it's popular for different reasons. Unfortunately, the cost of making something as feature-complete as Windows is exponentially more expensive than the cost of making something as extensible as Linux; I suppose that means Windows will be the one to fail, eventually. (It also means that the GPL was the magic ingredient that turned Unix from a failure into a success; and that the GPL has and would have nothing to do with the present/future success of Windows, since Windows' success doesn't come from its extensibility).

Do I have a point? Well, how about this: both Tim Bray and Joel are right, but they read different things into the 80/20 rule. Tim Bray says 80/20 is about starting simple and selling it before you're done - also known as Worse is Better. Joel is opposed to limiting your software to the necessary 80% in order to keep it simple and elegant, and then expecting to be wildly successful. It just doesn't happen. You start off mildly successful using 80/20, then you throw 80/20 aside and get serious... or you at least open source your thingy and use a really good plugin API so that people can fill in the other 19.9% of usefulness for you by doing the remaining 99% of the work.

I just discovered unfs3, a userspace implementation of NFSv3. Contrast with FunFS, a totally new filesystem, also in userspace, that I helped (a little bit) to design.

unfs3 is pretty good, but it will probably never be as fast as a kernelspace nfs server; FunFS, on the other hand, is already faster than any NFS, simply because the protocol is optimized for minimum latency and *real* caching, not cheeseball NFS-style caching.

On the other hand, the unfs3 server is already fully functional with a 54k binary; FunFS is still not fully functional, and it's much bigger (if you count its WvStreams dependency). Statelessness (ie. NFS) obviously gives several advantages, simplicity being a major one, but I wonder if these advantages will be worth the compromises (ie. stupid in-kernel servers vs. bad performance) we have to make?

Wow, has it really been that long since I posted something here? I guess I'm either lazy or overworked. (In fact, those two things are tightly related. Or the same?)

Branch Constraints Revisited

If you're looking for the actual article, you can find Branch Constraint Theory on Advogato.

It's been about 8 months since I posted the original paper and started implementing my own advice. So, how did it go? I've learned a few more things since then, but the advice turns out to be correct - math is a wonderful thing - and understanding it has led to some definite improvements in our inter-release time and bug counts. One missing detail: I didn't originally consider inverted CVS branches, where you create unstable branches and only merge into the release branch when the code stabilizes. This complicates the math considerably, but can allow shorter release cycles, especially for "big" features. There's more for me to say eventually...

Magazine Reviews

NITI products get lots of positive reviews in the press, but it seems that every computing product does. I used to think that this is because, as "everyone knows", you simply pay off a magazine and they give you a good review. Therefore I was suspicious when they - universally - not only didn't ask us for money before reviewing our products, but also gave us extremely positive reviews despite our lack of kickbacks or advertising. Someone finally explained it to me a few months ago: computer magazines depend on getting free samples of products in order to review them. If they got a reputation for putting out bad reviews, nobody would send them stuff to review. Only positive reviewers survive. So all reviews - even comparative ones - only say positive things. It's just that some are less positive, and some are more positive. If your product is really seriously sucky, you'll maybe get a neutral review (facts only), but you have to be really, amazingly bad.

Then I talked to a musician on the plane from Montreal to Thunder Bay, where I'm visiting for the Christmas holidays. He reminded me that in music, it's not the same at all: there are lots of music reviews that totally trash the music in question. And then he told me why that is: while in computers, a negative review could destroy you, in music, the worst possible thing that could ever happen to you is a "lukewarm" review. Art is about stirring up people's emotions, and, love it or hate it, as long as you care, they've been successful. So the hardest music review to find should be the lukewarm one.

Architecture, Implementation, and Glue

Speaking of plane-musician conversations, we also had some discussion about electronic music and the tendency to endlessly remix things, to the point where it's now an entire art form to simply mix other people's music together well. In the extreme other direction, I saw a concert by Kalmunity - Live Organic Improv in Montreal a few weeks ago. They generate their music in real-time, using only non-electronic instruments, and they're very good at it. Somewhere in between is a "traditional" musician that composes his own music and then assembles a band to play it with.

So programming is an art form, right? I see some parallels here: the above three types of musician correspond to gluers, virtuoso coders, and architects, respectively. I've been thinking of this a lot, because I'm trying to find a better way of organizing software development teams. Most effort in this area concentrates on finding a great architect (composer) and then combining him/her with a great team of virtuoso coders. If you can do that, you're doing really well. But open source, like electronic mixing, makes possible a new kind of developer that didn't exist before: the gluer. This person is an architect, in a way, but the architecture is imposed after the individual components have been built. The open source world doesn't have a lot of great architects or gluers. It seems to me that if you want to maximize your success with open source, the best way to do it is to become an excellent gluer yourself - you can become famous for assembling other people's work and making a final product. KDE is more famous than KMail.

Correlation, Continued

I've been playing more with document correlation in the last couple of days, this time trying to subdivide documents into groups. This is based on the vague feeling that our tech support call logs probably have some kind of trends, and if we could just identify the "most common" kinds of calls automatically, we could focus more effort on improving the product to get rid of that kind of support call. Auto-categories are kind of screwy, though, because they might just as easily group documents by support technician's name instead of by problem type - and that's obviously not what I want in this case, but I have no way to express that. (And don't even talk to me about metadata! I would have an entire rant about metadata, if anyone ever bothered to tell me what it was!)

According to http://net-itech.com/america/news/story_00025.htm, I said this about Guadec:

"Net Integration produces and contributes to several Open Source projects in the process of making our Net Integrator and other products, and we're happy to be able to give back to the community by assisting at GU4DEC," says Avery Pennarun, Vice President Development and Senior Systems Architect, Net Integration Technologies. "UniConf, our advanced object-relational web-enabled remote management console with cross-platform transaction-oriented XML content delivery, will totally change the way people do business and revolutionize the world of Open Source."

So here's the bad news: I did, in fact, say that.

And the less bad news: I was kidding when I said it, I swear!

I honestly expected the people in marketing to see my joke, especially that last part about "totally change the way people do business" and "revolutionize the world of Open Source." Unfortunately, it's now a press release linked off our front page. I think I have just learned some kind of important life lesson.

I'll let you know when I figure out what it is.