Older blog entries for apenwarr (starting at number 55)

Death by Abstraction

I put in some work this weekend trying to make UniConf maximally palatable to the KDE project. Since UniConf does pretty much everything that can be done, or else can rather easily be extended to do so, you might think this would be an easy job. Not exactly, because of one tricky bit: performance.

UniConf uses a rather neat abstraction, introduced to UniConf by Jeff Brown a couple of years ago, called a "generator." It's one minor step up from a filesystem in a Unix VFS. Because of the slightly more/less rigid (depending how you look at it) rules a generator has to follow, UniConf makes it easier to stack layers of generators together to achieve what you want, and so people do. For example, there's an ini-file generator, a tcp client-server generator, a D-BUS cilent-server generator, at least two kinds of generic caching generators, a list-of-generators generator, and so on. As of yesterday, there's also an automount generator (take that, autofs!) and a filesystem one-file-per-key generator, the two of which work together (with the ini-file generator) in a slightly complicated way to handle reading/writing your ~/.kde/share/config directory.

The power of the UniConf abstraction is that you can easily rearrange these individual parts to do something similar but different: for example, if you wrote an xml-file generator, you could combine it with the same one-file-per-key and automounter generators to produce something that can read/write your ~/.gconf directory instead. That's a lot less work than writing a tree-of-xml-files configuration system from scratch, and you benefit from other people having debugged the other generators you're using.

But in a system this general, performance can be a problem. In my initial simple-minded experiment, switching to UniConf took the start time of "kwrite" from 1.0s to 25s. Okay, so that was a bit extreme, and with minor tweaking I got it back down to 3.0s. But still: that's 2/3 of kwrite's startup time just fiddling with config values. Obviously not okay.

Now, what's amazing about Jeff's abstraction is that it's so complete - the slowness is really not the fault of the abstraction layer at all, because the way it works, the front end matches up almost exactly with the backend; that is, the API you call is pretty much the same as the API the generators each implement. So you could write a *very* fast generator if you wanted - optimally fast, because you can throw away all the layering and just implement the API calls directly. It's just that, because it's so easy to combine pre-existing modules instead of writing something from scratch, most UniConf setups end up being pretty badly non-optimal.

Here's the real killer example: profiling my UniConf-enabled kwrite shows that much of the time is actually spent converting UniConf keys to and from KDE QEntryMap objects. (KConfig keeps a QMap of all the groups/entries in the tree.) To make it "optimally" fast, I should really store all my config entries directly in a QMap, rather than storing them in a UniConf-style data structure and then copying them to a QMap. But there's clearly no way to do that outside of rewriting the pre-existing UniConf generators.

Since writing a completely special "KDE generator" wouldn't be "unified" enough for me, I'm going to instead try to optimize the other generators more until the performance is acceptable. The profiler shows some pretty obvious starting points. Flexible and fast - not the easy approach, but maybe the most fun. Wish me luck.

Confident statement #1:

"I know my place in the world."

Even more confident, but contradictory, statement #2:

"I make my place in the world."

Conundrum

How will you get from #1 to #2?

8 May 2005 (updated 8 May 2005 at 00:57 UTC) »
Canada and Copyright

There's been quite some discussion lately about Canada's proposed amendments to copyright law, and I'm afraid most people are completely misunderstanding the situation. What you have to realize is how incredibly clever these amendments are. (You also have to realize that a copyright system intended for printing presses, vinyl records, and live CBC radio broadcasts does need an update for the day of the Internet, and in itself, that's a very good thing.)

To dispel some scary misconceptions:

  • The "notice and notice" system they'll impose on ISPs is incredibly lax compared to the draconian "notice and takedown" U.S. system. When an ISP is informed that their customer may have posted infringing material, they have to... inform the customer about it by forwarding them the complaint! Uh, I'm slightly stunned that we even needed a law for this. Oh, they also have to keep a copy of it around to avoid tampering with potential evidence in case a lawsuit follows. Okay, so ISPs should make backups of their systems occasionally. What's the problem?

  • "The alteration or removal of rights management information (RMI) embedded in copyright material, when done to further or conceal infringement, would constitute an infringement of copyright." Don't you realize what a great miracle of phrasing this is?? It's only illegal to remove the RMI stuff if it's done to conceal or further infringement. In other words, hiding your tracks after doing illegal stuff - still illegal! What amazing insight! But notice that what it doesn't do is change the definition of infringement. Removing RMI to make your own damn music play on your music player after you paid good money for it is not infringement. (At least, not based solely on this amendment, and I haven't heard of any other amendments that affect the situation.)

If you feel like writing to your MP about Canada's copyright improvements, please do - but congratulate them for doing the right thing, and encourage them not to later cave to U.S. pressure. The last thing you should do is accuse these people, who apparently actually do have your best interests at heart, of being exactly the kind of people they're trying hard not to be.

Disclaimer: IANAL, but chances are that neither are you. At least I read the bloody web site before posting.

Like Nature, Only Better

Today was an ideal spring day, and it reminded me why I like Montreal the way I do.

I had nothing better to do, so I went for a walk in Parc Mont Royal. The Sunday "Tam-Tams" session was in progress, which as usual made me think of self-organizing systems, but that's only tangentially relevant to my story. I found a little stream of water coming down the mountain - it's been raining a lot in the last few days - and decided, again, because there was nothing better to do, to find out where water comes from.

"Up" is the obvious answer. I traced the stream to a culvert and out (well, in) the other side. Then another, and finally to a hole where it seemed to disappear underground - well, appear from underground. Not to be stopped that easily, I followed the slope a surprisingly far distance uphill, and found at least two places where streams of about half the size as the original one went underground. There being only one of me, I decided to follow only one of them.

The slope was getting steeper as I got to the more "serious" parts of the mountain, and I noticed my stream - it was "my" stream by now - was shrinking. Looking back, I could see that a few little irrelevant trickles, feeding from every direction, joined my stream occasionally; as I passed each one, my stream, of course, reduced just a little bit each time.

Time passed, and I followed my stream to what eventually turned out to be its source - another tiny, irrelevant trickle, just like all the ones I had passed, coming from a small puddle in some wet grass. There was no big impressive lake draining, no mountain spring shooting water from a crack between two rocks... just another trickle, just like all the others.

I try not to post anything here unless it has a point. Perhaps you think my point will be something trite, like, "There's no ultimate source of power, and nobody is very strong all by himself; real power comes from a lot of little, irrelevant ones acting together." Yeesh. I hope you give me more credit than that.

I looked back down the hill toward where I had started. One thing I hadn't quite noticed on the way up struck me: that the walking path through the park seemed to stray back and forth rather near my little stream, without ever getting wet. In fact, the streambed wasn't entirely random; every so often, a culvert, or ditch, or cobblestone bed, or bridge, appeared, designed to keep the stream flowing smoothly out of the way of the path without allowing any flooding. Humans had been there before me, not just tracing the path of the stream, but adjusting it, controlling it, making it just slightly different so that it would work the way they want.

People are not the little, irrelevant trickles that combine together to become a mighty torrent; don't think of it that way. The irrelevant trickles and mighty torrents are just the forces of nature. People are the things that - with the right, tiny effort, applied in just the right places - make those torrents go just the way we want.

And this is why I like Montreal: other places I've been, like Toronto or Ottawa or New York or even Thunder Bay, give you the feeling of a great battle with nature that we won; to build New York, you can imagine that we gathered some trickles, created a big torrent, and blasted it at everything in sight until we built a monument to our power. There's nothing wrong with that - New York is very inspirational that way - but that's not how I feel in Montreal. What I feel here is that we handled things more like we handled that stream; we let nature run its course, and adjusted just a few things along the way. No excessive force; just a little tweak here, a little tweak there. The mentality here is different that way. I like it.

21 Apr 2005 (updated 21 Apr 2005 at 17:41 UTC) »
dcoombs wins

"I bet I'm going to be delicious!" the sheep said, a tad pre-sumptuously.

Deep Thoughts

Must come up with pun involving the word "pre-sumptuous." It's a tricky one.

Stupidest anti-correctness-proof argument I've ever read. But worth reading anyway, just for insanity value.
On Being Obviously Smart

Adrian: if people tell you you're smart, try ordering them around and see if they'll do what they're told. Works for me.

Safe kernel crash dumps

So over the last few days, I've spent altogether too much time messing with kmsgdump, a tool that takes your 'dmesg' output and dumps it to a floppy or the parallel port. Although there are other similar projects, like LKCD and kdump (sorry, no link), they make me nervous, because: HELLO, MY KERNEL JUST CRASHED! STOP TRYING TO USE IT FOR STUFF! You'd think this would be obvious, but LKCD actually has the nerve to go and run your disk driver, spewing interrupts and DMAing data all over the place. Sure, it seems to work fine... but that's because all the crashes I'm having right now are simulated ones. What happens if it's my disk driver having the problem, or the interrupt handler is dead, or... anything? Answer: uh, well, there's no documentation that even discusses this possibility.

kmsgdump is better, because at least it's written by paranoid people. In fact, it's so paranoid that it reboots into real mode and uses the BIOS to write to a floppy. Well, that's pretty safe, but unfortunately most Nitix systems don't have a floppy disk. So what do we do? Adapt it to write to the hard disk, of course. Yes, I realize I'm a hypocrite. :)

I did this as carefully as possible: it reads each sector from the disk into memory first and checks that the sector's signature is exactly right before it writes it back, so it's extremely unlikely that you'll manage to write the wrong sector; and in the end, the only danger you have to worry about is writing the wrong sector. We generate the target file in advance from Linux, lilo style. And it actually works!

I'll be contributing the changes back to rddunlap in case he wants them, but he currently (rather rightly) thinks I'm crazy. Meanwhile, if you think you need something like this, let me know and I can send you the patch.

And once upon a time I used to think that Linux never crashed...

Blue Man Group

At a suggestion from mich, I went to see Blue Man Group while I was visiting New York this week. They're really good. And, like I've heard from other people, awfully hard to explain.

The thing that struck me was how well done the whole thing was. The best acoustics I've ever heard anywhere. Really excellent synchronization of all sorts of people doing all sorts of things. (Synchronizing multiple people with that kind of sub-second accuracy really ought to be impossible.) Audience participation that actually works. Metaphysical ponderings. Streamers. Paintballs. Flying toblerone. And all executed amazingly well. So go see it.

I have no idea if they're the same as the Blue Men in Las Vegas, but I bet the acoustics are different there.

On Perfect Execution

All that makes me think.

For the last few years I've repressed my perfectionist streak, the idea being that it's better to successfully do lots of very good things than attempt to do a single thing perfectly and then fail. I think I can trace this back to first year university, when my marks changed from formerly being "let's see if we can get 100%" to "I sure hope I pass this exam." It's also the first year I ever heard of "diminishing returns," which is really the most fundamentally important concept in all of engineering. Computer Science people, and artists for that matter, don't hear about it much. It's a simple concept: if the last 10% of the work takes 90% of the effort, then you can often just skip doing that part. Your test scores, for example, will still be about 90%. And it really works, but it's not very fulfilling.

I tend to judge my work and other people's work on different scales; a kind of double standard. A lot of work that I mostly consider fine when done by someone else wouldn't really be acceptable to me if I did it. And since I'm delegating more and more work all the time, my overall happiness with the result tends to suffer. With years of effort, products get more and more complete and useful but less and less perfect.

People who know me know that I can be a bit harsh sometimes when I'm disappointed in some piece of work. The real problem, though, is how little time I have to criticize (constructively, one would hope) the many little details that really deserve it.

Are they pleasant to work with? Hard to say. It's like he doesn't have much invested in any particular personal transaction, so this feels like an irrelevant question.

Do they work well in a group? Not from what I've seen. He needs to work on his own, just stepping into other groups to give advice or contributions of handy tools, so forth; exactly what he does now.

-- from the Deniz archive

46 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!