Older blog entries for tromey (starting at number 20)

Today I finally put Advo on my personal toolbar in Mozilla. I find I really enjoy reading the Advo diary entries, and I look forward to them. I think it is because some people put real thought into their entries, so they can be good reading. Also it is interesting to see what other people are working on or thinking about.

Today I did something I ordinarily dislike, which was rewiring so that I could boot the alpha box I have here. Honestly I think Red Hat forgot they ever sent it to me. It's nice for testing gcc; today I'm using it to test a gcj patch for a bug that occurs only on 64-bit architectures. What I really need to do is move that box plus the other old one to the basement and run wiring down there. Then the noise won't bug me so much. Given my general level of hardware procrastination, I think it is safe to say that will never happen.

Last night I did some hacking on a build tool prototype. Basically I'm trying to test some of my plans against reality; earlier hacking in this vein led me to revise how the database is laid out. Sometimes I find it easier to follow logic in the code, while I'm writing it, than I do if I try to plan everything about ahead of time. In situations like this I just consider the prototype code as part of the design process.

Anyway, the prototype tool is smart enough to avoid re-running a rule when the output won't be different. And since it is using signatures and not timestamps it is impervious to touch.

It is awfully daunting to have 500 LOC in a project and know you need 10,000 LOC.

There's a fire in Steamboat today and here in Boulder the sky has been yellow and the air smells of smoke. Ugh. It's almost enough to make me look forward to winter.

Today was pretty slow. I need to make some kind of change here.

Saw XXX today. This movie somehow managed to fall below my already low expectations. Amazing.

I'm finally back from my vacation. It was wonderfully relaxing and made me want to hack again. Nice.

I've done a lot more thinking about a replacement build tool. True to form, I thought about it a lot on vacation. I've written some design docs for the back end, and started thinking about the front and "middle" ends, but so far nothing suitable for public viewing.

One implementation issue I struggle with is what language to use. I wonder whether people will be willing to give up the portability of the current auto* tools in order to have a somewhat cleaner build environment. My current preference would be to write a new tool in one of the common scripting languages, probably Python (I don't know it, therefore I can't dislike it :-). Luckily implementation is a long way off, potentially infinitely long.

My Big Fat Greek Wedding. I finally saw this because several friends reported back enthusiastically. I thought it was quite boring. Also it espoused a very stereotypical view of Greek-Americans, on the borderline of offensively so. I guess I didn't see the humor in it that other people did.

A few weeks ago, before vacation, I saw The Sting at the outdoor theatre. Even though I had seen it just a couple months before, I enjoyed it again. Some movies I can see over and over.

Today I read through mjw's diary entries. I also have a difficult relationship with free Java efforts.

On the one hand, I'm proud of gcj. I think we've done some great work. On the other hand, we're still missing pieces that people seem to consider fundamental (AWT...). We still haven't found our killer app, that gets gcj widely used.

Also there's the issue that people still consider kaffe the premiere free Java. Being first really does have its benefits. Lately this hasn't bothered me much. I just work on gcj (less frequently right now), trying to make it as good as I can.

Mark has a very good point when he says that people in the community seem willing to build their projects on non-free infrastructure. From one point of view, the one I most commonly take, this is a mistake -- that work helps to enhance the hegemony of Sun, against the free community. On the other hand, I suppose one might consider it as a pragmatic approach similar to that taken by the GNU tools in 1990: we're doing this now, but when a free solution comes along, we'll migrate. Though so far the migration hasn't happened. And with the rate of change in the Java specification, it may never happen.

So, this is a concern. My own preference has been to use only free tools, to the extent possible, as soon as possible. I usually prefer to suffer a bit with less-than-perfect free tools than to support the non-free ones that may work better. I don't mind finding and reporting some number of bugs. The counter argument is "I have to get work done". Ok, fine, but let's keep an eye on the long term, the important thing: the free software.

gcj still has some legs. If you look at the web page you'll see a steady stream of news items as people add things to it. And more things are coming: Andrew has written a new inliner; we'll probably (eventually) get some nice optimizations out of the tree-ssa work; Bryce is working on the AWT merge with Classpath (when this is done I'll probably dig up some cycles to work on the peers again).

There are plenty of cool projects for interested hackers, too...

Lately I've been thinking quite a bit about the next generation build tool. I've been writing a design for the back end. Don't know if or when it will be available.

I was in Toronto last week, so I dropped out of my normal routine: little email, no Advo, etc. It's nice to do that occasionally, and another blackout starts next week...

thomasvs: I've never read Written on the Body, but I've read and enjoyed several of Winterson's books. I read The Passion about once a year; I've probably given it as a gift more often than any other book (or object).

While in Toronto I talked to Ben Elliston (who I think isn't on Advo and thus, in a sense, not a <person> :-) about requirements and thoughts for a new build tool.

Some other important (though perhaps more minor) considerations that I didn't mention before:

• You need to hack the tools to record untaken dependencies (see the automake dependency-tracking paper), and also to tell the build tool about the other programs they exec. This might seem like overkill, but it is required at least for correctly building gcc.

• Targets and rules require separate namespaces. That way you can still easily write a program named install or info, both examples which have actually occurred.

• Well, there was more stuff, but I've forgotten it once again. I think I have it written down somewhere though.

Lately I've looked at ant a bit, since Eclipse uses it. I'm a bit discouraged by the use of XML. I find it hard to figure out what is going on, there is way too much text overhead. I haven't really followed the current mania for XML, in that respect I'm either lazy, stupid, or hopelessly backwards (take your pick). I haven't come to a definite conclusion regarding ant yet, given my relative lack of experience with it. I'd be interested to hear opinions.

automake: Today I back-ported the versioned install code to the 1.4 branch. The Gnome hackers, particularly hp, convinced me it would help them transition to 1.6 more easily. So, despite my wish for 1.4 to simply die, it lives on... expect the next "final" release soon.

Really this problem comes from some deeper problems with the automake process dating back several years. Basically, I (at the time there weren't other active automake hackers) didn't do a release for 2 years, and when I finally did it was too divergent, had too many new features, and too many bugs. So now, even though 1.6 is really quite good, people are still stuck. This is made worse by the massive changes autoconf has gone through in the meantime.

Lessons continue to be learned here. (Which, incidentally, is unpleasant since it means we are still making mistakes.) We haven't provided smooth upgrade paths from release to release, nor have we even documented the way to upgrade. That hurts the users.

These probably seem like simple things to get right. And they are. Unfortunately for everybody, I still seem to approach automake as though it is the same project it was when I started: a small script, used by a few friends, which I can randomly hack to pieces without consequence. But that world view is really out of sync with reality now. I don't know what excuse the other developers have :-).

As you can see the changes must start on the inside, with the mindset of the developers. Over the years automake has gone through some process upgrades. For instance, we have a documented release process. However, things like this help only in limited ways: when misused they can be an attempt to address an internal problem (my carelessness in making certain changes) via an external method (increased process enforcement). If this works at all, it can only be in limited ways.

sab39: Your code in Classpath didn't suck; it was fine. During the libgcj/Classpath merge, we tried to preserve the best qualities of both implementations, and as a result everybody "lost" a little bit of their work. Though I have to admit, I dislike seeing my code rewritten, especially code I put real thought into writing.

Uche: Your poem impressed me. I've never understood how people can write verse. I don't know if you remember it, but we met at the Boulder Creek Festival a few years ago. I still have your card; it floats on my desk, and occasionally bobs to the surface with the current.

I'm on a lot of mailing lists. I wish Gnus had a mailing list wizard that would let me just enter the name of the list (and maybe the type of list manager). Then Gnus could subscribe me and set up a new group for it. And it would remember my password so I could just give one command to, for example, change my email address on all the lists. Or unsubscribe without remembering anything (other than the right command). Or, better yet, automatically detect whether Reply-To munging is in effect and then set the mode in the new group appropriately.

Tonight I wrote a patch for glib. I don't remember ever doing that before, but the glib ChangeLog does :-)

Recently I've been looking at Eclipse a bit. It is interesting in a way: on the surface it appears to be a pretty standard open source project. However, once you dig a little you find it is run along more closed lines than, say, gcc or apache. I wonder if anybody has tried to quantify how much the processes in place affect a project's "productivity" or "success" (whatever those mean). I'm sure various people have tried to write best-practice pages, though I've never gone looking for them. I imagine these are based more on lore and experience than measurement. That is, "this worked for us" instead of "this will hurt you by 15%".

raph: Thanks for your thoughtful response.

Treating the tool as an object does let you track dependencies on the tool (as well as on the particular command). This is useful in some cases, but a hindrance in others. The useful case I know of is for gcc's target libraries: it makes sense to test every compiler change by rebuilding all the target libraries. Right now you have to use make clean in the right directory to accomplish this.

However, treating the tool as an object also lets you push intelligence into the build tool and out of the various ad hoc places it now lives. For instance, by integrating the tools into the build tool in a more first class way, we could eliminate the need for libtool. We could also export multiple methods from the tool objects; with make all you have is "do the work", when it would be nice to have ways to extract dependency information and perhaps other things. This means we could get rid of depcomp.

Finally, this approach lets us more closely model the underlying program we (may) use to implement a given tool. For example, sometimes there are limitations on running two instances of a given tool in a directory at once. In automake we work around this with the ylwrap script, which serializes invocations of yacc and lex. In the next tool, this could be handled internally. This would be faster and more reliable.

So when I say "treat a tool as an object", I'm really commenting on a facet of the implementation.

I'm very interested in nice logging and other debug features in a future build tool. With make one is frequently left wondering, "why did that happen?". I fixed a bug like that in libgcj recently, and in the end I didn't really understand the patch I came up with. I envision simple ways to ask questions of the build tool, and perhaps a full-fledged debugger as well.

I'll try to write more about this later. Today it is too hot to really think straight.

Thanks to everybody who replied to my question yesterday.

raph: I agree strongly that something better could be done. I've often said that nobody would design the current auto* system from scratch. It is something that could only be grown -- each incremental step seemed pretty reasonable, but it left us with an unreasonable result.

In fact, I've spent a fair amount of time thinking about the properties we'd like in the successor tools. To wit:

• Integrate configuration with the build. This has a lot of nice properties, including parallelizability. Right now on an SMP box the build time can be dominated by configure.
• Read all the build description files at once and use a single invocation of the tool. No more recursive make, but we still preserve the nice qualities of having a separate Makefile in each directory. (We might move this direction with automake.)
• Represent tools as objects themselves. This lets us have automatic dependency tracking across all tools. It also lets us do command dependencies.
• Derived object caching. Right now people use cccache, which is nice; this would let us use that same idea for all steps of the build.

Some of this is what I was leading towards in my Software Carpentry entry. Unfortunately I behaved stupidly then and wrote it all on the last day, and as a result I can't recommend reading my entry :-(.

The bigger problems with such a tool are social, I think. Writing it probably isn't very hard. Deploying it would be, though.

For one thing, you'd have to address the reasons that make auto* popular: packages using these tools properly can be unpacked and work on minimal, ancient systems. Being forced to download a new tool is a pain.

For another thing, you'd need to rewrite existing Makefile.{am,in}s. This is a lot of work, though luckily it can mostly be pushed off on the package maintainers. Still, you'd need to do a few large packages by hand, just to make the new package popular enough.

It's hard to believe I can even advocate a from-scratch rewrite, given that I've read Joel on Software :-).

Anyway, in my entry yesterday I was more interested in exploring the emotional side of receiving (and sending) "sucks" email. I think it isn't so much about the reality of whether automake has had an effect or not, but rather techniques one could use to put such comments into the proper (ignorable :-) light.

For instance, in reality I know automake has had a real impact. I wrote a script last year to look at all the SRPMs in Red Hat Linux 6.2. About 50% use autoconf, and about 25% use automake. So that's pretty significant. I'm happy about that.

But even with this knowledge it is hard to read email about how the program sucks. We're not talking about reasoned criticism here, or even observations like raph's that the system as a whole is incoherent and overly complex. I think what happens is that the non-reasoned parts of the message somehow bring up my fears ("automake really does suck", or "everyone will read this and tomorrow switch to something else"), and that in turn makes me angry.

madscientist: I've tried the trick of writing an email and not sending it. Sometimes that satisfies.

Jim Meyering pointed out that my reply to yesterday's message wasn't really a flame anyway. And I guess this points towards another solution: fighting fair. For instance, comment on positions, not personalities.

I think the bigger picture, in terms of my response, is learning (once again) to insert my consciousness into the process. That is, don't simply respond, whether angrily or not. Instead, measure my own temperature, understand the reasons why, and only then respond in keeping with my "best" goals.

On the other side of things, I've learned over the years to temper my bug reports. I make a real effort to avoid the sorts of absolute, dualistic modes of speech I frequently see in hacking circles. For instance, I try not to write things like "this is completely broken" or "useless". Instead I find a more moderate tone suffices: "I think this is a bug", or "I would prefer that the program do ...". Changing my thinking, and thus my writing, in this way has really been an uphill battle. It's been interesting, though, as the more I'm able to do this the more I see how prevalent it is in the free software world.