Recent blog entries for johnm

Ob-it's been a really really really long time...

For today, this is just a wee note about my previous entry, in which I had found an inadequacy in ccache's hash. It had returned a cached object with incorrect file paths in its debug information, which the linker was then printing out in warnings and error messages.

Other people must have been noticing that too, because soon thereafter ccache acquired a CCACHE_HASHDIR option to put the whole path into the hash calculation.

That wouldn't help me much, of course, because with my build directory varying daily it would be the same thing as just disabling the cache entirely! Instead, I wound up using a small sed script to canonicalise the logs before I looked at them.

Pretty stone age, but effective.

Extreme panic!

Each time I do a full prc-tools build, I get a 1.3 Mb build log and sometimes I even look at it. (Building prc-tools means building two binutilses, two GCCs, a GDB, and a few other bits and pieces. Along the way you build four BFDs and six libiberties. So that log builds up pretty quickly!)

At the moment, due to popular demand (three FreeBSD whiners :-)), I'm working on making --enable-nls work properly. To be sure, there's a bug here in the current top-level prc-tools, and I'm glad to be finally looking into it.

So I'm comparing the logs from builds with and without --with-included-gettext. I look at the differences between successive builds regularly (all hail tkdiff!), but these two (of course) differ from each other in different places from the usual. And there's many more differences to look at than usual, alas.

This means I'm studying different parts of the logs from usual. Here's a few lines from tonight's builds:

gcc -W -Wall -Wstrict-prototypes -Wmissing-prototypes -g -O2 -o size size.o bucomm.o version.o filemode.o ../bfd/.libs/libbfd.a ../libiberty/libiberty.a
bucomm.o: In function `make_tempname':
/home/johnm/work/rpmbuild/BUILD/prc-tools-20020831/m68k-palmos/ binutils/binutils/../../.././binutils/binutils/bucomm.c:236: the use of `mktemp' is dangerous, better use `mkstemp'

Huh? What's that path from the end of August doing there? That build tree is long since blown away -- each one is over half a gigabyte, so I nuke them pretty quickly. So I look at the build logs from the last week, and they all say 20020831, and sure enough that directory certainly hasn't existed for a long time.

What's going on? Has ext3 let me down? Is my hard drive failing again? Is the end nigh?

Panic stations! Single user mode, fsck... no, nothing.

Have you guessed yet? It was ccache.

I last really compiled bucomm.c back in August when I switched ccache on. One of ccache's big features over compilercache is that it doesn't give up in the face of warnings. So that 20020831 directory doesn't really exist; it's just that I'm getting the old stderr log from August. It's hard to see how ccache could correct this without rerunning the compilation or parsing the basic error format and guessing, so it's understandable; on the other hand, the Web page does say

[ccache provides] exactly the same object files and exactly the same compiler warnings that would be produced if you use the real compiler. The only way you should be able to tell that you are using ccache is the speed.

Sure, the speed is great -- ten minutes instead of 40 with my machine on this project -- but I've just learnt one grain of salt to apply to the warning guarantees :-).

Sure, there's other lessons to be learnt here: like don't vary your paths when you're trying to do repeatable builds (but some form of date convention is a useful and commonly used way to stay sane) and don't use absolute paths (but sometimes your tools will make them absolute whether you want it or not).

Anyway. Onward to the next disaster...

My fascinating question for today: What should malloc(0) return?

Composing a critique of somebody else's crappy half-baked stdlib implementation today caused me to take another look at my own crappy half-baked stdlib implementation. In part, my critique puts the memory allocation functions malloc(), free(), calloc(), and realloc() under the microscope. One of the four is easy: calloc() is really just malloc() in disguise -- the only wrinkle is to remember to call memset() afterwards, which the somebody else in question forgot :-( --, but the other three engage in an interesting dance.

The C99 standard has the following to say about malloc(0) and friends, and the text in C90 is similar:

If the size of the space requested is zero, the behavior is implementation-defined: either a null pointer is returned, or the behavior is as if the size were some nonzero value, except that the returned pointer shall not be used to access an object.

I've always preferred the latter behaviour for malloc(): it avoids overloading the meaning of a NULL return, so, especially if your size was a variable rather than a constant, you can say "if malloc returned NULL, then abort due to out-of-memory" without having to fudge about and check that it wasn't really a returned-NULL-due-to-zero-size situation. (Of course, you could say that people who actually call malloc() with a size of zero deserve what they get!)

That (the latter) is also the behaviour required of a C++ allocation function. So if you're writing operator new() in terms of malloc(), you have to know that your malloc() has the same preference as I do, or take extra care instead of just calling malloc().

Now let's look at some of the other requirements that C90 and C99 place on these functions:

   free (NULL)    = ({})
realloc (NULL, n) = malloc (n)
realloc (p, 0)    = ({ free (p); return NULL; })    when p != NULL

The first is just saying that free(NULL) does nothing, which is, of course, very convenient -- you get to avoid some tedious checking. The other two are parts of the definition of realloc(). They are spelt out in the C90 text; the C99 text as pertains to the last one is quite different, but it can still be derived from the text.

Now, the "when p != NULL" condition on the last rule is not very pleasant: it makes for an ugly algebraic rule. Futhermore, I would contend that programmers who actually write code that's literally "realloc (p, 0)", with p variable but a constant 0, expect realloc() to Do The Right Thing even when p is NULL, just as free() does. Thus I would contend that programmers actually expect that they are using a stronger set of rules, in which the third rule holds regardless of the value of p. (In particular, this is what the Linux man page says realloc() does; yes, the man page is way stronger here than the C Standard!)

In that case, we can calculate the value of malloc(0):

malloc (0) = realloc (NULL, 0)
           = ({ free (NULL); return NULL; })
           = NULL

So that means that the choice is really between:

• either a realloc() that Does The Right Thing for size=0
• or a malloc() that does what I've always preferred for size=0

You can't have both! So I've changed my mind: being able to write realloc(p, 0) and have it Just Work, like free(p) does, is more important to me than having malloc(0), which I never actually use, be easily distinguishable from memory exhaustion.

In retrospect, this was all obvious. The specification of implementation-defined behaviour that I quoted at the beginning applies to both malloc() and realloc(). Wanting realloc(p, 0) to be useful in cleanup functions (by just freeing through p, and not also allocating some dummy "zero-sized" object that the caller will ignore and will become garbage) means that you've already made the choice: realloc() given a size of zero will return a null pointer. Either that means you've also already made the choice for malloc() too; or you've decided to make the choice differently for the two functions, which would be really ugly and I'm not even sure is allowed by the Standard.

But playing with the algebra was fun while it lasted!

(Looks like I'm going to have to put that "malloc (size? size : 1)" stuff back in my library's default C++ allocation functions.)

Ilan writes

If Jango Fett was the genetic template for the storm troopers (which is what I assume they were implying), why do they have American accents in the first three movies? Were English accents considered a genetically undesirable trait and removed from the cloning process?

That's a New Zealand accent, mate! (Oh, and the answer to your question is no, English and other non-USA accents are not genetically undesirable :-).)

The kid's (Boba's) accent was pretty strong too. I was laughing during their conversation while flying through the asteroid field: it's not every day I get to hear a Kiwi accent out here in Norway.

ObLongTimeNoDiaryEntry:   I've been submerged with work for ages. I decided a while back that I was going to ignore everything else until I got the new release of prc-tools out the door. It took at least six weeks longer than it should have, alas, and I finally released 2.1 on Monday. Phew! So now I can get back to having a life, working on more interesting parts of prc-tools, and replying to all the friends' emails I've been ignoring (sorry Graham!).

I've been back home from New Zealand and PalmSource for a couple of weeks now. It's great to be back, especially since we've had a nice sprinkling of snow over the last few days: the frozen lake near my house is looking really really good. It's a beautiful place for a (very flat) walk at the moment.

Palm tools stuff

A compatibility bug in PilRC's parser has been bugging me for almost a year:

INTEGER 1000 10

got unilaterally changed to

INTEGER 1000 VALUE 10

back then. There was some value in this new VALUE keyword because it was a way of specifying negative numbers, but it should have been made optional so as not to break people's old source code.

Aaron is back motivated to make releases again (yay!), and on Friday I made a fix to make the VALUE optional that was better than my original fix that hadn't been integrated. It was a nightmare btw: every time I do anything in the PilRC source code I swear that I'm never going to touch it ever again.

So there's now a 2.9p2 PilRC release, and I can finally put new binaries up on the Sourceforge site. (I refused to update them until the source code compatibility bug was fixed.) I've put up x86 Linux RPMs tonight, and will make packages for the Cygwin users sometime over the next few days.

I did something a bit naughty in the RPMs: I created a new Group for them, instead of using one of the standard ones from Red Hat's RPM groups file. I'm in two minds about this; I've previously used Development/Tools for my prc-tools RPMs and resisted creating a new group. But the fact is everybody else does it, and it's helpful to the users if they can find the Palm OS-related tools in one place rather than having to hunt through the 800 other packages in Development/Tools.

After the next time rpmfind crawls through Sourceforge, hopefully people will be able to find Palm OS-related tools in

Development/Palm OS

Pretty soon I'll be making POSE RPMs and putting them there, and the RPMs of the next prc-tools release (any month now :-(, really) will be moving there from Development/Tools too.

raph's rebar

Describing builds as a lazy functional program. Wow. That is just a gorgeous idea!

skipping the compilation if it's already been done (in a previous invocation)

Do you mean in a previous compiler invocation (in a different part of the calculation) during the same build, or possibly a (memoized :-)) compile of the same thing during a previous full build? (Or maybe the question is meaningless in a functional context. :-))

The latter is similar to compilercache, which was recently featured on sweetcode, and which I've just started using. It caches object files based on the compiler flags and preprocessed source used to produce them, and just spits back the cached object instead of calling the compiler if sees that the exact same compilation has already been done. It's working wonders on testing POSE RPM builds: each full build takes a few minutes instead of twenty or more.

Transit lounge interlude

Oddly enough, I'm back at Changi Airport. This time I brought an ethernet cable with me, so I don't even have to play with my Airport settings.

It was supposed to be a one month trip to Europe, but these things change. :-) Am currently on my way back to New Zealand for a week's holiday before PalmSource in California, and then back to my new home in Norway.

Oslo->London->Singapore->New Zealand->California->Oslo. I'm certainly taking the long way around!

ObFreeSoftware: I'm too tired this time for the flights to be very useful for getting any work done.

Transit lounge interlude

I'm currently in the transit lounge at Sydney airport, en route from New Zealand to the UK. The world is looking better than the last time I was stuck in a transit lounge in the US: they have free Internet kiosks here in Australia! Of course, it's a mutant version of Internet Explorer that will only let you have one window, and the keyboard's set to some weird Chinese layout (it took me a long time to find "w" and "."!). But you can get to the usual Windows telnet client without too much difficulty :-).

prc-tools

Since I'm sitting on a plane for 24 hours, I've got plenty of time to work on prc-tools. (I had better sign off here soon and go and steal some of the airport's electricity for my batteries though.) I've been working on installation issues for a while now. I kind of want to spend this time working on GLib shared libraries, because a few vocal people keep bleating about them and fixing them would shut them up once and for all, but I'm really not very motivated to work on those.

pilrc

Oops. I seem to have offended a few people by questioning the quality of some of their recent patches to this project. (Since I'm sitting on planes at the moment, I'm not going to be posting to the thread I started for the next day or two. Hopefully not too much momentum will be lost.)

Actually I certainly didn't intend to offend them; if anybody, I was intending to offend the maintainer.

Well, not really ;-). But if it were my project, I would be a lot more selective about what I accepted. It seems to me that pilrc is being pulled in several directions at the moment, and the dominant direction is not the long time user base of application developers. And that's fine, but some of the additions are making things worse for the project and for those developers, in particular by making pilrc's source code unmaintainable. If it were my project, I would be emphasising correctness and thought-out-ness rather than breakneck pace of development, which is what one faction seems to want at the moment. It's fine for them to want that, but not at the expense of the long term users.

It's not my project of course, but that doesn't mean I'm not allowed to lobby the maintainer!

And I know Aaron is reading this... :-)

[Later] More airports, and airports at airports

Now at Changi Airport in Singapore, where there's a free 802.11 network. If you don't have a wireless ethernet card, they'll lend you one. Sweet!

Unfortunately, at 4.00am I can't be bothered figuring it all out (Linux 2.4.whatever recognised the card, the interface partly came up, but the network wouldn't speak DHCP to me), so this is coming from Windows. Bugger. Can't download my email properly.

Ob-Code Red

jschauma mentioned grepping apache logs for Code Red droppings, so I had a look. I got 9 today. This surprised me: I'm only on a dialup, and I've only been online for an hour or so today.

When the Red Hat 7.1 installer asked me about what firewall settings I wanted, I said none because I would either be on a company ethernet behind their firewall or on a dialup, and who on earth's going to come looking for me there? It seems that may have been a little naive... (And it's a good thing I'm not running any servers on the Windows side of this dual-booting box!)

I've only received one email due to it though, so I won't be breaking out the procmail big guns yet. I didn't realise that email might have somebody else's interesting document in it, so I just deleted it without looking. D'oh! [Double D'oh: it's the other one that sends the entertaining emails. Oops.]

Hacking

There's not many things less fun than writing instructions and tools for installing your project on an operating system you don't care about and never use in earnest yourself. The installation's really not that hard: the old instructions really did work. The project is a compiler and other programming tools, so my attitude used to be

They're programmers, so they should be able to follow the instructions to the letter. If they can't, it doesn't much matter because they wouldn't have much luck with the compiler anyway

Alas, my attitude is being forced to change to "I'll do anything to stop getting email from these people about this pathetic issue!!!"

Admittedly, the part that is changing did seem to be a little fragile (although it always worked for me... go figure), and the new method (our own little setup.ini) is what I intended all along. And playing with Cygwin's setup utility is kind of fun.

But writing moron-proof instructions is never fun.

It's been a while.... I became one of those layoff victims about a month ago, and I've not felt much like computers since -- and I don't have one at home to write on anyway.

Today had a couple of firsts:

• For the first time, I turned down an appealing job offer today. It's a shame, but there it is. I'm accepting the other also appealing job offer though.
• Tonight, while I was riding my bicycle down Almaden Expressway in San Jose, for the first time in my life some guy in a car decided he needed to throw something hard at me. Suddenly something painful hit me in the back, there was much wetness that I couldn't immediately identify, and there was laughter and a violently accelerating SUV on my left. It took a while to figure out what he had thrown, and whether I was significantly hurt (I wasn't, of course).

  What kind of freak has stuff on hand to throw at cyclists while they're driving down the expressway about to get on the 85 onramp? One who's just been to the supermarket and bought eggs, I guess. And one who has the appropriately neanderthal mentality.

It seems somehow fitting that that happened on the very day that I decided to leave the United States.

Rode the bike in this morning (cycling shoes). Then bouldering and a sauna for a while (climbing shoes). Now wearing vaguely formal shoes and the others are arrayed around my cube.

I feel like Imelda Marcos...

Fortunately, you don't need special shoes to play with cygwin, which is what's on the cards for this morning.