Older blog entries for mbp (starting at number 236)

Seth Schoen makes a doubleplusgood point

"Trying to design a limited-purpose computer is like trying to design a limited-purpose spoken language. Imagine trying to design a language that can express only some thoughts but not others."

Seth replies with a worthy comparison of this approach to manipulation of language in Orwell's 1984.

Jem Berkes wrote a good essay about 1984 a while ago.

I entered the Shell / Economist Essay competition earlier in the year. I didn't win, but the winners are so well written that I can't feel bad about it. I think the copyright on my entry now returns to me, so I will put it up later. In particular, the gold prize winner Milksop Nation is just brilliant.

In the entire state of California there is no saloon with a clientele so reckless and depraved that the law will avert its eyes and permit them to take the insane risk of drinking a beer in a building occupied by a person who might smoke a cigarette.

(Good rhetoric is slightly exagerated and simplistic.)

We went to the California Academy of Sciences to see the skull exhibit, fish roundabout, and Eames Powers of Ten exhibit. (Didn't you watch Powers of Ten at school? Don't you have nerdy nostalgia too?). Very cool. mjs says that the Academy of California Sciences ought to have reiki, dolphin telepathy and homeopathy.

I've been listening to the BBC Radio Play of The Lord of the Rings while driving around California. I like it as a story, but I find the underlying philosophy a bit strange. The bad guys are evil in their bones -- there is no possibility of even a single orc joining the other side, or any question that there might be fault on both sides. Whereas in the real world, given sufficient perspective (say, a thousand years), it often seems that there is fault on both sides, or at least that evil is not so easily apportioned by race.

A war without death, but not what you might think:

Armored Combat Earth Movers came behind the armored burial brigade, leveling the ground and smoothing away projecting Iraqi arms, legs and equipment.

(I expect enthusiastic praise for US Army landscape gardening from mglazer.)

update:

movement, here is at least one reference for malloc returning memory to the OS:

Doug Lea's malloc (If anyone wants to be a better programmer, I would suggest they should read stuff by Doug Lea.)

The ``wilderness'' (so named by Kiem-Phong Vo) chunk represents the space bordering the topmost address allocated from the system. Because it is at the border, it is the only chunk that can be arbitrarily extended (via sbrk in Unix) to be bigger than it is (unless of course sbrk fails because all memory has been exhausted).

"wilderness" is such an excellent, vivid, clear name.

I agree that it will often not be the case that there is contiguous memory at the top that can be returned to the OS. However, (as dl says), for programs that allocate memory in phases, or in a stack pattern, it may well be that memory which is allocated last is freed first.

Big, long-lived allocations perhaps should perhaps be in mmaps (perhaps containing arenas), so that they can be returned. For example, Samba now stores a lot of private data in .tdb files, which are mmaped. When they're not used, the memory is returned.

However, I think being able to return memory is perhaps atypical. Most programs run to completion, allocating memory all the way (e.g. gcc), or reach a steady state and then remain within it (e.g. servers or applications.)

It would be nice if Linux let you find out how many pages were being used by a particular map, but I don't think there is any easy way at present. Perhaps with rmap...

Of course, the more common case of "returning memory" is just allowing pages to be discarded by not touching them. This also indicates why it can be worthwhile to have swap on boxes which have plenty of memory: data pages which are still allocated but never touched can be written out, allowing more ram to be used as a disk cache. Apparently swapfile support will be better in 2.6, reducing the problem of needing static allocation of swap partitions.

A Java implementation that used handles and did not rely on objects not moving in memory would have the option of defragmenting itself to allow wilderness to be returned to the OS, or even just to avoid paging. I don't know if this is ever considered worth the code complexity and CPU cycles that it would cost.

The "hotspot" effect would suggest that for most programs where memory usage is a problem, it will be a few routines or classes of allocation that use most of the memory. Changing them to use mmap, or less memory, or an external file might fix it.

Perhaps oprofile would let you find out what programs are "causing" paging? (Not that it's really any one process's fault...) I haven't tried it, but I really want to.

I checked quickly and Debian sid's libc malloc uses mmap by default for allocations of 200kB or more. (I'm too lazy to find the exact value.) They're unmapped when freed.

fxn writes:

The solution came yesterday night, I read in Perl Debugged that Unix processes only grow! On page 176 it says: Note that the amount of memory allocated by a Unix process never decreases before it terminates; whenever it frees memory it makes it available only for the same process to reuse again later. (It is not returned to the free pool for other processes to use.)

That is mostly true and a good way to start understanding it, but it is not completely true.

You can think of Linux as having a two-level memory allocation system: the kernel gives memory to the C library (via sbrk, mmap, etc), and then the C library gives it to the application (via malloc etc).

There is a little bit of slack in the C library: sometimes it will ask the OS for more than it needs at the moment, and it will not necessarily return freed memory. Instead, freed memory is hoarded because it will probably be needed again soon.

Above a certain high water mark the C allocator may return memory to the OS. I think there are some parameters that you can tune to control this behaviour but in general the defaults are fine.

And this explanation is a generalization too: some programs, particularly databases, request memory of their own using mmpa, independent of the C allocator.

In addition, some programs map files into memory, and if they release that mapping then the memory will be returned to the OS straight away.

Of course all this is only at the level of virtual memory. Normally we're interested in physical memory because it's more scarce. Even if the C library never returns memory to the kernel, the kernel may eventually page it out to disk and free up the physical memory for other uses.recentlog.html?thresh=3

itamar

The fourth talk was about raising exceptions in signal handlers in Python, and the problem this causes.

What an interesting problem!

If I remember correctly (and it's been a long time), the Java specification says something sensible about asynchronous exceptions. I suppose the Python people have read that.

Interesting post by Linus explaining the patch acceptance process.

I've been running a little development weblog for distcc. I'm not sure if people like it, but I think it works well -- at any rate, it's something I would find interesting if I was looking at somebody else's project. It lets people know what features are coming up, or what's going wrong, or what bugs are being worked on, without requiring them to read the mailing lists. It can be interesting to know if a project's stable or not, or active or not. Perhaps it's like a poor man's (or small project's) Kernel Traffic. Some game developers do something similar with a .plan file visible on the web.

I wonder how it would work to let people see interleaved posts from samba developers commenting on what they're doing? (Not that I have time to write that.)

At work I am doing some performance tuning on our Python project. This is easier than it might sound: in any language, most of the time is spent in just a few routines. Rework them, or at worst rewrite in C, and things get much better.

give war a chance

Well, hey, it's always worked so well in the past...

Based on past experience, a USA-Iraq war would cause a hundred thousand civilian deaths, or at the very least a few tens of thousands, and of course a greater amount of injury, homelessness, and ruin. This is only counting people who are not combatants and who just happened to be born in the wrong place at the wrong time.

One might make an argument that it is necessary for those people to die so as to prevent a greater war later but I don't think it's right to do it without grave consideration, or to feel jubilation at the prospect.

ps So good

I've just been to a ClearCase training course by Island Training. It was really good: the instructor knew his stuff, made it not boring, and nicely catalyzed discussion about how it relates to our own build system. Tim says that HP's in-house training is always excellent and it seems to be true.

I have more respect for ClearCase now: in it's favour, it is a nicely generalized database, and it's very consistently unixy (mkview, lsview, rmview, ...). It avoids some problems with CVS, like not handling directory restructuring, and not being able to mark tags or branches obsolete without removing them. I even worked out how to do merges in emacs rather than through the silly built-in tools.

On the other hand, I think putting it into the kernel is a perfect example of a design that looks good at first but really ought to be dropped on consideration. (At the moment, our build machine is rebooting probably because ClearCase somehow interferes with loopback devices.) And it's still the perfect antidote for distcc's speed.

I've now mostly moved into my new apartment. I feel really grown up to actually (fractionally) own it. I have taken advantage of my newly acquired right to bang and drill holes in the walls.

After much prodding earlier in the year from my brother in law, I finally read The Science of Discworld. It's really excellent. The title might make you think it will be a bit too silly, but it's not at all. As well as a nice and entertaining overview of major areas like physics and evolution, it also has probably the best explanation of the epistimology and social processes (Popper and Kuhn) of science that I've seen in a general audience book. It is quite fair to compare it to Gould or Dennett, and perhaps gives an even better under standing of the way science actually progresses, rather than presenting it as a body of immutable facts. And I like their suggestion that space elevators will have elevator music.

----

I was flabbergasted to read this story in the AFR, quoting John Moses speaking at a memorial for Bali bombing victims:

In Canberra, Prime Minister John Howard lit a candle in StPaul's Anglican Church and listened while priest John Moses sermonised on the inadequacy of "sentimental humanism". [...]

"Decency without doctrine ... spiritual laziness," he called it. [...]

Speaking to the Prime Minister, Professor Moses said it was necessary for Christ to become the conscience of the state and the role of the state was to be an instrument of God.

I don't know whether it's more distressing to me that somebody would espouse such an opinion in this century, or that it would apparently get a hearing from Howard. I suppose I'm not surprised by the second, but it still disappoints me to see it.

"Decency without doctrine" actually sounds to me like an superb practical approach to morality in a pluralist modern world. I cannot understand the imbalance of mind that makes one want to respond to supposed religious violence by re-establishing a state religion.

In any case, even if we wanted to go along with that, it seems to me that Jesus would have encouraged people to turn the other cheek. I don't suppose anyone holding that opinion would be invited to speak at official functions.

search-and-replace:

It is necessary for the Koran to become the conscience of the state, and the role of the state is to be an instrument of Allah.

Mm. Doesn't sound so reasonable (to western ears) now, does it?

After a lot of fiddling, I got XFree86 4.2, GARNOME 0.18, Mozilla, and OpenOffice to all agree to use FreeType fonts. I have to say the results look very nice indeed, very smooth and nicely antialiased.

It took me ages to work out that I needed to re-run mkfontdir, because Debian had somehow forgotten. (I guess I probably have used that command once, years ago.) It's fundamental to friendly/reliable software design that caches should regenerate themselves if they're out of date. X11 is, as they say, a complete fucking flying circus.

I guess it's getting better, through FreeType and the GNOME font installer and so on, but at the moment the functionality is nice but the usability is very nasty indeed.

I'm sure Windows is just as ugly on the inside. A friend of mine, by installing a TrueType font using the approved method , managed to trash a Wince device so completely that the motherboard needed to be replaced. (Soldered-on persistent memory, I suppose.)

--

linux.conf.au registrations are open; the programme should be announced any day.

I am really pleased with the distcc test suite. It catches a decent fraction of bugs, and new ones that are reported can usually be tested to prevent regression.

I think the trick to testing is to start early, and be disciplined about keeping on adding them as you go. Like washing up, if you leave it for three months then it looks very intimidating. :-)

--

being non digital:

Canberra is accurately described as having pulse-width-modulated weather: Spring days fluctuate between maximums of about 12C and 25C, until eventually warmth wins out.

Last weekend was warm. I took my motorbike up over the Snowy Mountains highway from Cooma to Adaminaby. It was very beautiful indeed. It does feel like the roof of the continent.

Also, I just got the keys to my new apartment, and I'm going to start moving this weekend. Yay.

A year or two after emigrating, she happened to be in Paris on the anniversary of the Russian invasion of her country (Czechoslovakia). A protest march had been scheduled, and she felt driven to take part. Fists raised high, the young Frenchmen shouted out slogans condemning Soviet imperialism. She liked the slogans, but to her surprise she found herself unable to shout along with them. She lasted only a few minutes in the parade.

When she told her French friends about it, they were amazed. ``You mean you don't want to fight the occupation of your country?'' She would have liked to tell them that behind Communism, Fascism, behind all occupations and invasions lurks a more basic, pervasive evil and that the image of that evil was a parade of people marching with raised fists and shouting identical syllables in unison. But she knew she would never be able to make them understand. Embarrassed, she changed the subject.

-- Milan Kundera, The Unbearable Lightness of Being

Zaitcev has a good point about OPN being lilo's demonstrated delivery.

I guess it is hard to tell how much of the value comes from rlevin himself, and how much from the server and channel operators and users. But that's true to some extent for any large project: the founder or leader gets most of the credit, even if their work is made possible by many smaller contributions. Nothing particularly wrong with that.

So there you go.

microkernels

Another interesting step towards microkernel design in Linux is kernel mode linux, which is supposed to allow user programs to execute in kernel context. I won't say the guy isn't on drugs, but it's an interesting hack.

I found this, by the way, at sweetcode, which is a nice little site.

the status is that rob levin should show us the code

(amateur social psychology for fun and profit)

rlevin writes

I agree completely that, when the issue is code, there is no substitute for at least providing a reference implementation that can be critiqued. My only point in commenting was to note that not every issue is a code issue. Sometimes some community people use the refrain "show me the code" in a way that denies the significance of social or procedural issues, and confuses implementation with goal-setting.

The motto "show me the code" doesn't literally mean that source code is the only acceptable demonstration. Other phrases that get at the same idea are "talk is cheap" and (my current favourite) "all hat and no cattle". (That doesn't, by the way, mean that you need to literally own cows to get respect.)

For a meritocracy to be meaningful, people's contributions have to be visible in some way that can be assessed by their peers. This can be in ways other than code: artwork, writing, speaking, web sites, and organizing conferences spring to mind.

The subtext of Rob's diary entries seems to be that he wants people to value his efforts in things such as goal-setting and procedures. Indeed, a little while ago, he claimed the work was so important, and has was so uniquely qualified to do it, that people ought to pay him to do it full time.

But this directly conflicts with the "show us the code" mindset, and I suspect this is why it irritated so many people. Nobody can actually see what it is Rob does, whether it's worthwhile, and whether he does it well. So who can tell? The default position encoded by "show us the code" is to assume that it's just vapor until demonstrated otherwise.