Older blog entries for crhodes (starting at number 132)

I was going to blog about the fact that using org-mode, referred to in my previous diary entry, made the important but non-urgent tasks more visible. I was going to use sorting out backups for interesting data (say, my e-mail archives) as an example, and discuss the solution I came up with, but I have just realised that the trust model is exactly backwards (my server trusts root on the backup machine). This is annoying, because I thought I'd got it right, and because getting it right would have been equally easy.

Oh well. So, instead, I'll return sorting out backups to the TODO (or maybe STARTED) state, and (prompted by some recent discussion on #lisp IRC) I'll blog about SBCL's interpretation of Unicode characters, with the up-front caveat that I'm Not An Expert in this languages, glyphs, graphemes, characters and all that jazz.

Common Lisp's string type is defined to be a vector specialized to hold only characters or a subtype thereof. This definition is already hard to wrap your head around, and has amusing consequences documented here in the past, but I don't want to get into it too much; merely to say that already this definition restricts to a fairly large extent the possible implementation strategies for supporting Unicode.

Why so? Because in Unicode there are several notions of ‘character’, and we have to decide which of them we're going to use as our Lisp character type (and use as string constituents). The simple answer from the implementation point of view (and the route that SBCL currently takes) is to define a Lisp character as an entity corresponding directly to a Unicode code point. This is simple and straightforward to implement, but unfortunately has the side effect of making various Common Lisp string functions less useful to the user.

How so? Well, consider the string comparison functions, such as string=. As specified, string= compares two strings, character by character. In SBCL, then, this compares two sequences of Unicode code points, character by character, for equality. The problem is that this operation doesn't in general have the semantics of ‘string equality’, because in Unicode there is more than one way to encode the same abstract character: for example, the e-acute ‘abstract character’, or possibly ‘grapheme’, e-acute (which is usually displayed ‘é’) can be represented either as the single code point U+00E9, or as the combining character sequence U+0065 U+0301.

So, that's OK; the Unicode FAQ on Combining Marks says that characters and combining character sequences are different, and even implies that programmers should be dealing with Unicode code points (SBCL characters). Unfortunately, Lisp has been around for longer than Unicode, and code has been written essentially assuming that string= performs a language-string equality comparison rather than a codepoint-by-codepoint equality comparison, simply because (pre-Unicode) these two concepts were conflated.

What about the alternative? We could try defining Lisp characters to be abstract characters, represented as combining character sequences. One problem with this idea is that there's the char-code function to implement: for every Lisp character there must be a corresponding unique integer. That's not so much a problem – Lisp has bignums after all – but it will make char-code-limit surprisingly large (in principle, I think every combining mark could be applied to a given base character). This means that we'd lose the ability to represent an arbitrary character as an immediate object, meaning that accessing characters from strings would in general cause heap allocation, and lead to surprises elsewhere in the system.

So, given that we stay with a Lisp character corresponding to a Unicode code point, what other pitfalls and details are there to consider? The memory representation of strings of type (simple-array character (*)) is worth mentioning; because there's a fairly strong cultural expectation of O(1) access time in vectors, we don't do any compression, but simply store each Unicode codepoint in a 32-bit cell. SBCL has a separate base-string representation, where each ASCII codepoint is stored in an 8-bit cell; a long time ago I gave a talk about this.

Also, interpretation of the contents of strings has caused confusion recently. Granted that a string is a vector of (effectively) code points, what does that mean for strings containing surrogate characters (code points in the range U+D800–U+DFFF)? These code points do not correspond to any abstract characters directly; instead, pairs of surrogates are (in certain Unicode encodings, such as UTF-16) interpreted as characters beyond the Basic Multilingual Plane. Some Lisp implementations (such as OpenMCLClozure Common Lisp) go so far to resolve this ambiguity as to forbid the creation of a Lisp character with a surrogate codepoint. In SBCL, however, we take the view that those characters exist, but should not be interpreted in any way; a string containing surrogate pairs should be considered to have individual surrogate characters in it, and no attempt should be made to combine them. If there is data in an encoding which uses surrogate pairs (such as UTF-16), then that data should be read in using the :utf-16 external format, so that no surrogates are present at the Lisp level; an attempt to write out a surrogate Lisp character in a Unicode encoding should generate an error. (NB: not all of this is implemented yet).

All of this merely scratches the surface of Unicode support; I'm hoping to find time to implement better support for finding properties of the Unicode Character Database, and to implement Unicode algorithms for normalization, collation and so on; I'm also planning to tighten up support for the Unicode encodings (to address the potential security issues that exist from nonconforming decoders) and generally to improve support for doing useful things with non-ASCII. As usual, there's likely to be a significant lag between planning and doing...

Although I've used Emacs for well over 10 years, and I've been a pretty serious Common Lisp programmer, I don't consider myself an Emacs power user by a long shot: I don't think I'm at all skilled in its use; I don't customize it very much; and I know almost none of the interesting bits of Emacs Lisp. The only applications I really use within Emacs are Gnus, the News and Mail reader (which I adopted after one too many "please find attached" errors) and, more recently, SLIME (but since much of my Common Lisp work is at the implementation level, the luxury of using SLIME for development is often unavailable.

Recently, though, I'm beginning to suspect that I might get more practice. Firstly, the additional responsibilities of a permanent job in UK academia (essentially boiling down to administration, lots of it) meant that my old strategies for managing my TODO list (scraps of paper, followed by writing in different colours on my office wall, followed when I ran out of wall space by a plain-text TODO.txt file) were no longer sufficient to manage the complexity, and I discovered org-mode.

My use of org-mode has, roughly, removed the feeling that I'm drowning in a morass of multiple different things to do with no idea of what's important or urgent along with a feeling that I've forgotten the really important things, and replaced it by a vague feeling of guilt that I'm not doing all the TODO, STARTED, WAITING items that show up in bold red in my *Org Agenda* buffer (and probably won't, ever). I haven't yet found a good way of integrating notes that I take on my often-disconnected laptop with the master org files kept on my server, but vague guilt is noticeably preferable to drowning, so this is a net win.

Also a net win for me is the new daemon functionality in GNU Emacs 23. My setup for mail is perhaps a little eccentric; I read all my mail, including work mail (over imap), from a machine at home which I might call a "server"; I used to log in over ssh, start emacs -f gnus, and go on from there. With the new daemonized emacs, I only need to start emacs once, and can connect to it later, whether that's from my office workstation, or my laptop at some arbitrary location (assuming that I have Internet access, anyway), or even someone else's Mobile Internet Device.

I found one problem with this new modus operandi, though; once I'd logged out of the shell session that started the emacs daemon, accessing remote files with tramp (over ssh) became more painful, as the ssh credential forwarding provided by the ssh-agent responsible for that session was no longer available. The solution I found was to run an ssh-agent instance within the emacs daemon at startup; I have lightly adapted some code by Will Glozer; with my version, a simple


(when (daemonp)
  (ssh-agent "ssh-agent -d"))
in my ~/.emacs does the job very nicely.

I've just come back from the 2009 edition of the European Lisp Symposium, held in Milan. It was fun! (It was hot, too).

I agreed, at something like the tenth hour, that I would present a tutorial on using SBCL internals; perhaps inevitably, I left it until somewhat past the eleventh hour to actually prepare for the tutorial, with the result that I'm not sure how effective it was – it certainly wasn't the kind of tutorial where the audience follows along each on their laptop, and that meant that I didn't really get any audience feedback during the session. In partial mitigation, the tutorial materials should appear on the ELS website at some point; they're also available from my talks page at work.

My somewhat lax preparation regime meant that I didn't get maximum benefit from the social time, since on Wednesday evening I participated in the steering committee meeting, had a little bit of a drink at the reception, and then went off and worked on my tutorial. I did have a chance to speak to some people later, particularly at the final dinner, but not as much as I would have liked (this was also partly dictated by my travel arrangements, which were extremely tightly placed around the conference's official schedule). The other presentations were fun, though; the accepted papers showed a good range, from very specific aspects of particular lisp evaluation models (stand up, Tom Burdick and OMG1) to using a web browser as the interactive output pane of a Lisp debugging session. I enjoyed the keynotes as well, even those where I disagreed slightly or profoundly with the speaker on the technical or social issues discussed; the debate on the future of CL (the panel: Pascal Costanza, Nikodemus Siivola, Stelian Ionescu, Marco Antoniotti, and Scott McKay) was also a success, I think; there was at least a range of opinions on display.

I'm looking forward to the 2010 iteration (to be held in Lisbon), not least because I will be rather more involved in the organization of it than I was this year. I'll be watching for any other wrapups of this year's event, and if anyone has any ideas for what they'd like to have in the programme for ELS2010, please don't hesitate to get in touch!

I've recently been working on improving SBCL's build behaviour.

Well, perhaps that's not exactly a novel or surprising piece of information for this diary: but maybe this is the last major effort that has to be made. One of SBCL's raisons d'être is to be a Common Lisp compiler, written in Common Lisp, that does not have any dependencies on the host Lisp compiler used to build it (assuming that the host implementation is sufficiently conforming, but not making any unwarranted assumptions about any implementation-defined or undefined behaviour). This method for building a Lisp compiler is sufficiently tricky to wrap one's head around that I wrote and delivered a paper describing it, including diagrams, for the Workshop on Self-Sustaining Systems last year.

So what's new now? Well, given this aim, it would be reasonable for the output of the SBCL cross-compiler, running as an application in the host Lisp, to produce bitwise-identical output files independent of which host Lisp was being used. Reasonable, perhaps, but in practice the state of SBCL until a month or so ago was quite a long way away from this ideal: every output file had a header declaring when and from which compiler it had been generated, which was not going to help bitwise comparisons: and even after such straightforward issues had been dealt with, every single output file from the build process (of which there are of the order of 350) exhibited non-trivial differences when compiled with CLISP from the corresponding output files compiled with SBCL.

The differences between output files that I've observed and fixed in the last month or so – by lavish use of cmp(1), staring at emacs buffers consisting largely of control characters, and the time-old debugging method of Thinking Very Hard – can be broadly split into three categories:

  • outright leaks: information being taken from the host compiler and erroneously treated as though it was applicable to the target. Sometimes this was simple carelessness (the cross-compiler's constant-folding for symbols used symbol-value, even for things like most-positive-fixnum); sometimes it was nastier than that (CLISP and SBCL disagree on the value of (log 2d0 10d0) in the last binary digit). It's not nice to find these things still showing up, but at least this set of changes should make these bugs stick out like a sore thumb if they're ever reintroduced (or still present on other platforms...)

  • traversal order: some operations have a net effect that's deterministic, but the order in which sub-operations was different; collecting a bunch of definitions by iterating over a hash table with maphash, or a set of storage classes with union, is going to be correct no matter the order, but different between different implementations of those functions. Also in this class, though not strictly speaking an issue of traversal order, was the dependence on the host's interpretation strategy: whether forms were always compiled or not, which influences the number of times gensym is called and hence the value of *gensym-counter*.

  • idiosyncrasies: working towards eliminating the effect of differences of interpretation in the standard. Most of these differences of opinion are legitimate: CLISP macroexpands more than SBCL does because it has an interpreter on by default, for instance; similarly, CLISP's compiler doesn't aggressively coalesce constants, whereas SBCL's does. Some of them are less so; CLISP, for instance, prints 'foo as "'FOO" whereas SBCL prints "(QUOTE FOO)" when *print-pretty* is nil. I think the standard mandates SBCL's behaviour, but for these purposes we have to work around it anyway.

In the process, I also fixed a terribly embarrassing bug in genesis, the Lisp application responsible for taking these output files from the cross-compiler and constructing a Lisp memory image file ready to start up. As has been discussed here in the past, the standard-mandated requirement on arrays in Common Lisp is not sufficient for us to be able to construct the image file in memory as an array of bytes: conforming implementations are permitted to impose a maximum array size as low as 1024 elements. Fortunately it is straightforward to implement a suitable data structure without this constraint; unfortunately, the implementation did not take sufficient care to zero-fill newly-allocated memory, and while most of the time Lisp implementations perform that zeroing, there are circumstances in which they don't.

All of this has now been merged to SBCL's CVS, and is awaiting the 1.0.28 release due on Thursday 30th April. The practical upshot? Well, apart from a certain amount of increased confidence in the implementation strategy, and perhaps soothing the nerves of extremely paranoid distribution package maintainers, these changes should make compilation of SBCL on new platforms more straightforward, as there are now two implementations capable of building SBCL which are themselves buildable starting just from gcc and system libraries: CLISP and Peter Graves' XCL.

Long time no diary: a partial explanation. (Warning: contains largely self-indulgent content not related to Free Software.)

My daughter Marianne was born on a highly auspicious date: the 8th August 2008, the most eightish date for 20 years. All went reasonably well; the paediatrician who checked her over at Newham General Hospital detected a heart murmur, but we were given an outpatient appointment a month away and discharged. It took a lot of effort and perseverance to establish breast-feeding, and her weight gain was slow, but she seemed otherwise to be healthy and doing well.

However, at the outpatient appointment at the cardiology clinic at Great Ormond Street Hospital, we were told that Marianne had a heart defect: Tetralogy of Fallot. The cardiologist immediately reassured us that this was an operable condition with extremely good prognosis, but that the operation wouldn't take place for a while, as the bigger the child the better the recovery from the necessary open-heart surgery. In the meantime, the combination of the ventricular septal defect (the ‘hole in the heart’ which caused the audible murmur) and the pulmonary stenosis (a narrowing of the pulmonary artery, the vessel responsible for taking deoxygenated blood to the lungs to be refreshed with oxygen), together with her partially-overriding aorta (the opening to the aorta being over the VSD) meant that Marianne's heart would allow deoxygenated blood to go around the body again without being refreshed with oxygen, and consequently would have to work significantly harder (leading to right-ventricular hypertrophy and also explaining her slow weight gain).

Although we had been reassured by the cardiologist, and also the GOSH cardiac liaison nurse we spoke to, the diagnosis was still a shock to me: I think I spent about three or four days being completely unable to think straight. But the mind is resilient, and after a little bit it settled down, with the reassurance that many medical professionals: health visitors, dieticians, nurses, cardiologists and general practicioners were available to help. Other sources of support were great as well: family, colleagues, friends and neighbours, and even groups on mumsnet were extremely helpful and understanding.

And so life continued. Many of the normal stages of child development happened: first smile, sleeping through the night, not sleeping through the night any more, playing with toys, discovering wrapping paper, sitting up, and beginning to eat food; all this punctuated by visits to the health visitor, cardiology clinic and paediatrics unit. And then, eventually, came the letter giving us a date for her surgery: Thursday 5th March.

First there was the pre-admission hurdle to jump: various measurements to make, various blood samples to take, and the consent form to sign. Except that we couldn't: no surgeon was available, as they were all too busy. Fair enough; we arranged to come in the following week for that. It was at that point, signing the consent form for my daughter's heart surgery, that I experienced shock and panic again, this time for the best part of a week: while the full repair for Tetralogy of Fallot is a relatively well-understood procedure (it's been performed for longer than heart-lung bypass machines have existed) the mortality rates are still of the general order of 1-2%. Because Marianne was outwardly healthy, it was very hard to believe wholeheartedly that this scary procedure was utterly necessary, and so it felt like I was needlessly making her play Russian roulette.

The 4th March rolled around, and we were essentially ready: bags packed, all urgent items at work completed, contingency plans in place. And then the hospital phoned: a more urgent case had come up and so there was no surgical team available for us; we were rescheduled for the following week. To be fair, we had been warned that this might happen – even on the day itself, so having a day's notice was a bonus. Still, there was a lot to unwind: not just the various plans, but also our own emotional preparedness. Back to the normal routine, with an extra week of voluntary quarantine for Marianne (coughs and colds noticeably impede recovery from operations).

There was to be no second stay of execution on the 11th March: we were given ‘starvation times’ for Marianne: no food after 2:30am, no breast milk after 4:30am, and no water after 5:30am. The alarm was duly set for 4am so that she could get a last stomachful. Then we were all set; we set off in plenty of time to get to the cardiac ward at GOSH for 7:30am.

Once there, Marianne was weighed, and then dressed in a hospital gown. She then played with a set of coloured wooden stacking rings, while Jenni and I debated the merits of the various colour combinations that she was inadvertently modelling – well, we had to talk about something. A surgeon came down to assess her: determining whether she was well enough for the operation to take place, and checking that she'd had nothing to eat or drink since the small hours. With a certain amount of humour, she drew a purple arrow on Marianne's chest: it is NHS policy that all surgical patients should be marked up before the operation, because there have been a few too many embarassing errors in theatre (amputating the wrong leg, and similar). As the surgeon put it, though, “there is only one heart”.

Then we waited for the most difficult part: taking Marianne to the anaesthetists and leaving her there. We were allowed to hold her while the nitrous oxide was given; it took hold quickly – without the wriggling which we had been warned might be a possibility. And then we left the room, because it was what had to be done.

Waiting while your child is having surgery is not a pleasant experience. Fortunately, the hospital is aware of this, and enforces a certain amount of activity; they require parents to sort out their hostel accommodation during this period, which involves bureaucracy and therefore takes time. Also, we hadn't had breakfast by this stage, so that was a good thing to do, and we had been given (not by the hospital!) £10 to convert into a grey book. Even extending all of these things as much as possible, it was only about two hours since we'd left Marianne (where the operation was meant to take about four hours in total), so it was an immense relief that, when we returned to the cardiac ward and a nurse was able to telephone the operating theatre, we were told that Marianne had been weaned off the heart-lung bypass machine successfully: in other words, her heart was beating again.

In fact, we were lucky: throughout our stay in hospital, we never really needed to be given any bad news. By the evening of Thursday (the day of her operation), she was taken off the ventilator, able to breathe for herself. The following afternoon, she was removed from intensive care, with her breathing and temperature no longer needing to be continuously monitored. Even the one complication, a buildup of air in her chest cavity, was rapidly dealt with, and her chest drain (a tube attached to suction, drawing fluid away from the chest), pacing wires (to correct any irregularity in the heart rhythm) and intravenous lines (used for delivery of pain relief, mostly morphine) came out at lunchtime on Saturday, leaving her with a single stitch (not the only mark, of course, but the wound on her chest is held together subcutaneously). On Sunday morning she was able to breastfeed, and she did so with enthusiasm and vigour; the nurses then removed her nasogastric tube, and told us that we were likely to be discharged the following day.

And so it proved: after one last echocardiogram, we were given some medicines (paracetamol and diuretics), an outpatient appointment for mid-April, and a discharge letter, and told to go home. And, not without some trepidation, but also with a great deal of optimism, so we did.

All of which is one way to say that, if I have dropped patches on the floor, failed to fulfil administrative duties, or ignored requests for assistance in whatever context: I apologise, but I have had other things on my mind. It seems to be common knowledge amongst parents that worry never goes away, but with luck the worries that I experience will be less acute for a good little while.

This entry is mostly a late report on the European Lisp Symposium, held a couple of weeks ago at the LaBRI in Bordeaux. I think both Pascal Costanza (Program Chair) and Robert Strandh (Local Organizing Chair) deserve a pat on the back or two, because everything seemed to go off without a hitch; an interesting programme, plenty of things to do, good food, and (best of all) bad weather, so it didn't feel too depressing to be in a lecture hall for most of the day.

I actually went to Bordeaux a couple of days early, partly to meet with old friends (a big thank you to Rachel and Tim Moore for putting me up for the week) and partly so that I could give a talk about my main research project to the Image et Son research group there. Some useful discussion ensued, which was good.

The first day of the symposium had a closed writers' workshop session, run in parallel with Birds-of-a-Feather sessions. I wasn't a writer (so I wasn't at the writers' workshop), but from what I heard it was a useful exercise for those present: the idea is that written work that is in progress is briefly presented to a audience of peers, who then discuss it without reference to the author; the author can then garner a lot of useful feedback for improving the paper. (There are also formal mechanisms for defusing tension, and the reason for closing the session is to make sure that everyone present is on the same level.)

The BoF sessions were probably the thing that need more care in future; they were organized a little bit at the last minute, so session leaders were unprepared (and there wasn't much of a choice: only one BoF at a time meant that people defaulted to that rather than anything else, quite understandably). I attended a session about concurrency, but without paying much attention as I rediscovered the fact that hacking for 45 minutes continuously is much more than three times as productive as hacking three times for 15 minutes. I skipped the next one to talk some more with Pierre Hanna and Matthias Robine about Music Information Retrieval (or “work work” as opposed to “play work”).

After lunch I tried to act as chair of a CLIM / McCLIM BoF session. The problem with it being the default choice was that we spent a lot of time on background rather than on discussing the future – not that that's too bad a thing when Scott McKay is sitting at the back of the room, because we could get a lot of background. The really unfortunate thing, caused mostly because of the short notice, is that almost none of the demos actually worked: I had a CLIM Listener that I couldn't type into; Troels managed to wedge SBCL's threads somehow, and even Andreas and his Classic CLIM demos managed to send keyboard focus somewhere. We agreed that the next release would come with working demos. Still, choice quote from Scott: “oh yeah, I wrote that!”

Apparently there was then a BoF focussing on replacing ASDF, which maybe got further because of better focus. I was in a meeting at that point, tentatively planning (with others) ELS 2009, which will be in Milan, hosted by Marco Antoniotti and chaired by António Leitão – put May 2009 in your diaries now! After that, we went to the mairie, technically as invitees of Alain Juppé, for drinks and a speech about how historical and forward-looking Bordeaux is. Nasty surprise: I got volunteered to do simultaneous translation, which is hard; I learnt afterwards that the mayor's underling who'd been volunteered to give the speech was also a first-timer, so that explains the utter confusion and sentences like “Bordeaux is nice” that came out.

The second day of the symposium was dedicated to the technical programme: a fine set of talks, I think. Mine (work done in collaboration with Jim Newton) suffered from being the third different talk I'd had to give in ten days; I think I was not as clear as I would have liked to be. But I did get some questions, including one from Didier Verna about what good class-eq specializers are, about which more in a later diary entry. I enjoyed Didier Verna's talk a lot; though it didn't tell me anything I didn't know, it was a clear presentation of a problem of expression and how to solve it using CLOS and the MOP; the talk by Sebastián González about Ambient Objects and Context-Oriented Programming was extremely well judged, I think.

And then it was all over bar the banquet, which was nice; plenty of useful discussions with colleagues past, present and future; good food, good wine (well, you'd hope so...) and good company. And then back home in time to perform in the English Music Festival and fall asleep, fortunately in that order.

I'm in Potsdam, at the Hasso-Platner Institute, attending the Workshop on Self-Sustaining Systems. Sounds esoteric? I thought so, too, though that didn't stop me from submitting a paper about the build process of Steel Bank Common Lisp and how that affects the user and developer community.

The call for contributions invited

submissions of high-quality papers reporting original research, or describing innovative contributions to, or experience with, self-sustaining systems, their implementation, and their application.
and my paper, as well as the technical details of the SBCL build, essentially argued for considering the user and developer community as part of the system: software is really just a parasite, but its host is not the hardware it's running on, but the humans who allow it to ‘reproduce’. (A related point was made by Ian Piumarta of the Viewpoints Research Institute, about the survival of the species rather than the individual being the important thing, in the talk before mine).

My talk was a bit rushed and confused, I think; it suffered from insufficient preparation time. What I was utterly flabbergasted by was the attendance: I was expecting a complement of about 12 people, maybe mostly the authors and their colleagues. Instead, there were about 70 people at the opening, and while some of those were local PhD students or other easy prey, there were still about 50 people attending the research talks. If I had expected that, I might have adjusted the material and the presentation a little bit, perhaps with a bit more advocacy (a captive audience is valuable, after all!), but at least in response to my audience poll, very few people confessed to being unfamiliar with Lisp, so maybe the advocacy would have been unnecessary.

In any case, I got some fair feedback, and I had the chance to have a nice chat with Kim Rose about the use of Squeak Smalltalk and related tools in education. Luke Gorrie introduced me to some other of his Smalltalk chums – that did seem to be the most popular language, based on a highly informal sample.

Dan Ingalls gave a demo of the Sun Labs Lively Kernel, which frankly makes my idea of writing a web backend for McCLIM simultaneously achievable and passé; a whole Morphic system built on SVG and Javascript, complete with animated stars and clocks with Roman numerals, is just the ticket at 9 o'clock in the morning. “Runs best in Safari 3 or Firefox 3.0 beta 5”.

The other talks were about implementing 3-lisp (a reflective dialect of Lisp), a Squeak VM in PyPy, Huemul Smalltalk, and Pico (an interpreted Lisp). It's very interesting to see the development of these systems; my main problem is that many of the presenters say things like “this implementation is not efficient (yet)” or “this implementation is incomplete at the moment” – this makes it very hard to judge whether the necessary future development will invalidate some of the nice things about the systems, such as their size, speed or dynamicity. I suppose that's in the nature of the research beast, though.

After all that, and some more chats, time to head off home (briefly, to rehearse for Arne's Judgment of Paris in the English Music Festival) before setting off again to Bordeaux and the European Lisp Symbolsium.

I spent the weekend in Amsterdam, doing ordinary touristy things on the days when I wasn't attending the 2008 edition of the European Common Lisp Meeting. I suspect that most readers of this diary are likely to prefer a summary of the ECLM to the details of my wanderings on the (rather cold) Saturday, so that is what I shall provide.

Walking through the streets of continental cities early (around 08:00) on a Sunday Morning is usually a pleasant experience – everything is closed, most people aren't even yet going to appropriate religious observances, and so you get an unimpeded view of the town, devoid of the hordes that can cause obstruction or aggravation. Unfortunately, Amsterdam is a somewhat popular destination for British ‘stag’ parties; and, indeed, as I was walking down Leidsestraat, two such parties, who did not seem to be the sort to be going either to an appropriate religious observance or to the ECLM, passed each other. Noise ensued.

On arriving at the Felix Meritis, at 08:30 on the dot, I saw a crowd of people, all seemingly with the same idea: to have the first coffee of the day. Fortunately, the doors opened shortly afterwards, and caffeination was allowed to take place. Conversation happened as well, though I can't remember very much of it, not yet having been fully invigorated by the coffee. Mostly I think I explained why I wasn't on the previous night's boat trip: a combination of somewhat tardy registration and a second mouth to feed...

And then the real action, the talks, got under way. Unlike many an academic conference, of of the ECLM's distinguishing features is that the talks are attended by the vast majority of the participants; there's much less of a feeling that people simply attend to chat to colleagues: probably partly because the talks are very much the ‘deliverable’ of the meeting (there are no proceedings or anything like that) but also, I think, because the talks cover interesting ground, and offer perspectives based on a solid amount of experience.

Jeremy Jones (from Clozure Associates) started the ball rolling, with a talk on the production of InspireData: an application built for data visualization in an (American, pre-University) educational context. Jeremy wisely started off with a demonstration of InspireData's features; it contains some very impressive-looking tools, and seems to present them to the user in a sensible way. I haven't tried it or gone beyond the demo, but it looks like a huge advance on, say, using an off-the-shelf spreadsheet program to do data analysis, even (dare I say it) for professionals – though whether it scales up to professional-sized data sets is another question. Some other take-home messages from Jeremy's talk: it is possible to sell shrink-wrapped software, even today, even written in Lisp; having a proper designer on the team (or as your client) can help enormously in producing a usable interface; and having a programmer as your client can be both a help and a hindrance – specify the acceptance process carefully.

Nicholas Neuss followed Jeremy, with a discussion of the FEMLISP framework for solving partial differential equations. Being in a somewhat darkened room, early in the morning at a weekend reminded me a little of my undergraduate days, where discussion of differential equations, fields and the like was par for the course – and the talk took me right back (in a good way). In particular, watching FEMLISP compute the eigenmodes of Lake Constance (surface waves, I think) was entertaining. One difference in kind between this talk and the previous is, I think, a function of the possible ‘market’ for the two tools; InspireData is sold in quantities of the order of tens of thousands at present, while, let's face it, FEMLISP is never going to have that kind of exposure: and so the resources aren't really available to make FEMLISP into a product usable by even other domain experts. Of course, the fact that Nicholas' boss makes a competing product might also have something to do with that...

There followed a talk about large Internet systems, from Stefan Richter. There was an interesting survey during the talk, asking people about the size of their userbase (assuming that they worked on web applications at all). An interesting distribution; certainly applications with millions of users are no longer rare – and Juho showed commendable restraint in not snorting something along the lines of “millions of users?” One other interesting moment was when Nick Levine stopped Stefan, to give him time to write down the long list of libraries that is already available to help build Lisp web applications, from the database to the front-end.

Kilian Sprotte gave the last presentation before lunch, talking about the GL-enabled Patchwork music visual programming system. The presentation unfortunately appeared to be a little bit unrehearsed, and so I'm not sure that Kilian got everything that he wanted to across to the general audience. From my perspective, though, it was sufficiently close to the day job that I could see the point (and catch some of the references; B-A-C-H and so on) – I'll be able to report back to people in the lab, who were asking about it, and maybe we can find some useful musical analysis algorithms in there. Also, it occurred to me that GSharp could usefully provide some import or export functionality for the chord and score editors, maybe through MusicXML, or maybe writing directly into the PWGL notations for notes (tighter couplings are likely to be hard, given PWGL's current non-Open-Source nature.)

Then we broke for lunch; Juho, Nikodemus and I ended up sitting on a table with Hannes Mehnert and Luke Gorrie; among other conversations, we had The Great SBCL Maintenance Debate, and we now have a proposed Plan. (I don't know if any other plans have been proposed, but hopefully soon there will be less uncertainty.) The rest of the lunch break was spent grilling Luke about OLPC and Kathmandu, and eating interesting interpretations of café food. (Pasta arrabiata with beans and squash? I don't think so.)

After lunch, a talk about cheap apartment design architect assistance software: Knowledge-Based Engineering, where in this case the knowledge base is about Norwegian building regulations. The idea of the House Designer product developed at Selvaag is to allow architects to experiment with designs, while tracking that building rules (both governmentally-imposed and the house style) can be accommodated, along with all the necessary pipes and electrics and so on. It was good to see a variety of techniques on display, and I liked the flashes of humour: for instance, that the user-interface group asking for XML descriptions was OK by the Lisp group, but that then the user-interface group wanted to send XML back, and that was not OK. It's good to know that Frode Fjeld has a Lisp job, too; he was solidly namechecked in the presentation.

Then Juan José García-Ripoll came to the stage to talk about ECL design and implementation. There were some wry moments for me there; starting with the observation that he wasn't really a computer scientist at all, but rather a physicist. Also, many of the motivations for the ECL design appear to be direct analogues of some of SBCL's PRINCIPLES: completeness, clean bootstrappability, and preferring maintainability over whizzy features, for instance (I hope I'm not mischaracterizing here; it's possible I'm just evaluating what he said through an SBCL-tinted lens). Given that, I think it's interesting how different the two systems look, maybe just from having different starting points?

After the final coffee break, yet more talks. Marc Battyani, of FractalConcept HPC Platform, talked about using Lisp to program FPGAs for custom high-performance solutions; in particular, applications in the financial world, such as derivative valuations and automatic share trading platforms. (A word of advice to Marc: it's not often that you get to say that you're 1000 times as fast as your nearest competitor, nor that you can process packets faster than the test network can send them to you – so don't let that message get hidden by the constant fumbling for a particular Microsoft Image and Fax viewer window!) The products he has look interesting; best of luck to him for finding a buyer in these more financially-challenged times...

And finally, the pièce de résistance: Kenny Tilton took the stage, to give “a rant on the state of Lisp and Lispniks touching on Algebra software, Lisp libraries, Open Source, Cello, Cells, and somewhere along the way introducing Triple-Cells, animated data modelling with persistence for free”. Unfortunately, it seemed that we weren't going to get this rant; instead, we got a small demo of the Algebra tutoring software, along with some discussion of the dataflow paradigm that Kenny believes is central to all simple applications, and a fair number of sound effects. Nice anecdote about Lisp and speeding tickets, though.

Then it was all over bar the dinner; I chatted to Pascal Costanza and Charlotte Herzeel about the busy workshop season, to Jeremy Jones and Marco Baringer about the (lack of) checkin policy to SBCL's CVS, to Edi about how much Heathrow Airport sucks, among many conversations. As we were about to call it a night, Nick mentioned that the Lambda Express had some spare tickets, and that we could probably hitch a lift back to London by train (rather than by plane to Terminal 5...), so we arranged to meet Dave Fox and his crew the following morning. While on the train, in a spirit of cross-implementation co-operation, we essentially finished the Araucaria from the Saturday Guardian – documentary evidence will be forthcoming. And then it really was all over.

Long time no blog. As ever, there are a number of reasons for this: complicated employment situation; too much work to do... but there's also the slightly addictive personality to consider. Since I've already been here once before, it should have come as no surprise to me that I could become engrossed in beating a roguelike game; in the end, I did beat Tales of Middle Earth (ToME) into submission, as I did with Nethack two years ago. (Two years? Well, consider that the only solid playing time is around Christmas, and then learn that last year I almost beat ToME into submission...)

“Complicated employment situation”, I hear you ask? The life of a junior academic researcher, while rewarding, often feels precarious: my main project research funding ran out at the end of May, and since then I have been employed on a succession of temporary research contracts. One thing this has led me to discover is that I am just not very good at having the next thing lined up before the current thing expires. Another is that I do not like the scary feeling of not knowing what I am doing next month – or perhaps more pertinently, where the next paycheque is coming from.

The good news is that this has now resolved itself; I have accepted a lectureship at Goldsmiths, a part of the University of London. (For readers not fully versed in the UK Higher Education terminology, this is a Real, Permanent Job involving a mixture of teaching, personal research and filling in lots and lots of forms.) I work in a group specializing in investigating sound and musical processes, from analysis and synthesis, through cognition, designing systems for cataloguing musical artifacts, all the way to automated Music Theory.

So what do I do there? My primary research responsibility is to the OMRAS2 project, which aims to do all sorts of clever things for recognizing and searching music: current fingerprinting services work on exact (but possibly degraded) copies of notionally entire tracks, whereas I have an interest in automatically deducing musical structures both from audio and from notation, and in working with approximate kinds of similarity (detecting that a track is a remix of another, or a cover version, and so on).

One of the reasons that I like this group so much, though, is that when I first met them, one of the PhD students (who is still with us, in a more advanced capacity) was working on his laptop using CMUCL and ILISP (this was late 2003, so SLIME was if not a glimmer in the mind of Luke Gorrie, at least not the clear choice for development on Free Common Lisps as it is today). This means that my interests in the engineering and language design aspects of SBCL development can go hand-in-hand with attempting to come up with protocols for interacting with musical corpora in many, many different formats: attempting to make it so that the same routines can operate both on representations of lute tablature and on a database of MIDI files designed for karaoke machines, being used to investigate memory for music. I finally have the excuse to use gsharp at work, too!

Speaking of gsharp, Brian Gruber handsomely completed his Summer of Code project, to add MusicXML import and export facilities. His work has been integrated into gsharp's CVS repository, and is being used and further developed here at Goldsmiths, and I got a T-shirt from Google. If you're reading: thanks, Brian.

It's not quite “all quiet on the SBCL front”, but all of the rest of my life leaves me with less time to spend on it than I would like. I did manage to restore support for the alpha architecture, which had bitrotted a little in the last year; this puts me in a reasonable position to finish up my work on improving the interaction between modular arithmetic and representation selection. In addition, I have been pushing forward a little bit the work that I reported on at ECOOP last year regarding user-defined subclasses of specializer; stay tuned for more details. (Maybe sooner than in six months' time!)

It's been a while. Lots of interesting[*] stuff to talk about. Since my previous diary entry was about my work for the European Lisp Workshop, why not start with my recollections and observations about that?

The workshop itself was one day long, and had one invited talk, five papers presented (one of which was mine) and a more informal demo / presentation of current advances in ContextL (a Lisp implementation of Context-Oriented Programming), given by Pascal Costanza.

The invited talk was given by Alexander Repenning, and was titled “Antiobjects” (and subtitled “Mapping Game AI to Massively Parallel Architectures using Collaborative Diffusion”, for what that's worth). The central idea, I think, was to convert a goal-oriented architecture – expressing behaviour of agents in the system in terms of what their goals are, in what is maybe the typical object-oriented programming style – into an optimization problem, viewing the system as a whole.

This change of perspective (the “Antiobjects” of the title allows for some interesting emergent properties. The compelling example Alexander gave was for ghosts chasing Pacman: the difficult bit in the traditional object-oriented style is in implementing collaborative strategies: if there are two ways to reach Pacman, and two ghosts, you (the evil games programmer) would like the two ghosts to take one way each. Generalising for large values of “two” is tricky. The Antiobject solution is instead to define a Pacman field, which diffuses away from the Pacman source, and have the ghosts be sinks of that field. Then the ghosts simply need to do hill-climbing of the Pacman field, and the desired behaviour of collaborative-Pacman-eating appears as if by magic.

Sébastien Mosser gave a talk about meta-models which, I have to confess, I didn't understand at all. I don't know what a “model” is, let along a “meta-model”, and unfortunately Sébastien didn't really give me an idea why I should care. I welcome enlightened discussion on this, if I should in fact care. The next presentation, reporting work by Pierre and Simon Thierry, was a general introduction to transparent persistence, which (as Pierre said) is almost a textbook example of using the Metaobject Protocol. The picture is slightly clouded because unfortunately update-instance-for-redefined-class is not quite able to cope with multiple class redefinitions in between accesses to instances. I believe I have a solution for this issue, but the margin of this blog is too small to contain it. (In practice, people work around the problem without too much pain: I discussed this at the workshop with not only Pierre, but also Arthur Lemmens of rucksack fame.

After lunch, I gave my talk about user-extensible specializers, details about their implementation, and examples of their use. In the course of giving examples, I inevitably made the mathematician's mistake of attempting to persuade the audience that simplifying (* _x 0) to _x was a good idea, but apart from that I don't think I embarrassed myself too much. What's interesting is that Jim Newton's talk, scheduled immediately after mine, was about an application of user-defined specializers, though in the SKILL (or maybe SKILL++, I forget) language of Cadence rather than in Common Lisp. (For the Common Lisper who is intrigued at seeing an application for user-defined specializer classes, as well as reading Jim Newton's paper I can recommend getting my implementation of the ideas by doing darcs get http://common-lisp.net/~crhodes/vclos, and playing around with the source files and examples therein.

After that, Nikodemus talked about his work on making the various caches in SBCL's implementation of CLOS thread-safe; we got to see some current directions in Context-Oriented Programming development (as well as news that a very few people are actually using it in the wild!), and then it was off to dinner, where I manage to surprise Edi by telling him that his webserver was too complicated, and David managed (not entirely) to surprise me by not talking at all about putative CLIM backends...

[*]In My Opinion.

123 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!