Older blog entries for crhodes (starting at number 119)

We're close two working weeks into Google's Summer of Code, and the student I'm mentoring, Brian Gruber, is making good progress in his project to add MusicXML support to Gsharp – sufficiently good to blog about it, in fact.

Discussing things with Brian, and fixing some of the embarrassing bugs in gsharp that he found, has also encouraged me to tidy up a little some of my accumulated divergences from the Gsharp CVS and commit them; in particular, the lack of line- and page-based motion commands was a significant annoyance last time I edited a score in Gsharp, so I spent an evening working out a sufficiently good implementation, and now C-a, C-e, C-x [ and C-x ] work as an emacs user might expect. (It's ever so slightly trickier than it might seem, because there's no information in the gsharp buffer recording layout information per se; the gsharp buffer contains a complicated data structure representing each individual musical (or graphical) object, but the page and line layout is computed separately – so motion by layout element needs additional computation).

Other things coming soon to a CVS repository near common-lisp.net, to whom thanks go for their hosting: correct placement of whole-bar rests (in the middle of a bar, if they are the only element in their layer in that bar); a hacky printing facility; maybe even a reworking of the currently-useless panes on the right-hand side of a gsharp window...

In my previous diary entry, I somewhat sarcastically obeyed the letter of the BSDish-with-advertising licence of cl-pdf. Since then, Marc Battyani wrote to me to say that he was removing the advertising clause from the licence of cl-pdf (and cl-typesetting), replacing it with a request (not part of the licence) to acknowledge its use in significant applications, to assist in promoting the Common Lisp language. Thanks, Marc, both for the software and for the friendlier licence.

Some people have expressed mild interest in the code used to produce the name badges for ILC 2007. I used cl-pdf to generate PDF, and otherwise there's only a very little bit of cleverness to get nicely filled and indented lists of features and similar. Feel free to download it and use in whatever way you wish, subject to the licence of cl-pdf (which seems to have an advertising clause, so I must say of defperson.lisp that This product includes cl-pdf for generating documents.)

As I wrote in my previous entry on ILC 2007, I was away from the conference on Tuesday. From talking to people whose judgment I trust, it appears that I missed some spectacularly awful stuff; however, now is probably not the time to comment further publically on such matters. Of the good stuff, I was most sorry to miss Richard Jones talking on Garbage Collection; having the book and the paper that Richard wrote for the proceedings aren't a substitute. Apparently, Charlotte Herzeel made some people begin to see some point in Aspect-Oriented programming, too, which is no mean achievement given that the usual response is that it's just one sufficiently-advanced method combination (well, it is, but it's also using it pervasively).

Anyway, Wednesday beckoned. The start at an ungodly hour was redeemed by Manuel Serrano's demonstration of HOP, an environment for programming the Web. I don't really live in the Web programming sphere; it doesn't interest me a great deal: but the presentation was excellent, with well-timed automatic slide changes, fun demos, and a clear explanation of the problem space; I don't know whether having the need for a proxy server will render it difficult to deploy in practice. Good stuff anyway; in fact, it worries me ever so slightly that the best of the invited speakers for me – Sperber, Queinnec and Serrano – are very firmly in the Scheme camp.

After that talk, I found speakers failing to command my attention quite as much as the impending delivery of my own, about extensible sequences; I did enjoy Geoff Wozniak's about dynamic data types, defining a protocol whereby instances would be able to change their own implementation details based on runtime profiling data; I think I need to look at the details of his ideas.

Then the time came for me to stand up, and – wouldn't you know it? – the laptop refused to display simultaneously on screen and on projector. Which is ridiculous, because it was fine on Monday for Herbert Stoyan's talk. I bravely soldiered on without the comfort blanket of my notes in front of me, and I don't think that any problems in the reception of my talk were particularly because of that. In retrospect, the example sequence class I used in the talk was too simple, and I didn't give compelling-enough other concrete examples; fortunately, I was able to make up for some of that in the coffee break afterwards, talking to a couple of people (and Cyrus did a sterling job talking to Jans Aasman about his space-efficient implementation of strings of DNA, and Jans then started talking about treating memory-mapped files as sequences, so I think he probably got the idea).

Then it was all over bar the Liskell talk (not particularly interested, though good on Clemens for having the gumption to stand up there and tell it like he sees it), the Franz programming-in-the-large talk (Jans is a fun speaker; I thought that Luke asked a rather good question at the end about the expected reliability of garbage collection), and a little bit (fortunately, not a lot) of self-congratulation at the end. Dinner at the Hotel Felix with, among others, an exhausted local organizer wrapped up the day, and I was homeward-bound twelve hours later.

So, that was Monday. (The observant will realise that this is Tuesday^WWednesday; since I'm not at ILC on Tuesday this still counts as being on time). I got asked^Wsuckered into chairing the first session, which inevitably involved faffing around with projectors and broken laptops, and I didn't get time to eat the croissant that I had thoughtfully purchased on the way in to the Law faculty. So, I chaired with a rod of iron, so that we could get to lunch on time: but Clare staff weren't ready for us. Grr.

Oh, the talks? Well, I'm probably not the best judge of the early talks; a breakfastless chair is not a happy one, but also I couldn't see the slides from where I was sitting. Herbert Stoyan's discussion about the early days of Lisp history was fun; I was less excited about the source to source semantics-preserving compilation that he discussed – I didn't really understand why it was relevant. This was actually a common theme in the talks: the speakers rarely succeeded in explaining what the problem that they were addressing was. So Stephan Frank didn't really explain why constraint propagation was important or interesting; I'm not a screamer user, so comparisons involving it ("it's like screamer, but faster") didn't really help. Still, when he's demonstrating benchmarks, it's nice that the fact that he used SBCL is accepted as a matter of course.

Similarly, the Dylan DSL talk (by Hannes Menhert and Andreas Bogk) didn't really state up front that part of what they were addressing was the lack of procedural macros in Dylan. They got to it eventually ("we rely on the compiler to constant-fold this 1+1+1+1+1+1+1+1+1+...") but if you hadn't read the paper in advance, that might not be obvious. The demo was good, but I found the rest of the talk a little bit disappointing: lots of explicit examples of macros which would be a lot easier to write in Common Lisp. Hey ho.

Continuing the theme of non-explanation, we got a talk on FREEDIUS, a reimplementation of 3DIUS funded by SRI. We had to infer that the interesting stuff was nifty 3d scene reconstruction and object detection and interaction; obviously, the demos made that rather clearer. It was somewhat heartening to see that McCLIM is not the only thing that suffers from redisplay glitches: the demos indicated that there's clearly an OpenGL (or similar) buffer being held on to for slightly too long.

Then we broke for lunch, and I was allowed to give up my chair. Hooray. We had an ESA talk, which I basically knew about; an LTk talk which I'd basically heard the entirety of in Hamburg, and a talk about gene patterns and pixel-based clustering which was cool and new; it's a bit of a shame that Cyrus ran out of time to talk about the lisp-side stuff, but on the other hand I am of the opinion that too much "this is lisp, isn't it amazing" is bad for the brain.

Then we discovered that Antonio Leitão had not arrived. So we had a long coffee break. Awesome! Also awesome was Christian Queinnec's talk about teaching Scheme to undergraduates (800 at a time: there were some very interesting effects resulting from automated grading systems, including an equalisation between the sexes' performance, in UMPC's implementation at least), and Michael Sperber's talk about R6RS was good too: clearly the editors have suffered in the name of standardization...

And then I had to head off to London, only returning in time for Tuesday's banquet. Which was excellent, and great fun. Now all I have to do before tomorrow is to write my own presentation. (And get some sleep.)

ILC 2007. Hooray, it's actually international—that is, it's not in the USA. In fact it is in Cambridge (the real one), and it is going swimmingly. Well. It's certainly nice to have of the order of 120 interesting people around; there's a good breadth and depth to the field, what with academics, hobbyists, and the occasional person who uses lisp for Real Work: sometimes even SBCL explicitly. The organization has been impeccable so far, so well done Ravenbrook, and particularly Nick Levine.

Today (Sunday) was tutorial day; I have to say, I found the tutorials disappointing. In the morning, I went to the Memory Pool System tutorial. While I couldn't resist a snort of derision at the news of the failure of MPS/CMUCL integration ("we couldn't work out how to build it with trivial source changes, let alone a new allocator"), I was more disappointed with the disorganization of the tutorial—which was probably exacerbated by the tendency of some in the audience to indulge in reminiscences and anecdotes about development avenues of the 1980s. There was some interesting stuff: I was particularly impressed by the notion of reference rank, and I also need to think more about the five-phase collection algorithm. Still, I spoke briefly afterwards with Richard Brooksby, and I think I know what to do to help them to move forward with their ideas.

In the afternoon, I started off by going to the tutorial about Debugging and Optimizing in Allegro CL; however, I confess I didn't get very much out of it: I already know what it's possible to do with a bit of assembly (or pseudo-assembly), and it doesn't really look like they are so far ahead of other ways of doing things. There are only so many ways you can convince gdb to do something useful, for instance. So for the second half of the afternoon I went to Pascal Costanza's tutorial about ContextL, which was at least challenging and thought-provoking. (And Pascal had clearly done enough preparation not to be left floundering...)

Under my temporary stewardship, Luke Gorrie's clbuild now has support for two new applications: the Climacs text editor and a graphical front-end to mplayer, climplayer. In addition, it supports using OpenMCL as the host lisp in addition to or instead of SBCL; an option for using the GTK+-based McCLIM backend rather than the default X11 (CLX) one; and a bugfix to the build (making sure all of the source files for the beirc IRC client are compiled).

All of this clbuild work was done by other people: Eric Marsden, David Lichteblau and Anthony Chaumas-Pellet. Go them.

Luke Gorrie is going on holiday for a little while. I sent him some bug reports and patches for clbuild, and his response was “Care to fork it for a little while?” Typical.

  darcs get http://common-lisp.net/~crhodes/clbuild

for your clbuild needs, until Luke returns from whatever sunny/snowy/rainy clime he will be enjoying.

One of the things I like about X11 is that it's not the library API which is specified, but the wire protocol. This allows building of abstractions in an idiomatic way for every given language, with none of the penalties involved in layering atop an existing foreign library – which penalties can be both from impedance mismatches between the implementation languages, and from maintenance headaches should the details of the library change. Instead, the X11 wire protocol is extensible (through server extensions) in a backward-compatible way, so code written for old versions of the protocol continues to work, if perhaps less beautifully.

So, the Common Lisp community has inherited some work done by Texas Instruments in 1987 to write an interface, named CLX to the X11 protocol. Generally, the interface is quite idiomatic and naturally Lispy; anecdotally, it's nicer to use than Xlib (a C binding) – I haven't heard enough about the new C bindings (XCB) to judge.

The downside of this language neutrality is the possibility of not having enough eyes to find bugs and hands to fix them: given that very few people use CLX, and fewer still know much about the X11 protocol, problems can languish unfixed for a long time. One such was exposed by the McCLIM backend using freetype and the RENDER X11 extension for font rendering: an attempt to display strings of more than 254 characters in length would generate nasty-looking X errors (BadGlyphSet errors for strings of 255 characters, followed by LengthError for longer strings).

From inspecting the X server source code, it would appear that this is because the encoding of glyph sequences on the wire for the RenderCompositeGlyph32 request (and its 8-bit and 16-bit siblings) seems... how can I put this... odd. The protocol documentation is unclear (to me) on this issue, but it seems that there is an 8-bit length field for a sequence of glyphs from the current glyphset, and that 255 is an escape code in that length field to indicate a change of glyphset. I don't know why it was done this way, given that there are then three spare bytes which are unused by this protocol – maybe no-one anticipated that displaying strings of 255 or more characters might be desired? In any case, this, coupled with a naïve interface to the protocol request, explains the errors we were seeing, as attempting to render a string of 255 characters triggered the escape code, meaning that x- and y-offset values were in fact interpreted as a glyphset – unsurprisingly, an invalid one. Strings longer than 255 characters would have only the low 8 bits encoded in the length, and so would generate length mismatch errors. Hooray, everything makes sense.

I've constructed a putative fix for this in CLX, and sent it to the mailing list. It's not quite as efficient as it could be, and there will no doubt be extra hair as and when a :translate keyword argument is added to that function, analogous to the functionality for draw-glyphs, but it seems to work on the limited testing I've given it. I'd appreciate comments and corrections from people who actually know X11 and/or CLX, though.

As I discussed previously, I submitted a document (CDR 3)to the Common Lisp Document Repository. It's now getting closer to the time when that document needs to be finalized or withdrawn; I have some minor changes to make to it, but if there are any other issues with it they need to be brought up soon. (There has been some discussion on the cdr-discuss mailing list, as well as some discussion of CDR 2: this is encouraging).

The work that inspired that CDR submission was work I've been doing to allow extensions of the sequence type in SBCL. I've now merged a preliminary version of that work into SBCL's CVS. It's currently undocumented; although I expect the final version of this to be very close to what is in CVS now, I'd like to verify that other people find it possible to use the protocol.

Instead of documentation, which will be forthcoming once some details about publication are sorted out, I posted an example of using the protocol to the development mailing list. The basic idea is that there are five protocol generic functions which need methods for a new sequence class; once those methods are implemented, all of the Common Lisp sequence functionality will work on instances of the class (so functions like find, position, delete-duplicates – in fact, all functions from the Sequences Dictionary, as well as a few stragglers (the quantifiers such as some and every, and the read-sequence and write-sequence I/O functions).

However, the default implementation of iteration over sequences in my protocol is based on accessing the sequence by index; for sequence types such as a doubly-linked list, both forward- and backward-iteration can be implemented more efficiently; there are some extensible generic functions for performing that optimization. Additionally, some standard functions (like nreverse on doubly-linked lists) can be implemented more efficiently than through the iterator protocol, so there are likewise some extensible generic functions for expressing that.

I look forward to receiving feedback, both on the CDR and on the extensible sequence facility...