Older blog entries for crhodes (starting at number 113)

Under my temporary stewardship, Luke Gorrie's clbuild now has support for two new applications: the Climacs text editor and a graphical front-end to mplayer, climplayer. In addition, it supports using OpenMCL as the host lisp in addition to or instead of SBCL; an option for using the GTK+-based McCLIM backend rather than the default X11 (CLX) one; and a bugfix to the build (making sure all of the source files for the beirc IRC client are compiled).

All of this clbuild work was done by other people: Eric Marsden, David Lichteblau and Anthony Chaumas-Pellet. Go them.

Luke Gorrie is going on holiday for a little while. I sent him some bug reports and patches for clbuild, and his response was “Care to fork it for a little while?” Typical.

  darcs get http://common-lisp.net/~crhodes/clbuild

for your clbuild needs, until Luke returns from whatever sunny/snowy/rainy clime he will be enjoying.

One of the things I like about X11 is that it's not the library API which is specified, but the wire protocol. This allows building of abstractions in an idiomatic way for every given language, with none of the penalties involved in layering atop an existing foreign library – which penalties can be both from impedance mismatches between the implementation languages, and from maintenance headaches should the details of the library change. Instead, the X11 wire protocol is extensible (through server extensions) in a backward-compatible way, so code written for old versions of the protocol continues to work, if perhaps less beautifully.

So, the Common Lisp community has inherited some work done by Texas Instruments in 1987 to write an interface, named CLX to the X11 protocol. Generally, the interface is quite idiomatic and naturally Lispy; anecdotally, it's nicer to use than Xlib (a C binding) – I haven't heard enough about the new C bindings (XCB) to judge.

The downside of this language neutrality is the possibility of not having enough eyes to find bugs and hands to fix them: given that very few people use CLX, and fewer still know much about the X11 protocol, problems can languish unfixed for a long time. One such was exposed by the McCLIM backend using freetype and the RENDER X11 extension for font rendering: an attempt to display strings of more than 254 characters in length would generate nasty-looking X errors (BadGlyphSet errors for strings of 255 characters, followed by LengthError for longer strings).

From inspecting the X server source code, it would appear that this is because the encoding of glyph sequences on the wire for the RenderCompositeGlyph32 request (and its 8-bit and 16-bit siblings) seems... how can I put this... odd. The protocol documentation is unclear (to me) on this issue, but it seems that there is an 8-bit length field for a sequence of glyphs from the current glyphset, and that 255 is an escape code in that length field to indicate a change of glyphset. I don't know why it was done this way, given that there are then three spare bytes which are unused by this protocol – maybe no-one anticipated that displaying strings of 255 or more characters might be desired? In any case, this, coupled with a naïve interface to the protocol request, explains the errors we were seeing, as attempting to render a string of 255 characters triggered the escape code, meaning that x- and y-offset values were in fact interpreted as a glyphset – unsurprisingly, an invalid one. Strings longer than 255 characters would have only the low 8 bits encoded in the length, and so would generate length mismatch errors. Hooray, everything makes sense.

I've constructed a putative fix for this in CLX, and sent it to the mailing list. It's not quite as efficient as it could be, and there will no doubt be extra hair as and when a :translate keyword argument is added to that function, analogous to the functionality for draw-glyphs, but it seems to work on the limited testing I've given it. I'd appreciate comments and corrections from people who actually know X11 and/or CLX, though.

As I discussed previously, I submitted a document (CDR 3)to the Common Lisp Document Repository. It's now getting closer to the time when that document needs to be finalized or withdrawn; I have some minor changes to make to it, but if there are any other issues with it they need to be brought up soon. (There has been some discussion on the cdr-discuss mailing list, as well as some discussion of CDR 2: this is encouraging).

The work that inspired that CDR submission was work I've been doing to allow extensions of the sequence type in SBCL. I've now merged a preliminary version of that work into SBCL's CVS. It's currently undocumented; although I expect the final version of this to be very close to what is in CVS now, I'd like to verify that other people find it possible to use the protocol.

Instead of documentation, which will be forthcoming once some details about publication are sorted out, I posted an example of using the protocol to the development mailing list. The basic idea is that there are five protocol generic functions which need methods for a new sequence class; once those methods are implemented, all of the Common Lisp sequence functionality will work on instances of the class (so functions like find, position, delete-duplicates – in fact, all functions from the Sequences Dictionary, as well as a few stragglers (the quantifiers such as some and every, and the read-sequence and write-sequence I/O functions).

However, the default implementation of iteration over sequences in my protocol is based on accessing the sequence by index; for sequence types such as a doubly-linked list, both forward- and backward-iteration can be implemented more efficiently; there are some extensible generic functions for performing that optimization. Additionally, some standard functions (like nreverse on doubly-linked lists) can be implemented more efficiently than through the iterator protocol, so there are likewise some extensible generic functions for expressing that.

I look forward to receiving feedback, both on the CDR and on the extensible sequence facility...

wnewman released SBCL 1.0 yesterday.

This has been a long time coming; Bill announced the existence of SBCL to the world on the CMUCL development mailing list in December 1999:

Date: Tue, 14 Dec 1999 18:53:46 -0600
From: William Harold Newman <william.newman@airmail.net>
To: cmucl-imp@cons.org
Subject: It's alive! (SBCL, a CMU CL variant which bootstraps cleanly)
Message-ID: <19991214185346.A1703@magic.localdomain>

Since then, a large number of people have worked on the system (including me since at least October 2001, according to this diary entry); it's been a pleasure to work with so many bright and motivated people in a kind of directed anarchy: towards goals that we mostly shared, even if we differ in the details. Since branching off, we've developed support for Unicode, native threading, 64-bit architectures and executable delivery; people have contributed support for esoteric (and sometimes even ubiquitous) platforms; we've removed an interpreter, leaving us with only an interactive compiler — and then developed another interpreter to put in its place. Some of the development team have found employment to enhance the system, while others have moved on to pastures new; users have been encouraged to adapt the system to their needs, many of them contributing enhancements back to the mainline codebase; startups are using it as part of their toolchain; established companies too, for application development and deployment.

I've learnt a lot while being a part of this, and I hope to learn more in the future.

As our release manager says, we have released McCLIM 0.9.3, codenamed “All Souls' Day” (though the observant will spot that there was a certain amount of confusion over the name). Others have posted about what this release contains, so I'll just say that I enjoyed not having very much to do with the release process itself, performing minimal tests but otherwise letting the wheels tick on around me. Thanks, Andreas, and more of that next time, please!

I'm testing the waters of the Common Lisp community's willingness to break with the past. The ANSI CL Standard [*] is often touted as a triumph and a great advantage for the Lisp community, and rightly so, I think: it is the result of years of experience of Lisp vendors and developers, and by and large the standardized functionality is harmonious, well-thought-through and consistent.

I've mentioned here before some unintended consequences of the standard, such as arrays specialized to hold nothing, and type upgrading being required to solve the halting problem; while working on a language extension that the standard was careful to permit implementors, I stumbled across another such oddity.

The sequence type in CL is carefully specified not to be necessarily just (or vector list); implementations are permitted to offer non-standard sequence types. However, while tidying up one corner case in the make-sequence function, I believe the standardizers overreached and created another such corner case: forbidding implementations from allowing natural extensions to make-sequence.

Rather than just ignoring this issue, though, or putting it up on the CLiki page, I wrote it up properly and submitted it to the Common Lisp Document Repository where it's been accepted as CDR 3. That's not really to say that I think the CDR has more visibility than CLiki – truth be told, it seems to have somewhat less, at least judging by the mailing list subscriptions. However, maybe the implied permanence of the CDR will encourage more users to demand that their implementors pay attention to it, or the implementors themselves to follow the documents and adopt sensible recommendations.

Why would an implementor support a CDR that explicitly involves no changes to the implementation? Well, one issue is that it warns users that some of the corner cases might change in behaviour (I give an example in the CDR document); but more importantly it can be a statement of intent: a declaration that bugs in the standard will be treated as bugs, and that new material can supersede old where the cost to users and implementors is low.

[*] Don't follow that link; it's just the most authoritative place I can find. Go to the HyperSpec instead.

Rumours of my disappearance from the internet are greatly exaggerated. Though an unfortunate series of events, juxtaposed in time, might have fooled the casual observer: the announcement of the read-only nature of advogato coincided neatly with my alma mater's discovery that I was no longer entitled to an e-mail address. Sorting out the e-mail thing took priority, so it was a pleasant surprise that when I started investigating the web diary situation I was informed that advogato was saved; thanks to all responsible for that.

Lisp stuff continues apace. SBCL is planning to release something called “version 1.0” next month; it would be nice if I could land my dangerously experimental branches before then, so that all the extra users of something blessed as version 1.0 can find the bugs for me. There will be a Lisp Conference in Cambridge next year, and I should probably aim to get my act together and submit something there (and so should you!).

(Work stuff continues too. Lisp is beginning to entrench itself there; ironically, I'm probably the researcher who uses Lisp least in the group that I work in – apart from providing technical support, of course. This will pass, eventually.)

I just spent some time making anonymous classes work in SBCL. It turned out to be harder than I thought, so it might be worth mentioning some of the nasty details for posterity (or, alternatively, so that I can bask in the appreciation of my indomitable spirit).

The description "anonymous" applied to classes, in the terminology of the Art of the Metaobject Protocol, is in fact a misnomer, because in fact anonymous classes typically have names (that is, cl:class-name returns something), but not proper names. So, creating an anonymous class can be done with (make-instance 'standard-class :name 3), and class-name on this class would return 3; (find-class 3), however, would return an error by definition.

Regular ("nymous", if you will) classes are specified by AMOP to be created by calling the function ensure-class, which itself ends up calling ensure-class-using-class. At some point in these calls, (setf find-class) must be called, to arrange for the class to have a proper name; it must certainly be called before class finalization, because it is entirely reasonable for a MOP user to want to call find-class from within methods on finalize-inheritance. In SBCLs prior to those of a week ago, which finalize classes eagerly, we dealt with this requirement by setting find-class as soon as the class was created; this is obviously incompatible with anonymous classes.

So to support anonymous classes, while still allowing classes with proper names at class finalization time, we have to delay class finalization. That's fine; it's explicit in the MOP description that finalization can be as late as creating the first instance. But delaying finalization turned out to unleash a hydra, with new heads growing every time I fixed an assumption in the code. If classes are finalized as soon as possible, then other codepaths – such as discriminating function caches, the type system, and so on – must be updated to deal with as-yet-unfinalized classes. This ended up taking a long time: even though the final patch only touched some 300 lines of code or so, it's depressingly easy to get the code into a state where it doesn't boot (and that situation is hard to debug, given that much of the printing machinery depends on a working CLOS implementation...)

The good thing about having a moderately-extensive test suite (24kloc of tests for 150kloc of code) is that it catches a lot of mistakes when changing things, and also documents what is expected to work. (The downside, or possibly "another good thing", is that it is quite hard to satisfy all those constraints with a random walk through codespace, even if it's directed.)

Robert Strandh is that ideal colleague: bright, hard-working and tenacious. Which means that when he says he's going to do something like reimplement enough of MetaFONT to support GSharp's needs, it gets done. The problem with the previous iteration of gsharp's display routines, which used metafont directly, was that there was no way of reconstructing the original curves from the resulting pixmaps, and so no way of reusing the glyphs for printing.

A small amount of cargo-culting (on my part) later, taking advantage of the antipodean night, and gsharp is capable of printing, up to a point. Visible deficiencies in this output include misplacement of minim and semibreve rests, too-small dots, fairly ugly bass clef, tie placement oddities, slight misalignments of stems and noteheads, discontinuities in the flag glyph, greying out of three of the lines, and of course the blissful unawareness of how big a page is. Oh, and sloping beams don't work at all; I had to break the beams to produce that output (which is the opening to Les Cris de Paris by Clément Janequin, incidentally); slightly ironic given that only recently I was hacking on making beams remember their clipping region (for partial-beam repainting).

So, I think Gsharp's future is bright, both because of the new functionality (printing! scrolling! Oh my!) and because the new functionality is accompanied by new hackability; it's likely a lot easier to debug the visual glitches now that the glyphs are generated by lisp itself, and supported by a glyph viewing tool that Robert wrote. Hopefully this will also draw in one or two more interested parties to add the functionality that they need.