Advogato blog for crhodes

22 Apr 2008

Tue, 22 Apr 2008 14:25:43 GMT

I spent the weekend in Amsterdam, doing ordinary touristy things on the days when I wasn't attending the 2008 edition of the European Common Lisp Meeting. I suspect that most readers of this diary are likely to prefer a summary of the ECLM to the details of my wanderings on the (rather cold) Saturday, so that is what I shall provide.

Walking through the streets of continental cities early (around 08:00) on a Sunday Morning is usually a pleasant experience – everything is closed, most people aren't even yet going to appropriate religious observances, and so you get an unimpeded view of the town, devoid of the hordes that can cause obstruction or aggravation. Unfortunately, Amsterdam is a somewhat popular destination for British ‘stag’ parties; and, indeed, as I was walking down Leidsestraat, two such parties, who did not seem to be the sort to be going either to an appropriate religious observance or to the ECLM, passed each other. Noise ensued.

On arriving at the Felix Meritis, at 08:30 on the dot, I saw a crowd of people, all seemingly with the same idea: to have the first coffee of the day. Fortunately, the doors opened shortly afterwards, and caffeination was allowed to take place. Conversation happened as well, though I can't remember very much of it, not yet having been fully invigorated by the coffee. Mostly I think I explained why I wasn't on the previous night's boat trip: a combination of somewhat tardy registration and a second mouth to feed...

And then the real action, the talks, got under way. Unlike many an academic conference, of of the ECLM's distinguishing features is that the talks are attended by the vast majority of the participants; there's much less of a feeling that people simply attend to chat to colleagues: probably partly because the talks are very much the ‘deliverable’ of the meeting (there are no proceedings or anything like that) but also, I think, because the talks cover interesting ground, and offer perspectives based on a solid amount of experience.

Jeremy Jones (from Clozure Associates) started the ball rolling, with a talk on the production of InspireData: an application built for data visualization in an (American, pre-University) educational context. Jeremy wisely started off with a demonstration of InspireData's features; it contains some very impressive-looking tools, and seems to present them to the user in a sensible way. I haven't tried it or gone beyond the demo, but it looks like a huge advance on, say, using an off-the-shelf spreadsheet program to do data analysis, even (dare I say it) for professionals – though whether it scales up to professional-sized data sets is another question. Some other take-home messages from Jeremy's talk: it is possible to sell shrink-wrapped software, even today, even written in Lisp; having a proper designer on the team (or as your client) can help enormously in producing a usable interface; and having a programmer as your client can be both a help and a hindrance – specify the acceptance process carefully.

Nicholas Neuss followed Jeremy, with a discussion of the FEMLISP framework for solving partial differential equations. Being in a somewhat darkened room, early in the morning at a weekend reminded me a little of my undergraduate days, where discussion of differential equations, fields and the like was par for the course – and the talk took me right back (in a good way). In particular, watching FEMLISP compute the eigenmodes of Lake Constance (surface waves, I think) was entertaining. One difference in kind between this talk and the previous is, I think, a function of the possible ‘market’ for the two tools; InspireData is sold in quantities of the order of tens of thousands at present, while, let's face it, FEMLISP is never going to have that kind of exposure: and so the resources aren't really available to make FEMLISP into a product usable by even other domain experts. Of course, the fact that Nicholas' boss makes a competing product might also have something to do with that...

There followed a talk about large Internet systems, from Stefan Richter. There was an interesting survey during the talk, asking people about the size of their userbase (assuming that they worked on web applications at all). An interesting distribution; certainly applications with millions of users are no longer rare – and Juho showed commendable restraint in not snorting something along the lines of “millions of users?” One other interesting moment was when Nick Levine stopped Stefan, to give him time to write down the long list of libraries that is already available to help build Lisp web applications, from the database to the front-end.

Kilian Sprotte gave the last presentation before lunch, talking about the GL-enabled Patchwork music visual programming system. The presentation unfortunately appeared to be a little bit unrehearsed, and so I'm not sure that Kilian got everything that he wanted to across to the general audience. From my perspective, though, it was sufficiently close to the day job that I could see the point (and catch some of the references; B-A-C-H and so on) – I'll be able to report back to people in the lab, who were asking about it, and maybe we can find some useful musical analysis algorithms in there. Also, it occurred to me that GSharp could usefully provide some import or export functionality for the chord and score editors, maybe through MusicXML, or maybe writing directly into the PWGL notations for notes (tighter couplings are likely to be hard, given PWGL's current non-Open-Source nature.)

Then we broke for lunch; Juho, Nikodemus and I ended up sitting on a table with Hannes Mehnert and Luke Gorrie; among other conversations, we had The Great SBCL Maintenance Debate, and we now have a proposed Plan. (I don't know if any other plans have been proposed, but hopefully soon there will be less uncertainty.) The rest of the lunch break was spent grilling Luke about OLPC and Kathmandu, and eating interesting interpretations of café food. (Pasta arrabiata with beans and squash? I don't think so.)

After lunch, a talk about ~~cheap apartment design~~ architect assistance software: Knowledge-Based Engineering, where in this case the knowledge base is about Norwegian building regulations. The idea of the House Designer product developed at Selvaag is to allow architects to experiment with designs, while tracking that building rules (both governmentally-imposed and the house style) can be accommodated, along with all the necessary pipes and electrics and so on. It was good to see a variety of techniques on display, and I liked the flashes of humour: for instance, that the user-interface group asking for XML descriptions was OK by the Lisp group, but that then the user-interface group wanted to send XML back, and that was not OK. It's good to know that Frode Fjeld has a Lisp job, too; he was solidly namechecked in the presentation.

Then Juan José García-Ripoll came to the stage to talk about ECL design and implementation. There were some wry moments for me there; starting with the observation that he wasn't really a computer scientist at all, but rather a physicist. Also, many of the motivations for the ECL design appear to be direct analogues of some of SBCL's PRINCIPLES: completeness, clean bootstrappability, and preferring maintainability over whizzy features, for instance (I hope I'm not mischaracterizing here; it's possible I'm just evaluating what he said through an SBCL-tinted lens). Given that, I think it's interesting how different the two systems look, maybe just from having different starting points?

After the final coffee break, yet more talks. Marc Battyani, of ~~FractalConcept~~ HPC Platform, talked about using Lisp to program FPGAs for custom high-performance solutions; in particular, applications in the financial world, such as derivative valuations and automatic share trading platforms. (A word of advice to Marc: it's not often that you get to say that you're 1000 times as fast as your nearest competitor, nor that you can process packets faster than the test network can send them to you – so don't let that message get hidden by the constant fumbling for a particular Microsoft Image and Fax viewer window!) The products he has look interesting; best of luck to him for finding a buyer in these more financially-challenged times...

And finally, the pièce de résistance: Kenny Tilton took the stage, to give “a rant on the state of Lisp and Lispniks touching on Algebra software, Lisp libraries, Open Source, Cello, Cells, and somewhere along the way introducing Triple-Cells, animated data modelling with persistence for free”. Unfortunately, it seemed that we weren't going to get this rant; instead, we got a small demo of the Algebra tutoring software, along with some discussion of the dataflow paradigm that Kenny believes is central to all simple applications, and a fair number of sound effects. Nice anecdote about Lisp and speeding tickets, though.

Then it was all over bar the dinner; I chatted to Pascal Costanza and Charlotte Herzeel about the busy workshop season, to Jeremy Jones and Marco Baringer about the (lack of) checkin policy to SBCL's CVS, to Edi about how much Heathrow Airport sucks, among many conversations. As we were about to call it a night, Nick mentioned that the Lambda Express had some spare tickets, and that we could probably hitch a lift back to London by train (rather than by plane to Terminal 5...), so we arranged to meet Dave Fox and his crew the following morning. While on the train, in a spirit of cross-implementation co-operation, we essentially finished the Araucaria from the Saturday Guardian – documentary evidence will be forthcoming. And then it really was all over.

18 Feb 2008

Mon, 18 Feb 2008 10:15:25 GMT

Long time no blog. As ever, there are a number of reasons for this: complicated employment situation; too much work to do... but there's also the slightly addictive personality to consider. Since I've already been here once before, it should have come as no surprise to me that I could become engrossed in beating a roguelike game; in the end, I did beat Tales of Middle Earth (ToME) into submission, as I did with Nethack two years ago. (Two years? Well, consider that the only solid playing time is around Christmas, and then learn that last year I almost beat ToME into submission...)

“Complicated employment situation”, I hear you ask? The life of a junior academic researcher, while rewarding, often feels precarious: my main project research funding ran out at the end of May, and since then I have been employed on a succession of temporary research contracts. One thing this has led me to discover is that I am just not very good at having the next thing lined up before the current thing expires. Another is that I do not like the scary feeling of not knowing what I am doing next month – or perhaps more pertinently, where the next paycheque is coming from.

The good news is that this has now resolved itself; I have accepted a lectureship at Goldsmiths, a part of the University of London. (For readers not fully versed in the UK Higher Education terminology, this is a Real, Permanent Job involving a mixture of teaching, personal research and filling in lots and lots of forms.) I work in a group specializing in investigating sound and musical processes, from analysis and synthesis, through cognition, designing systems for cataloguing musical artifacts, all the way to automated Music Theory.

So what do I do there? My primary research responsibility is to the OMRAS2 project, which aims to do all sorts of clever things for recognizing and searching music: current fingerprinting services work on exact (but possibly degraded) copies of notionally entire tracks, whereas I have an interest in automatically deducing musical structures both from audio and from notation, and in working with approximate kinds of similarity (detecting that a track is a remix of another, or a cover version, and so on).

One of the reasons that I like this group so much, though, is that when I first met them, one of the PhD students (who is still with us, in a more advanced capacity) was working on his laptop using CMUCL and ILISP (this was late 2003, so SLIME was if not a glimmer in the mind of Luke Gorrie, at least not the clear choice for development on Free Common Lisps as it is today). This means that my interests in the engineering and language design aspects of SBCL development can go hand-in-hand with attempting to come up with protocols for interacting with musical corpora in many, many different formats: attempting to make it so that the same routines can operate both on representations of lute tablature and on a database of MIDI files designed for karaoke machines, being used to investigate memory for music. I finally have the excuse to use gsharp at work, too!

Speaking of gsharp, Brian Gruber handsomely completed his Summer of Code project, to add MusicXML import and export facilities. His work has been integrated into gsharp's CVS repository, and is being used and further developed here at Goldsmiths, and I got a T-shirt from Google. If you're reading: thanks, Brian.

It's not quite “all quiet on the SBCL front”, but all of the rest of my life leaves me with less time to spend on it than I would like. I did manage to restore support for the alpha architecture, which had bitrotted a little in the last year; this puts me in a reasonable position to finish up my work on improving the interaction between modular arithmetic and representation selection. In addition, I have been pushing forward a little bit the work that I reported on at ECOOP last year regarding user-defined subclasses of specializer; stay tuned for more details. (Maybe sooner than in six months' time!)

15 Aug 2007

Wed, 15 Aug 2007 20:12:36 GMT

It's been a while. Lots of interesting[*] stuff to talk about. Since my previous diary entry was about my work for the European Lisp Workshop, why not start with my recollections and observations about that?

The workshop itself was one day long, and had one invited talk, five papers presented (one of which was mine) and a more informal demo / presentation of current advances in ContextL (a Lisp implementation of Context-Oriented Programming), given by Pascal Costanza.

The invited talk was given by Alexander Repenning, and was titled “Antiobjects” (and subtitled “Mapping Game AI to Massively Parallel Architectures using Collaborative Diffusion”, for what that's worth). The central idea, I think, was to convert a goal-oriented architecture – expressing behaviour of agents in the system in terms of what their goals are, in what is maybe the typical object-oriented programming style – into an optimization problem, viewing the system as a whole.

This change of perspective (the “Antiobjects” of the title allows for some interesting emergent properties. The compelling example Alexander gave was for ghosts chasing Pacman: the difficult bit in the traditional object-oriented style is in implementing collaborative strategies: if there are two ways to reach Pacman, and two ghosts, you (the evil games programmer) would like the two ghosts to take one way each. Generalising for large values of “two” is tricky. The Antiobject solution is instead to define a Pacman field, which diffuses away from the Pacman source, and have the ghosts be sinks of that field. Then the ghosts simply need to do hill-climbing of the Pacman field, and the desired behaviour of collaborative-Pacman-eating appears as if by magic.

Sébastien Mosser gave a talk about meta-models which, I have to confess, I didn't understand at all. I don't know what a “model” is, let along a “meta-model”, and unfortunately Sébastien didn't really give me an idea why I should care. I welcome enlightened discussion on this, if I should in fact care. The next presentation, reporting work by Pierre and Simon Thierry, was a general introduction to transparent persistence, which (as Pierre said) is almost a textbook example of using the Metaobject Protocol. The picture is slightly clouded because unfortunately update-instance-for-redefined-class is not quite able to cope with multiple class redefinitions in between accesses to instances. I believe I have a solution for this issue, but the margin of this blog is too small to contain it. (In practice, people work around the problem without too much pain: I discussed this at the workshop with not only Pierre, but also Arthur Lemmens of rucksack fame.

After lunch, I gave my talk about user-extensible specializers, details about their implementation, and examples of their use. In the course of giving examples, I inevitably made the mathematician's mistake of attempting to persuade the audience that simplifying (* _x 0) to _x was a good idea, but apart from that I don't think I embarrassed myself too much. What's interesting is that Jim Newton's talk, scheduled immediately after mine, was about an application of user-defined specializers, though in the SKILL (or maybe SKILL++, I forget) language of Cadence rather than in Common Lisp. (For the Common Lisper who is intrigued at seeing an application for user-defined specializer classes, as well as reading Jim Newton's paper I can recommend getting my implementation of the ideas by doing darcs get http://common-lisp.net/~crhodes/vclos, and playing around with the source files and examples therein.

After that, Nikodemus talked about his work on making the various caches in SBCL's implementation of CLOS thread-safe; we got to see some current directions in Context-Oriented Programming development (as well as news that a very few people are actually using it in the wild!), and then it was off to dinner, where I manage to surprise Edi by telling him that his webserver was too complicated, and David managed (not entirely) to surprise me by not talking at all about putative CLIM backends...

[*]In My Opinion.

22 Jun 2007

Fri, 22 Jun 2007 15:12:21 GMT

People* like to say that Common Lisp is the programmable programming language, or the #1=(programmable . #1#) programming language: possibly because much of what Common Lisp provides is in some way malleable: the reader, the printer, the pretty-printer, the object system and the syntax are all to a large extent customizeable, through *readtable*, print-object, *print-pprint-dispatch*, the Metaobject Protocol and macros respectively.

Not everything in CL is in such a marvellous state of perfect adaptability – though perfect adaptability does not necessarily mean unlimited customizeability, as there are concerns about being able to compile to efficient code where that is desired as well as being able to mold the system to meet a set of needs. I've already discussed one case of (by default) non-extensibility, that of subclassing the sequence System Class; Common Lisp has a wide range of functionality that is applicable to sequences, but no way of defining new kinds of sequence, instead restricting the applicability of those sequence functions to just singly-linked lists and vectors. In this modern age, that's a little bit unambitious, and so I presented a paper at the International Lisp Conference this year about how to extend CL to allow the user to define his own sequence classes.

For the 4th European Lisp Workshop (in conjunction with ECOOP), I've looked at a similar example of user-extensibility, though maybe a little more esoteric: examining the ability of the user to define subclasses of the specializer metaobject class (and how to use those new specializers in the existing CLOS metaobject protocols). The first draft of the writeup around this issue is available; the document linked there is not final, but it is correct in the broad brushstroke summary, which is that in this area the Metaobject Protocol is poorly implemented (an even more executive summary might be: get SBCL 1.0.7 if you want to try it out...)

[*] some people, anyway. It's possible for people to get caught up in hype and to make claims that they cannot themselves justify, appealing instead to some higher authority when challenged; one common pattern with respect to Common Lisp could be called the brucio pattern.

15 Jun 2007

Fri, 15 Jun 2007 20:34:45 GMT

More Gsharp blogging, I'm afraid; we will return to pure Lisp implementation hacking in a future edition. Until then, I'll mention that magnusjonsson has showed up out of the blue, with an avowed interest in providing support for microtonal mutations (in the form of notation and performance of sagittal accidentals, which are apparently good enough for the microtone community). I think this is awesomely cool, and it even potentially ties into my own musical research interests, such as whether one can infer the tuning system required for best performance of certain Renaissance repertoires (e.g. syntonic tuning in Willaert motets).

Relatedly, I recently discovered that I work (at one remove in management structures) with a developer on a different (competing! or maybe that's giving Gsharp a little too much credibility) music notation programme. My hope is that I stay around in my current position for long enough that interesting work in that space can happen, because in neither of our cases are we in fact employed to work on music notation – even though it is extremely close conceptually to the things that we are paid to do...

11 Jun 2007

Mon, 11 Jun 2007 09:25:36 GMT

Continuing the theme of being guilt-tripped into making Gsharp tolerable to use (rather than just pretty in screenshots...): over the weekend I implemented selecting the active layer in a DWIM-like manner for the common case, moving up and down between staves using C-up and C-down (and then implicitly selecting the first layer on that staff).

Additionally, I implemented a hack so that the cursor is always visible in the viewport. It's a hack, because (despite the neat double-buffering support in McCLIM), there is visible flicker in some circumstances. The reason is that the viewport positioning is done at the wrong time – after the score has been drawn to the pane. Fixing this would involve more work than I was prepared to commit to on a lazy Saturday afternoon, though.

8 Jun 2007

Fri, 8 Jun 2007 08:55:26 GMT

We're close two working weeks into Google's Summer of Code, and the student I'm mentoring, Brian Gruber, is making good progress in his project to add MusicXML support to Gsharp – sufficiently good to blog about it, in fact.

Discussing things with Brian, and fixing some of the embarrassing bugs in gsharp that he found, has also encouraged me to tidy up a little some of my accumulated divergences from the Gsharp CVS and commit them; in particular, the lack of line- and page-based motion commands was a significant annoyance last time I edited a score in Gsharp, so I spent an evening working out a sufficiently good implementation, and now C-a, C-e, C-x [ and C-x ] work as an emacs user might expect. (It's ever so slightly trickier than it might seem, because there's no information in the gsharp buffer recording layout information per se; the gsharp buffer contains a complicated data structure representing each individual musical (or graphical) object, but the page and line layout is computed separately – so motion by layout element needs additional computation).

Other things coming soon to a CVS repository near common-lisp.net, to whom thanks go for their hosting: correct placement of whole-bar rests (in the middle of a bar, if they are the only element in their layer in that bar); a hacky printing facility; maybe even a reworking of the currently-useless panes on the right-hand side of a gsharp window...

30 Apr 2007

Mon, 30 Apr 2007 09:19:56 GMT

In my previous diary entry, I somewhat sarcastically obeyed the letter of the BSDish-with-advertising licence of cl-pdf. Since then, Marc Battyani wrote to me to say that he was removing the advertising clause from the licence of cl-pdf (and cl-typesetting), replacing it with a request (not part of the licence) to acknowledge its use in significant applications, to assist in promoting the Common Lisp language. Thanks, Marc, both for the software and for the friendlier licence.

26 Apr 2007

Thu, 26 Apr 2007 13:02:38 GMT

Some people have expressed mild interest in the code used to produce the name badges for ILC 2007. I used cl-pdf to generate PDF, and otherwise there's only a very little bit of cleverness to get nicely filled and indented lists of features and similar. Feel free to download it and use in whatever way you wish, subject to the licence of cl-pdf (which seems to have an advertising clause, so I must say of defperson.lisp that This product includes cl-pdf for generating documents.)

6 Apr 2007

Fri, 6 Apr 2007 16:21:46 GMT

As I wrote in my previous entry on ILC 2007, I was away from the conference on Tuesday. From talking to people whose judgment I trust, it appears that I missed some spectacularly awful stuff; however, now is probably not the time to comment further publically on such matters. Of the good stuff, I was most sorry to miss Richard Jones talking on Garbage Collection; having the book and the paper that Richard wrote for the proceedings aren't a substitute. Apparently, Charlotte Herzeel made some people begin to see some point in Aspect-Oriented programming, too, which is no mean achievement given that the usual response is that it's just one sufficiently-advanced method combination (well, it is, but it's also using it pervasively).

Anyway, Wednesday beckoned. The start at an ungodly hour was redeemed by Manuel Serrano's demonstration of HOP, an environment for programming the Web. I don't really live in the Web programming sphere; it doesn't interest me a great deal: but the presentation was excellent, with well-timed automatic slide changes, fun demos, and a clear explanation of the problem space; I don't know whether having the need for a proxy server will render it difficult to deploy in practice. Good stuff anyway; in fact, it worries me ever so slightly that the best of the invited speakers for me – Sperber, Queinnec and Serrano – are very firmly in the Scheme camp.

After that talk, I found speakers failing to command my attention quite as much as the impending delivery of my own, about extensible sequences; I did enjoy Geoff Wozniak's about dynamic data types, defining a protocol whereby instances would be able to change their own implementation details based on runtime profiling data; I think I need to look at the details of his ideas.

Then the time came for me to stand up, and – wouldn't you know it? – the laptop refused to display simultaneously on screen and on projector. Which is ridiculous, because it was fine on Monday for Herbert Stoyan's talk. I bravely soldiered on without the comfort blanket of my notes in front of me, and I don't think that any problems in the reception of my talk were particularly because of that. In retrospect, the example sequence class I used in the talk was too simple, and I didn't give compelling-enough other concrete examples; fortunately, I was able to make up for some of that in the coffee break afterwards, talking to a couple of people (and Cyrus did a sterling job talking to Jans Aasman about his space-efficient implementation of strings of DNA, and Jans then started talking about treating memory-mapped files as sequences, so I think he probably got the idea).

Then it was all over bar the Liskell talk (not particularly interested, though good on Clemens for having the gumption to stand up there and tell it like he sees it), the Franz programming-in-the-large talk (Jans is a fun speaker; I thought that Luke asked a rather good question at the end about the expected reliability of garbage collection), and a little bit (fortunately, not a lot) of self-congratulation at the end. Dinner at the Hotel Felix with, among others, an exhausted local organizer wrapped up the day, and I was homeward-bound twelve hours later.