Older blog entries for crhodes (starting at number 149)

New laptop video (intel 855GM) drivers holding up pleasantly well. One crash so far, from attempting to play a video; I haven't tried to reproduce it, since that's not something I do very often in any case. (Said video was of my own extremely minor contribution to UK televisual arts programming, so maybe my video drivers were wisely preventing me from narcissistic overexposure...)

Meanwhile, back in the land of sort-of-industrial research and development, I've been using (and simultaneously learning) R for analysing the meaty chunks of data that my colleagues are generating. A couple of my academic labmates were already R users, unashamedly using R for their data treatment needs, even when their data came from Lisp programs. Shocking, I know. So, when substantial datasets started landing on my lap a few months ago (not just from a group of people in mobile broadband; energy use data and computer usage metrics also crossed my desk) I decided that the time was ripe to learn some new tools and techniques.

Initial reactions: mostly positive. To help put my observations into context: I've dabbled in MATLAB before, and it's never quite stuck. The everything-is-a-matrix aspect was painful; graphical output was OK but nothing to write home about; and environmental support pretty painful. (GNU Octave suffers from all of these problems too, and then some; it does have the significant advantage, from my point of view, that at least there isn't the attempted lock-in from purchasing a bare proprietary wrapper over BLAS and LAPACK with the option to buy yet more functionality wrapping BLAS and LAPACK in slightly different ways). I've also used (a long, long time ago now) IDL, again a vector-oriented language; my memory of it is mostly faded, but I remember being satisfied with its graphing facilities and much more satisfied with an Emacs-mode than with its default User Interface. This was 1998; I dare say things would be much the same today...

By contrast, R has data types that are mostly comfortable to my inner Lisp programmer. Yes, number crunching is best done through vectors (matrices being a thin wrapper around vectors rather than a distinct language data type), but lists of vectors are fine, and are used to collect data into data.frames. It has a lightweight object system, with single dispatch on the class of the first argument; mind you, classes of objects are a pretty mutable concept in R, settable at arbitrary points in program execution (there's mostly no relationship between object class and object contents). Speaking of settable, there's the `<-` operator both for assignment and for mutation, like Common Lisp's setf. “Mutation” there might actually not be quite the right word; the evaluation semantics are mostly lexical binding and call-by-value, with the interpreter attempting to perform copy-on-write and deforestation optimizations. Speaking of interpreters, there's a reified environment and call stack at all stages of program execution, which makes the language mildly tricky to compile, particularly since environments are just about the only thing in the language which can be mutated; this aspect of the language has recently been the subject of discussion in the R community (and, would you believe it, one of the camps is advocating a rewrite of the engine using Common Lisp both as a model and as an implementation language, mentioning SBCL by name... sadly without increasing my citation count. Oh well.)

In any case, once I saw the reified environments, and also spotted parse (analogue to read) and eval, I started wondering whether the comint-based Emacs mode for R, Emacs Speaks Statistics, could be enhanced or replaced with some SLIME-like functionality. I was particularly interested in whether there was enough introspective capability to replace the default debugger, which I was not having much success in using at all. So I started digging into the documentation, finding first try, then tryCatch (OK, that's like handler-case, so far so normal), and then on the same help page found withCallingHandlers and withRestarts, direct analogues of handler-bind and restart-case. At that point it seemed logical, instead of trying to write some SLIME-like functionality for ESS, to simply write an R backend for SLIME. With some careful (ahem) adaptation of a stub backend Helmut Eller wrote for Ruby, I got cracking, and swankr now supports a slime REPL, slime scratch buffers, SLDB, the default inspector, and presentations.

Here's a teaser screenshot, demonstrating functionality not yet merged: image output for lattice objects, presented in the SLIME REPL, and usable in subsequent input. There's plenty in swankr that doesn't work, and plenty more that's buggy, but it might be good enough for the enthusiastic R/Lisp/Emacs crossover community to give it a try.

(No, my screen is not that tall.)

In my new life as an itinerant entrepreneur, spending significant amounts of time both working from strange places and travelling on trains, my productivity depends at least in part on having a good development laptop. In my other current life lecturing in Creative Computing, I also need to be able to display reliably to a data projector. Neither of these was a problem until relatively recently.

For the last four years or so, after obtaining a recommendation from one of the masters of Linux and laptops, I've used an IBMlenovo X40: light, fairly rugged (has survived at least one drop), almost full-sized keyboard, and – importantly – fits into more or less any carrying equipment. However, the relative instability (which I am not the only one to experience) brought about by the change in Linux and X.org's to Kernel-Mode Switching was beginning to get worrying; the X40 has an Intel 855GM graphics card, which since Intel is participating heavily in KMS support and development, means that new things got turned on early; I follow squeeze (Debian testing), which gives me some of the thrills and spills of being on the leading edge of consumer Linux development; most pertinently for the start of the new academic year, I've been suffering from odd display corruption on the VGA output.

In various combinations of Xorg and kernel versions, trying to work out a set of versions that work for me, I have experienced

• intermittent (about once a day) X server crashes, leaving the hardware in a sufficiently inconsistent state that the X server refuses to start again (linux v2.6.33ish, xserver-xorg-video-intel 2.9.1)
• failure to start the X server on boot at all (linux v2.6.33ish, xserver- xorg-video-intel 2.12.0+legacy1-1)
• missing mouse cursor on X server start (linux v2.6.35, xserver-xorg- video-intel 2.9.1)
• substantial (~0.5s) latency about once every 10 seconds, with kslowd or kworkqueue processes taking about 20% of the CPU (linux v2.6.35 and v2.6.36-rc3, xserver-xorg-video-intel 2.9.1)

The good news is that a large number of Intel-graphics-related fixes appeared in Linus' master branch yesterday, and at least some of these problems are fixed; with a module parameter poll=0 for the kms_drm_helper module, the 0.5s latencies are gone (put options kms_drm_helper poll=0 in a file in /etc/modprobe.d). The VGA corruption I was experiencing seems to have been fixed somewhere between Linux versions 2.6.33 and 2.6.35; I have hopes, too, that the X server crashes might be substantially less frequent (but I haven't been running a single kernel long enough to check yet). The one remaining issue that is definitely still present is the missing mouse cursor when X first starts; a suspend/resume cycle works around this fairly reliably for me. Thank you to all those who've worked on fixing these problems.

I'm particularly glad that all of this is just about sufficiently fixed, because the alternative would have been to get a new laptop, and as well as my natural disinclination to purchase new stuff when not strictly necessary (and let's face it, the initial phases of a startup are not usually those where money is abundant), it seems to be largely impossible to get a new one with the form factor of the X40: with the growing prevalence of widescreens, just about every modern laptop is substantially bigger than this one, and thus would not fit in some of my carrying equipment. I will just have to preserve this one as long as possible.

I'm on a train! A train, heading towards Christchurch, with an amusingly bad synthesized (at least, I hope it's synthesized) announcement of train stops.

Being on a train would not normally be worthy of a diary entry. However, I'm travelling not as part of my heretofore stable academic enquiries, but as a mostly-fledged member of the high-flying technological startup community. No, I haven't quite sold out; as of yesterday, I work four days a week for a bright-futured company in the general area of mobile and wireless broadband; the other working day will continue to be spent at Goldsmiths, where I retain most of my responsibilities (but not the management of the music-informatics OMRAS2 project, which finished at the end of August); this arrangement with Goldsmiths will continue for a year.

So, why? A number of factors came together to make this a very attractive opportunity. Firstly, as I've already mentioned, the research project whose technical side I was responsible for came to an end last month; although the project itself was interesting, and its existence was at least partly responsible for me having a permanent academic position, I think it's fair to say that there were all sorts of wetware headaches as a result. The end of the project was therefore a natural point to start afresh — and in particular, to try to focus on aspects of work that I enjoy. Secondly, my PhD students are largely getting towards the end of their studies; most should be submitting their dissertations in the next few months. While PhD students are usually an asset to an academic, it is true that they can take up significant amounts of time, particularly in the early stages; not having to guide any in the early stages of their research next year gives me the freedom to investigate other activities, and to try to broaden my experience. Thirdly, when the person who comes calling is as awesome as our CTO, and when the rest of the gang is full of names I recognize, well, I expect to learn a lot. (I already have!)

And of course, the fact that some of the underlying technology used in the company uses is software that I'm fairly familiar with is attractive; it would be good to show the world that there are more SBCL-using companies with a healthy business than just the poster child. If I'm lucky, the Free Software and Lisp-related content on this blog should noticeably increase, because although the academic life gives a huge freedom to think, it doesn't actually give a large amount of time to do very much with those thoughts; in this newly-acquired business role, as well as the inevitable all- hands pitching in until the small hours to make the demos work with bits of string and glue, I hope that there will be time to implement and reflect on interesting things.

It's been a while. It's in fact almost embarassingly late for me to be blogging now about the 2010 European Lisp Symposium, which was over a month ago now – in my defence, I point to the inevitable stress-related illness that follows excessive concentration on a single event, coupled with hilarious clashing deadlines at work and in my outside activities.

So, we cast our minds back to April. When I booked my flight to Lisbon, I deliberately chose not to fly with British Airways, on the basis that they were likely to strike. It turned out that the activities of British Airways cabin crew was going to be the least of the problems associated with getting to an international conference...

Yes, shortly after I booked my tickets, Eyjafjallajökull went *boom* and most of Europe's airspace closed for a week, with ongoing disruption for the best part of a month. There were moments during the disruption when I wondered whether there was an actual curse affecting ELS, but in the event things cleared up and there was only minimal disruption to delegates and speakers, both getting there and getting back.

But enough about transport! How was the symposium itself? Well, I enjoyed the programme – but given that I had as much control over it as events would allow me, perhaps that's not the most unbiased endorsement of quality. Still, I was entertained by all the keynote speakers, from the window into the business world opened by Jason Cornez of Ravenpack, via the practical philosophy and view on history afforded by Kent Pitman, to the language experimentation and development in PLT Scheme Racket described enthusiastically by Matthias Felleisen. Pascal Costanza's tutorial on parallel and concurrent programming was highly informative, and there was a good variety of technical talks.

It's often said, though, that the good stuff at conferences happens between the programme, and for that the local organization needs to be on the ball. I'm glad to say that António Leitão and Edgar Gonçalves, and their helpers, enabled a huge amount of interaction: lunch, coffee and tea breaks, evening meals (including a fabulous conference banquet, but also a more informal dinner and a meal punctuated by Fado. I gather the excursion to Sintra on the Saturday was interesting; by then I was at the airport, looking nervously at the departure boards...

I enjoyed meeting and talking with many of the attendees; some whose names I knew but whose faces I didn't (and some whose names I knew because they shared them with other people who I knew already: take a bow, both Luís Oliveiras); but with always one eye on the next thing that could go wrong, I didn't get to go very deeply into interesting conversations. Maybe next year, in Hamburg for ELS2011 (expect the paper deadline to be around 31st December 2010) – in the meantime, there's likely to be a journal special issue with an open call for papers, coming soon, and of course the ALU are holding an International Lisp Conference this year, whose call for papers is currently open. So get writing!

Many people will have heard that Nick Levine's health has meant that he had to withdraw from giving a talk at the upcoming European Lisp Symposium; I wish him a speedy and comfortable recovery.

I'm very glad to be able to announce that Jason Cornez of RavenPack International has agreed, at very short notice, to give a talk at the Symposium: Reading the News with Common Lisp. The abstract for his talk is:

The financial industry thrives on data: oceans of historical archives and rivers of low-latency, real-time feeds. If you can know more, know sooner, or know differently, then there is the opportunity to exploit this knowledge and make money. Today's automated trading systems consume this data and make unassisted decisions to do just that. But even though almost every trader will tell you that news is an important input into their trading decisions, most automated systems today are completely unaware of the news – some data is missing. What technology is being used to change all this and make news available as analytic data to meet the aggressive demands of the financial industry?

For around seven years now, RavenPack has been using Common Lisp as the core technology to solve problems and create opportunities for the financial industry. We have a revenue-generating business model where we sell News Analytics – factual and sentiment data extracted from unstructured, textual news. In this talk, I'll describe the RavenPack software architecture with special focus on how Lisp plays a critical role in our technology platform, and hopefully in our success. I hope to touch on why we at RavenPack love Lisp, some challenges we face when using Lisp, and perhaps even some principles of successful software engineering.

Many thanks to Jans Aasman and Craig Norvell of Franz Inc., as well as to Jason and RavenPack, for making this possible.

A little news regarding the upcoming European Lisp Symposium is enough excuse to repost the Call for Participation, particularly since the Early Registration Deadline is approaching (it's this Thursday, 22nd April). António Leitão has managed to resurrect an Explorer Lisp Machine, cannibalising parts from a second, and some of the applications from yesteryear (or rather two decades ago) will be demonstrated at the Symposium.

Things are beginning to take shape for the European Lisp Symposium (May 6-7, Lisbon): I'm glad to say that the review process is over and the preliminary programme is now available. The symposium website has also undergone a facelift: out with the spartan, old-school look, and in with the tasteful and functional; Edgar Gonçalves put the new look and content together in very short order, so thanks to him; thanks also to the event's supporters.

The final organizational details are being sorted out, and registration should be open very soon.

Because of multiple requests, the deadline for contributions to the 2010 European Lisp Symposium has been extended by one week, to 5th February 2010. Submissions will of course be accepted in advance of this new deadline; don't feel you have to wait until the last minute...

The deadline for submissions to the 2010 European Lisp Symposium is now less than one week away. In the meantime, I'm pleased to be able to announce the invited speakers for the symposium:

• Matthias Felleisen, of Northeastern University and the TeachScheme! project will speak on the topic of PLT Scheme;

• Nick Levine, independent consultant at Ravenbrook, has twenty years of Lisp experience after a serendipitous encounter, and over that time has taken on roles from system implementor to contractor; from educator to conference organiser; and most currently, author of a new book on Lisp;

• Kent Pitman, with his background of over three decades of involvement in the design, implementation and use of Lisp-family languages, will offer historical perspectives and thoughts for the future.

I hope that this lineup provides additional motivation for people to complete their submissions!

I feel I don't get to do very much hacking any more.

I shouldn’t complain, really: I have a decent and stable job, which is mostly fun; I have a certain amount of freedom in what I do, as long as everything that has to get done gets done; I work with all sorts of interesting people, both formally and informally. But things that I want to do have to live a long way down the priority queue; preparing lecture materials, paper drafts, committee agendas, bursary agreements, course proposals, courseworks, exams, student feedback, paper redrafts, reports, meeting notes, grant proposal drafts, paper reviews, examiners’ reports, reading lists, grant proposal redrafts, and the like all seem to take priority over even the research on a funded project that I am part of, let alone the discretionary research that I might actually want to do.

So sometimes I have to be sneaky, and combine my hacking with teaching-related work instead. One of the more fun things I’ve learnt over the last couple of years is enough colour theory to be dangerous; it started off because I was casting around for ideas on what to teach students on our Creative Computing programme – and I do teach them about colour, among other things – but it’s sufficiently interesting as a technical area in itself that I can see writing code to illustrate aspects of it. So, here’s a (not very good) colour picker “application” for McCLIM, whose only redeeming feature is that it uses knowledge of the colour attributes of consumer-grade display hardware to present colours of the same intensity together. That’s a bit hard to visualize, so here’s a screenshot, where all the colours in the triangle should seem to have about the same brightness (viewers might need to adjust their viewing angle):

Source code is here; I’m not particularly proud of it, and it needs work in all sorts of directions (optimizing, generalizing, cleaning up). One of the reasons I had put off blogging about this is that I was hoping for a lovely literate-programming system to optimized for single-file Lisp programs to appear, generating HTML and PDF output from minimally-marked-up Lisp code. Sadly, that hasn't happened, and my best attempt can only be described as, well, deranged... so no impeccably formatted and indexed code snippets in this blog, not this time anyway.