Older blog entries for crhodes (starting at number 100)

2 May 2006 (updated 2 May 2006 at 13:06 UTC) »

So, the 2006 European Common Lisp Meeting is now over, and I had fun.

For me, the meeting began on Saturday evening, though I hear that some of my younger colleagues had arrived earlier and sampled more of the available tourism and alcohol. Which, in retrospect, was a good idea, because the Saturday dinner was the only time I had available to visit Hamburg itself, and the dinner was held in a restaurant off the Reeperbahn, which turns out to be a touristy red-light district. Oh well; I will have to defer the pleasure of admiring the Gate of the World to a future visit.

The dinner crammed 90 people into a first-floor restaurant room, where we sampled typical Hamburgian fare and made or renewed acquaintances. I chatted to Peter van Eynde about Debian (Lisp and politics, sadly intertwingled), Rudi Schlatte and Dave Murray about multimedia content indexing and metadata. At one point, I was pointed out from afar, apparently to demonstrate that I am not as scary in the flesh as I am as a disembodied pair of nicks on IRC.

Since we hadn't already paid for our drinks at the dinner, consumption was moderately restrained, which I think helped attendance at the 9:30 start of the meeting proper. The room wasn't quite full, which probably meant that there had been some last-minute cancellations – there was capacity for 125 attendees. The first talk, AllegroGraph: a large scale graph database applied to the Semantic Web and telecom fraud detection was given by Jans Aasman of Franz Inc., and described an application of their AllegroCache product for storing RDF, querying OWL and buzzwording the Semantic Web. I couldn't get myself enthused about the technology, though of course it's fine that Franz is making money selling this stuff to companies; the ontology/semantic network stuff is too static to represent changing information, which is the interesting bit. AllegroCache and the Prolog as query engine is neat stuff, of course, but since that was last year's presentation from Jans it doesn't count as new and exciting. The final disappointment was that no mention was made in the talk of telecom fraud detection.

The organizers of ECLM2006 are both smokers. Since they got to design the schedule, they put 5 minute breaks between every pair of talks that didn't otherwise have longer intervals. For those of us who don't smoke, this was an opportunity for coffee and chat, which is the important bit of these meetings. I spoke to people who I only new from IRC and mailing lists, and even to some whose names I didn't recognize at all. Good stuff, and I hope that this habit will continue (though the smoking should stop: I was surprised at how many of the attendees smoked; it was at times quite oppressive).

James Anderson's talk was titled Lisp tools for time series computations, which was of potential interest. Sadly, most of the talk content was about the difficulties of dealing with real-time data: Ravenpack build tools to make predictions of market indices based on non-market data (e.g. news stories) as well as market data (past prices, trading volumes). Again, the actual material was slightly unconvincing, this time because it was revealed that the company backers do not use the company's software, but merely sell it to day traders. (As my neighbour whispered to me, "day traders will buy anything"). The scientist within me was extremely disappointed that the predictions didn't even come with error bars attached, or indeed any kind of stability analysis: indeed this was quoted as a problem, where notably different predictions are made if data come in or are analysed in slightly different orders (or if a butterfly flaps its wings in the data centre). Oh well.

We got a talk about design tradeoffs in building a persistence library for CLOS from Arthur Lemmens' talk Rucksack: a flexible, lightweight, open source persistence library. Indexing strategies, root sets for the (incremental) garbage collection, collection strategies, concurrency issues, and so on, were discussed in some detail. Arthur also had the honesty to say that the library was still evolving – even saying that it was not finished yet – so there's no download link. One interesting thing about Arthur's design is that it is very much a userland library; the way he was describing it, he'd used very few hooks into the compiler and environment he was developing on, so it should (famous last word) be relatively easy to port between Lisps.

Then the folks from Cadence GmbH talked about The PCMan meta version control system: Common Lisp from top to bottom. The problem they were describing was how to manage multiple versions of code bodies from multiple sources, with various restrictions on redistribution, and various properties (such as whether they are binary blobs or text). They described their system, which ended up being a discussion of how to keep information in a database synchronised across multiple clients; a problem that Marc Battyani has talked about and demonstrated before. They also demonstrated ltk, which is neat – I've heard good things about ltk-remote, though they didn't demo that – but uncompelling to someone who doesn't do the fields-and-widgets style of GUI.

By this time I'd had just about enough of talks about difficult things one can desire to do with Lisp and databases. Fortunately it was lunchtime, so I got a chance to refuel and to talk with others. Nikodemus and Rudi had a long discussion about extensible streams (a topic that Jörg Höhle and I chatted briefly about later), and I think it was at this point that I talked to Lutz Euler about vectorization and x86-64 stuff. There were plenty of people I would have liked to chat to longer or indeed at all, and failed to; such is life.

The next talk, Common Lisp in a high-performance search environment was from Martin Cracauer of ITA software. As he started talking about the size of the database and the search space for the problem of finding an optimal price for travelling between A and B, my heart sank, but I was soon relieved as we neatly avoided another talk about interfacing to databases and instead heard a discussion about some impedance mismatches between Common Lisp and really high performance: things such as structures having header words (for type and length information); arrays of structures necessarily being arrays of pointers; the tag bits in a lisp object necessarily restricting the range of an unboxed integer, and so on. He then talked about the various ways in which ITA uses Common Lisp to solve these problems: using CMUCL's system-area-pointer; going beyond that and punning fixnums as pointers to word-aligned data (which works because of the tag bits); and discussing features of the Lisp compiler underlying CMUCL which assist (such as being able to do full untagged 32-bit arithmetic within function boundaries). He did also assure the audience that there were applications in use and development at ITA that do use more traditional CL style, with CLOS, functions, conditions, and so on.

cl-muproc: Erlang-inspired multiprocessing in Common Lisp from Klaus Harbo did more-or-less what it said on the tin: it was an embedded language for doing Erlang-inspired multiprocessing in Lispworks. It built on the provided concurrency of the underlying Lisp, so it suffers in comparison with Erlang's processes as each Lisp process is relatively heavyweight, but the basic demos that we were given looked quite like the Erlang demos most of the audience had already seen.

The final talk of the day, SigLab: a Lisp-based signal and image processing and modeling facility, was given by David McLain, and it was worth the journey in itself. SigLab is an embedded language for describing and implementing signal processing networks; it wraps various platform libraries and provides an exceedingly slick environment; the textual (code) representation is primary, but it provides very good visualization of both the network and its results when applied to given signals. Some of the examples used to demonstrate it were good fun, too. Also mentioned in the talk was a differential equation for the ear's frequency response, which I need to check out; it's claimed to be a damped oscillator with a spring constant proportional to 1+γ<y²>. (I mostly want to know if this is a first approximation, or if there are mechanical reasons to believe this beyond the fact that it must be something like that to stop the ear exploding). If there were video of the presentations available, this one would be the one to download; I don't know if there is, though.

And then it was all over bar the dinner and impromptu demos. I found a lutenist and demoed the tablature editor (based on climacs) developed for the ECOLM project. Later, I gather I got quite a crowd to look at gsharp (favourite comment: "it has keyboard macros; shame it doesn't scroll and it doesn't print"); I was concentrating too hard on not causing a lisp error to notice, as I had foolishly disabled the debugger in the image I was using. David Lichteblau's eclipse-in-cloak-on-sbcl demo is both cooler and more useless than even a score editor which can't print or export, though, and he also got a good audience. Later, I also finally got to see beirc and closure integrated and usable, hidden in the fog of one or two too many beers.

And then it really was all over, and after a short train ride through Hamburg (I didn't even get to see the river) I was back in the UK and to reality. Next year, ILC in Cambridge.

Many Free Software developers will have been aware of the SourceForge CVS outage over the weekend (from Friday to Wednesday morning). This certainly caused some frustration among SBCL developers, particularly since there were some bugfixes (for the cl:listen problem described in my last entry and some new features – notably, working sb-bsd-sockets under Windows – queued up for submission by Friday lunchtime.

So, on Wednesday morning (GMT+1), the sequence of events was roughly:

SourceForge CVS back online;
T + 10 minutes:
Rudi Schlatte commits Windows contrib megapatch;
T + 20 minutes:
My new colleague, Daniel Müllensiefen, asks me for a Windows Lisp environment to start learning in.

This sequence of events meant that, rather than take the path of least resistance and simply go for the trial or personal edition of one of the proprietary Lisps (the first hit is free!), I spent a little time at Daniel's laptop (German keyboard layout, Windows in German localization... oh what fun!) getting emacs, the new and exciting SBCL, and slime talking to each other. It seems to work, at least superficially, given a patch to slime: I got the animation, the CL-USER> prompt, (* 2 3) gives 6, the debugger works. I didn't stress-test it in any way – it wasn't my laptop, after all, and besides, a newcomer to Lisp is likely to stress-test it much more than I could.

That said, experienced Lispers might be interested in some more testing of SBCL under Windows now that SLIME works. Unfortunately, they'll either have to wait for SourceForge anonymous CVS syncs to be re-enabled (probably by Friday) or use Andreas Fuchs' arch gateway.

We released McCLIM 0.9.2 ("Laetare Sunday" – go silly names for releases!) and SBCL 0.9.11 at the weekend.

One relatively new feature in SBCL is the ability to make executable images in a fairly straightforward manner: simply pass :executable t to sb-ext:save-lisp-and-die, and the system will attempt to dump one file that plays relatively nice with the executable loader on your given system. Since it is fair to say that there are a large number of dependencies in McCLIM, and additionally a non-dependency that is in practice needed to get windows to display on the screen, it seemed reasonable to use this executable-generating functionality to distribute a binary that could be tried out before deciding whether or not to get the source to a fair number of software projects.

So, Andreas Fuchs duly stepped up to build a binary of mcclim-0.9.2 for x86/linux, and all seemed well. However, I remembered that some people had expressed an interest in a Mac OS X binary, so I went about building one of those[*]. And all seemed well; a few things had to be changed from Andreas' binary, mostly because SBCL on x86/linux supports threading, while on ppc/darwin it has to make do with select(). However, trying the binary revealed some odd things going on: using Apple's X11.app, you would need to hit RET twice to get your input accepted: only not quite systematically: also, sometimes, the REPL's answer would only appear after a subsequent keystroke.

Amusingly, the same behaviour was far less systematic when running the same binary over an ssh-forwarded connection to an xorg server. This is perhaps not overly surprising, but it did make debugging the problem, or even identifying the proximate cause, a lot harder. Tim Moore and Juho Snellman did end up tracking it down, though: Tim worked out that read-char-no-hang was hanging, and Juho spotted an error in a recent bugfix in SBCL for listen: sometimes the system would report that data was available on a socket even when it wasn't. The difference in behaviour of the buggy binary between X servers is I think explained by a difference in double-buffering: Apple's windows are double buffered, so they never receive any Expose events, whereas xorg's aren't, and so they receive many more events in total; and a failure to respond in a timely manner to an Expose event is less noticeable than not doing anything when hitting the Return key...

The upshot is that SBCL 0.9.11 is unfortunately unsuitable for McCLIM work; however, armed with a hotfix for listen, I have made available an OS X binary of mcclim 0.9.2 for people to try: you'll need X to be running, and also to have libfreetype.dylib in /usr/X11R6/lib/.

[*] For the record, the build script ended looking like this for OS X:

(mapcar #'require
        '(:asdf :clx :mcclim :mcclim-freetype
          :clim-listener :clouseau :swank :clim-examples :functional-geometry))
(load "/isms/home/mas01cr/lisp/cvs/clnet/mcclim/Apps/Debugger/clim-debugger.lisp")
;;; listener fixes / demo commands
(clim:define-command-table clim-listener::demo-commands)
(clim:define-command (clim-listener::com-plot-fishes
                      :name "Plot Fishes with Functional Geometry"
                      :command-table clim-listener::demo-commands :menu t
                      :provide-output-destination-keyword t)
  (functional-geometry::clim-plot functional-geometry::*fishes*))
(clim:define-command (clim-listener::com-show-demos
                      :name "Show McCLIM Demos"
                      :command-table clim-listener::demo-commands :menu t
                      :provide-output-destination-keyword t)
   (clim:make-application-frame 'clim-demo::demodemo)))
 :inherit-from '(clim-listener::application-commands
                 clim-listener::show-commands clim-listener::demo-commands)
 :menu '(("Application" :menu clim-listener::application-commands)
         ("Lisp"        :menu clim-listener::lisp-commands)
         ("Filesystem"  :menu clim-listener::filesystem-commands)
         ("Show"        :menu clim-listener::show-commands)
         ("Demos"       :menu clim-listener::demo-commands))
 :errorp nil)
(push (lambda ()
        (setf mcclim-freetype::*freetype-font-path*
               (make-pathname :directory '(:relative "fonts"))
                :directory (pathname-directory sb-ext:*core-pathname*))))
(setf sb-ext:*invoke-debugger-hook* #'clim-debugger:debugger)
(sb-ext:save-lisp-and-die "/tmp/mcclim-listener-0.9.2/mcclim"
                          :executable t
                          :toplevel (lambda ()
                                      (sb-ext:quit :unix-status 0)))

If I have seemed less present, less immediately responsive these past couple of weeks: it is because I was tricked into having my nethack addiction rekindled. Fortunately, I can report that I achieved my first ascension this evening, and am therefore better able to concentrate on Free Software matters henceforth, until the next serotonin-producing arrangement of punctuation comes along. Lucky I never really programmed in Perl, eh?

Bother. I was going to do a before-and-after comparison of rendering of ties in the Gsharp score editor: before, from when I first implemented ties, and after, when Robert reimplemented them, including the beginnings of support for tie end glyphs, and for adjustment of tie heights for varying widths between tied notes. However, I managed to use the same file name for the latest screenshot as I did for the one almost two years ago (well, yes, ties.png is of course the obvious name for this: so obvious I chose it twice...), so you will just have to take my word for it that the new version is prettier than my hacked up version which just drew lines.

Instead I will just have to provide more evidence of prettiness, and since I have a long-held fondness for the music of Clément Janequin, I give you the beginning of les Cris de Paris, with the beginning of a lilypond version from about 2000 (in an evince window, behind) for a not-quite-fair comparison, and (what is more; see, how generous I am!) a MIDI file from Gsharp, just to prove that I'm not cheating.

Cyrus Harmon has been working on a port of the mostly-copying generational garbage collector in SBCL, based on previous work by Raymond Toy for CMUCL. At the seventh merge candidate, I decided I'd made him suffer enough with my tyrannical requests for whitespace changes and function renamings, so I merged it, having tested on multiple platforms, and went to sleep the sleep of the just.

Sadly, the testing wasn't quite exhaustive enough, and I discovered this morning that we'd managed to kill the overnight automatic build-and-benchmark host run by Andreas Fuchs. Fortunately Cyrus was awake (whether again or still I don't know; for him it was 0300 local time) and diagnosed the problem pretty quickly: we had managed to subvert the garbage collector almost, but not quite, completely on threaded platforms. Interestingly, the system was stable enough to build itself, but benchmarking it (or indeed running the regression tests) exercised different-enough allocation patterns that the existence of the error was pretty obvious.

A simple one-line fix later, and things started working on threaded platforms too.

25 Jan 2006 (updated 25 Jan 2006 at 13:24 UTC) »
John Wiseman wants a workaround installed in SBCL for a buggy, closed OS[*]. If that summary were all that there was to the issue, it certainly wouldn't merit a diary entry, even given the prominence of John's weblog among a section of the Lisp-using public; I don't think it hurts to discuss merits and faults of software out in the open, and of course no publicity is bad publicity.

However, John appears to misunderstand how people work. In particular, he wants to spur SBCL developers to fix the problem he observes for him[**]. Of course, this is a reasonable want, but as children are told, I want doesn't get, and this holds in the adult world too. Those who take out a support contract or similar arrangement have exchanged a quid or 500 for the quo that they then have a right to expect; those who don't must depend on the interests of those who do being aligned with their wants.

Now, here is where it gets interesting. There are all sorts of reasons why depending on the interests of others can work well in the world; after all, there's reasonable agreement over what is undesireable versus what is desireable; what is good vs what is bad. It's not Universal agreement, but it is there, and the more select the community the more likely it is that the agreement within that community is strong. So, for instance, it's clear that reporting a straightforward bug in a Free Software system, without providing a fix, is nevertheless likely to lead to the bug being fixed, as other individuals will weigh up the chance of being hit by the bug in the future against the effort required to fix it now.

What is different in this specific case, then? Why isn't this just a bug report? Well, as a cursory read of the relevant thread on the development mailing list would show, this issue was reported nine months ago, and no real attempt was made then to provide a workaround for the buggy OS vendor's tool. So this weblog posting isn't a bug report, it's an attempt to get other people to reprioritize what they are doing.

However, empirically, this has not bothered other people overmuch over the last nine months: no-one else has voiced a complaint, and no workaround has been written (obviously, since otherwise John would have nothing to complain about). So this might well be one of those cases where one's interests are not aligned closely enough with others; what to do, what to do? Well, one thing that John could do is to attempt to develop and install his own workaround for his situation.

John claims that he "... looked at trying to fix it [him]self, but as a first SBCL hacking project it was a little beyond [him]." This is certainly to be expected if he is deprived of all human contact, but here's where the interests of others can be put back in alignment: it is most certainly in the interests of developers of SBCL to have more people with enough understanding to be able to make non-trivial modifications to the source code. So while John may find that his attempt to spur developers to do his bidding is unsuccessful, he may also find that they are receptive, even without other consideration, to assisting him in solving his own issue.

The general point to make here is that, in at least this little corner of Free Software, where all the labour is volunteer, any user and developer community is an emergent property of individual and small-scale interactions; in this context, it is far from clear that making a loud complaint is productive for any party, whereas attempts to take personal responsibility, even if they ultimately need assistance, are more likely to bear fruit.

[*] All OSes are buggy. Not all of them prevent bug fixes.

[**] To his credit, he also claims to want to document the issue. Why he thinks a weblog post is better than the top hit that Google returns for "SBCL crash reporter", pointing to the mailing list discussion of nine months ago, I don't know.

[ Edited to fix a broken link ]

After a bit more fiddling around, I've merged Alastair Bridgewater's work (referred to in my previous diary entry) into HEAD of SBCL's CVS. This is far from an announcement of official support for SBCL on Windows; there is quite enough that doesn't work (see Alastair's TODO list or mine for ideas of what you could contribute) that interested parties need to fix before we could contemplate treating it as a stable platform.

Still, one step closer to destroying the software industry...

Windows can be made to look similar to Unix, from a suitable vantage point and given sufficient use of green-tinted glasses. (I watched The Muppets' Wizard of Oz yesterday, and oddly enough this seems like a good analogy: we're definitely not in Kansas any more).

The background: SBCL is a lisp environment (native-code compiler plus runtime) which is fairly solidly Unix-centric, and has been for the best part of 20 years or so; at its inception, it also ran on a Mach kernel. Among the Unixisms pervading the code are simple things such as parsing filenames; mild things like having streams associated with fds (rather than HANDLEs); and fairly fundamental differences in signals, exceptions, and handlers. This latter is important because a fair amount of runtime support is implemented in terms of Unix signals: the garbage collection is assisted by a write barrier; the debugger wants to be able to set breakpoints by writing an illegal instruction at the start of a function; and so on.

Why is this relevant? Well, I'm staring at a 65-page diff from Alastair Bridgewater, containing a sufficiently-complete port of SBCL to Windows. ("sufficiently-complete" here means that it self-hosts and mostly doesn't crash-and-burn instantly; it certainly does not mean that it's stable enough to run your mission-critical server on; whether you'd want to do that on Windows in any case I leave open.) This is the second iteration of this patch; Juho Snellman has already blogged about the first iteration, which was eight pages shorter; this was not the direction I hoped the patch would go, but oh well: I am assured that the extra eight pages include extra functionality, and certainly there is a lot less that causes screams of horror.

Alastair's made patch and binaries of the latest attempt available, as well as the beginnings of a TODO list: if there are Lisp-interested Windows hackers out there, more eyes and hands would be very welcome.

We have the beginnings of Climacs/PAIProlog integration. Code and screenshots soon. That is all.

Well, no, that is not at all "all". Work and multiferous other projects also continue to advance, some more glacially than others. Does anyone know where I can buy bottled time?

91 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!