Older blog entries for joolean (starting at number 74)

gzochi

That was a quick four months! I've mostly had my head down developing this thing. I spent several weeks cranking out documentation, and then close to a month just working on the example game I wanted to ship with it, both on code clarity and on restructuring art assets. It's the first time I've ever shipped really comprehensive example code, and I'm very proud of the result, even if it has a decidedly retro flavor.

...Because I now have an 0.1 release that you can go download! It's actually very cool and hopefully easy to get started with (I also spent a week just futzing with Autotools scripts.)

As I pointed out on the GNU Guile mailing list, this was a project I started right after graduating from college, and it's what got me into Guile development in the first place: I had a much worse design in mind for how I was going to handle computing resource management back in 2003 (thread cancellation), which Guile didn't support at the time, so I committed to adding it. I've since learned almost ten years' worth of lessons about software architecture, and so this system doesn't capitalize on many features I was sure I needed way back when and thus set out to build from scratch -- dynamic user interface mark-up, e.g. -- but every detour was worth it. I think.

Now, if I could just remember what I was planning to do with a game development framework back in 2003...
gzochi

After releasing libRUIN a couple of months ago, I had some bandwidth to take on some new stuff, and I decided to revisit the project that had initially gotten me interested in user interface markup and GNU Guile in the first place: gzochi, my Free MMO game platform, ambitiously named after Xochipilli, the Aztec god of games and gambling.

My initial work on gzochi, while it took me down some interesting and fruitful paths (i.e., practically every Free Software project I've been working on for the past eight years), was probably all wrong. The system I'd envisioned involved a direct, first-class server-side model of the UI state of all connected clients, with practically nothing between that and the networking layer. I still want to use gzochi to explore things like dynamic UI deployment, but in the intervening years I've also learned that there's a layer for everything, and I wannt to keep the scope of each layer manageable.

So this time around I decided to decompose the problem a bit more aggressively, and I'm taking my architectural cues from a software project I'd worked with in a professional capacity, the Sun Game Server, aka Project Darkstar, aka RedDwarf Server, which takes the approach I had in mind insofar as providing a server container for games, but which makes no assumptions about the details of the games it hosts beyond providing some general "services" to them. To wit: There are message delivery services, a data retrieval service, and a task scheduling service; and "userland" game code is executed transactionally with respect to these services in order to hide the nasty details of concurrency.

The implementation of gzochi I'm currently pursuing steals the best parts of this architecture while replacing the server container bits with C code (with help from GNU Serveez) and replacing the game bits with Guile Scheme. I've got a working prototype of most of the system. Think this sounds interesting? Come aboard!
libRUIN

...version 0.2.0 is out! Holy cow, it's been four and a half years. There are a ton of improvements (do see the NEWS file if you're interested), but just as significantly, this release was made possible by improvements in and releases of a bunch of libRUIN's dependencies -- specifically GNU Guile and the SCSS and SDOM Scheme libraries.

And there were improvements (I hope) to my thinking about software development. If I can wax enlightened for a moment, here are a few observations / edicts that occurred to me as I was working on this release:

Avoid friction in code by getting the data model right.

I've found while working on libRUIN and other projects that code that's difficult to extend or refactor is often the result of an incomplete or incorrect data model. When I sketched out the initial architecture for the library, I think I was trying to keep the number of data structures it depended on low, both in the user-facing API and the internals. If data structure A has the same general shape as data structure B, I thought, why not just wedge A into this B-shaped hole? In retrospect, the additional size and complexity created by adding a few new struct typedefs was trivial compared to the disadvantages in legibility and flexibility brough about by inappropriate re-use of data structures. So: If your model needs a rich set of data structures to be accurate, so be it!

When a specification suggests a model, try that one first.

I spent a long time trying to figure out which parts of the CSS recommendation I could ignore by virtue of my only needing to render markup to a terminal. In doing so, though, I failed to pick up on the fact that a lot of the conceptual framework suggested by the recommendation is actually prescribed by it as well. For example, the "box model" isn't just a useful way of thinking about layout and rendering; it's really quite difficult to implement certain features of the recommendation without actually creating a first class representation of boxes. In retrospect this seems obvious, but at the time I was sure there was no way all of the formal complexity I saw in the spec could possibly be required for the work I was doing. The deeper I got, though, the clearer it became that the recommendation was the result of a long and fruitful two-way collaboration between the standards body and a number of successful "reference" implementations, and that ignoring the models derived from this relationship was actually making my life more difficult: Implementing support for things like the "white-space" property would be practically impossible unless I'd also baked the concepts of line boxes and inline content splits into my code. And I suspect a lot of successful specifications have in common the property that they are both describing requirements as well as the shortest path to meeting them. So: Until you've got a reason to do otherwise, let the specification guide your thinking.

Unit tests are worth it.

The original "test suite" for libRUIN was a set of XHTML documents and accompanying stylesheets from the W3C's CSS2.1 site. When I wanted to verify that a new feature was working or that I hadn't broken anything, I would feed these documents into a sample application and visually inspect the output to make sure it looked right. Obviously, moving to a suite of automated tests for each phase of the rendering pipeline made assessing the quality of the code much quicker. But I think the real benefit of unit tests is more subtle. By applying compile- and execution-time constraints to code, you control the way in which it can change. And by requiring that your code have entry points for unit tests to call, they enforce a layered, more modular architecture. Unit tests apply pressure on code that keeps it in good shape.

(This is probably all covered by a bunch of books and web sites about testing that I haven't read.)

That's all I've got. Take a look at libRUIN, and jump in if you can -- there's a ton more shit to do.
SCSS

I've just release SCSS 0.4.2, a bugfix release that builds upon the stability of 0.4.1. The bugfixes and API changes added here are a direct result of my making a lot of really exciting progress recently on a new, long-overdue release of libRUIN: The list of "I'm going to have to address that at some point"-type details -- for both SCSS and libRUIN -- is getting shorter and shorter.

Tarball is here. See also NEWS.
r6rs-protobuf

...version 0.2 is out! Downloads are here. I've added support for default values for optional fields and fixed some issues (mostly around enumerated types) that I discovered along the way.

One thing I noticed while writing SRFI-64 unit tests, though, was how neatly the `environment' procedure in the R6RS `(rnrs eval)' library works for testing code generation. My code generation routines return lists of datums representing expressions for creating the required bindings for a particular data type definition. `environment' allows for the creation of ad hoc "sandboxes" in which these binding expressions can be evaluated -- and in which test expressions can also be evaluated so that you can make assertions about the behavior of the stuff you bound. To wit:


(define test-env (environment '(rnrs)))
(for-each (lambda (expr) (eval expr test-env))
(protoc:generate-message my-message-def))
(test-assert (protobuf:message-builder? (eval '(make-my-message-builder) test-env)))


And you can create a new sandbox for every test, so that you don't have to worry about bindings generated by previous test cases polluting the environment of subsequent test cases that generate code for the same input definitions.
r6rs-protobuf

...version 0.1 is out! Check out the project page or go straight to the downloads. The thing's almost feature complete (haven't tackled "services" yet), but I decided to keep the version number low because the test suite is a bit anemic, and, of course, to account for bugs.

That was fun!

r6rs-protobuf

In the wake of the glorious Guile 2.0 release, I've been working on a prototype re-implementation of GIMP's Script-Fu plugin system that uses Guile in lieu of their embedded TinyScheme. I actually got reasonably far, but then the onboard video on my beloved ZaReason UltraLap SR straight up died, so I had to sideline that project while I waited for a replacement (in the form of a new ZaReason Terra HD) to arrive.

Although I'd temporarily lost access to my GIMP patches, I did have comfortably obsolete desktop machine available to me to use to satisfy my constant, ravening need to write programs all the time. I decided that doing an R6RS Scheme implementation of Google Protocol Buffers would make a cool, discrete project: I use and enjoy protobufs at my day job, and I had a hunch that their requirements would map nicely onto the features provided by R6RS records and enumerations. It turns out I was right about that, although I've now sunk a bit more time than I was expecting into the project. But I did just manage to get an initial import of some working libraries uploaded to Google Code. Check it out here.

Guile 2.0

...is finally out! Publicity follows:

And we got a link on the GNU Project home page!

Ludovic Courtès, who actually assembled the release, notes in the release docs that 2.0's been in the works for three years. I could've sworn it was longer than that, maybe because the impetus for the changes in this release has been building for quite some time. I've been working on Guile (and whining about it, and more often than not sitting back and watching other people do brilliant things in it), at various rates of productivity, since I left school eight (!) years ago, and consequently I've been able to watch it as it transformed from a project that was more or less in a holding pattern into one that's rapidly improving and incorporating modern language features without compromising the level of stability it's known for.

Even though I came to it because of its designation as the "official" GNU extension language, I've always liked Guile -- I've found it to be more accessible than other Scheme platforms for both the embedded and interactive use cases -- but I'm aware that some people hold a negative opinion of it. Whatever your past experience with Guile, I think it's changed enough with this release that it's strongly worth another look. The pain points have been addressed head-on, and the good parts have gotten even better. Look at that feature set and tell me you don't start imagining applications.

Many, many thanks to Ludovic and to Andy Wingo, who tackled the hardest problems and accomplished some unbelievable things.

SCSS

I've just released SCSS 0.4.0, which brings with it some pretty major changes. For one, like the recent release of SDOM, it repackages the code as an R6RS library. Not having to chase Scheme platform-compatibility issues is an incredible relief. But I also decided to change the way queries to the cascade work.

In previous versions, you'd specify a node and a property whose value you wanted to obtain for that particular node. The API guaranteed you a non-null response to your query, even if satisfying it meant that additional computation had to be done, like extracting a sub-value from a more general property for use as the value of a more specific property that wasn't set (e.g., figuring out border-top-style from the value of border); or re-using the result of repeating the query for an alternate, related property (e.g., obtaining the value for border-color from color); or just looking up a default value in a reference table. To offset the significant cost of repeat lookups to the cascade, which are not cheap by any means, I'd devised an elaborate system of cascading caches to store values that were discovered but not currently required. This worked reasonably well, but brought with it some frustrating complexity -- for example, those caches had to be flushed based on criteria external to SCSS, like the activation of pseudo-classes and pseudo-elements.

So I've decided to ditch all of that internal complexity at the cost of passing on a fraction of it to the user of the API: I've modified the selection API such that you no longer ask about specific properties, and you get back the complete set of property values that have been assigned to the node. If you need to increase their specificity or look up a default value, there's an auxiliary API for doing those things. It's already had a positive impact in the only client of SCSS that I know of, libRUIN, enabling some more accurate inheritance behavior, and allowing me to scrap a major swath of CSS lookup code that was coupled to assumptions about the way SCSS worked. It would be nice to think that I've reached a point in my development as a software designer that I could afford to make some tentative pronouncements about "best practices." So: APIs that purport to hide, in one shot, all of the complexity that's inherent in a particular source of data are almost certainly not going to be able to make good on that claim. Better to design software that addresses the problem in layers, with each layer solving a very small part.

Another pronouncement I'd like to make is that there's a lot of benefit to be had by taking advantage of language features for structuring data. I'd been relying on Scheme lists for modeling pretty much everything in the system, from the arguments to the value selection calls to the intermediate result values passed around internally. I've since switched those things over to use R6RS records, which makes reasoning about the input and output of functions a lot easier. There used to be a half dozen inputs to the selection API functions, many of them strongly related; I've grouped them into records such that there are now two arguments, one of which will often be invariant over a single application's use of the API.

Phew!

Guile

2.0 is nearly upon us! The release is scheduled for the second week of February. I'm pretty excited about it, given that it represents several years of sporadic work by me, not to mention years of more consistent efforts by lots of brilliant and talented people. We're trying to figure out how to frame the transformative nature of this release -- Guile's finally become what it set out to be, an actual scripting language platform. Now we have to figure out how to communicate that to people. This is where marketing comes in, I suppose.

SDOM

While waiting for Guile 2.0 to come out, I decided to eat some of my own dog food, so to speak, and take on a serious project that made use of Guile's new support for R6RS. I'd been meaning to revisit SDOM for some time now, in order to address some of the shortcomings I'd noticed through using it over the years -- primarily its adherence to SXML's expression format, which was a performance drain when it came to doing things like looking up attribute values and child and parent nodes; and which required a fair amount of nastiness to manage things like metadata on text nodes. Plus I can't think of any other DOM implementations out there that store data "inline" with an XML representation (unless, I guess, they do parsing as well as DOM manipulation).

So I rewrote the thing as an R6RS library using R6RS records to model the Node interface and its children, and released the result as SDOM 0.5. The process was time-consuming but not hellish, per se, mostly due to the fact that I'd had a fairly extensive test suite in place. And working with records instead of semi-circular lists is just so much neater and easier. I'm still trying to gauge its performance relative to the older version; I'm still doing some complicated things, mostly to optimize the read case for complex properties like "wholeText." And, in theory, the whole thing is now portable to every Scheme that supports R6RS, which is getting to be most of them.

Except that...

SXML

...doesn't have a "standard" R6RS library packaging. At the moment, there are a couple of distributions that some industrious people have assembled -- I'm thinking of Wak and Xitomatl -- but these both provide SXML as part of (and with dependencies on) a larger framework of code that I'm just not interested in (and I think one or both of them is missing the crucial make-parser macro). My feeling is that an XML parser is such a fundamental library for a language platform that it's gotta be pretty much an atom. I should just be able to say (import (sxml ssax)) and be done with it.

I wonder if guile-lib's version (now incorporated into Guile 2.0) might make a good candidate for a "universal" SXML R6RS library distribution.

65 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!