Older blog entries for joolean (starting at number 78)

r6rs-protobuf

I've started working on a project that depends on some of the code generation features of r6rs-protobuf (by way of a build-aux helper script) and I realized it's been handling library generation all wrong, at least for multi-.proto builds. The source of the trouble was my decision to map a Protocol Buffers package directly to an R6RS library. That's a fine choice if you can be sure that you know about all declarations of that package, but in a lot of cases, you don't -- for example, when you're applying the compiler to each .proto file in a list. Each file might re-declare that package for its own set of definitions, and since R6RS libraries are effectively immutable -- unlike Java packages or C++ namespaces -- you'll wind up generating a bunch of mutually conflicting libraries that share the same name.

So I've made a new release that changes that behavior. Going forward, the library generator will create a library per top-level definition in each package and name that library accordingly. There'll be a lot more libraries generated, but they'll actually be usable. Get it here.
gzochi

I've just released a new version of gzochi, my online game development framework. You should go get it! This release features resolves the most disruptive bugs from the first release and adds a bunch of cool new things like an interactive remote debugger, support for Berkeley DB, and new concurrent vector and hash table implementations. I think this is the point at which I'm going to start more actively dog-fooding with it, especially given that the Liberated Pixel Cup and the OpenGameArt folks have done such a great job of eliciting content from talented pixel artists.

One thing I'd known but hadn't really internalized: Example code takes forever to write. I added a new example game in gzochi 0.2, a scaled-down, Scheme-based port of the original AberMUD with an Ncurses client in C, and it took me weeks and weeks to get the structure of the code into a comprehensible, reasonably factored state and write all the code comments. Obviousy, there's tremendous benefit in having clear, well-annotated example code -- it's often the first thing people look at when they download your project -- so I think it's probably okay that it took me so long. But considering how quickly I did the rest of the work for the release, I think this might be the last example I add for a while.

A few notes on some older projects:

r6rs-protobuf

In the course of writing the lexer and the corresponding tests for r6rs-thrift I realized that r6rs-protobuf just flat out didn't support "//"-style comments. That's embarrassing -- not least of all because it made the library more or less unusable for real work, and thus nobody must have been using it successfully. But I've fixed that and assembled a new release. Get it here.

SCSS

Antono Vasiljev correctly pointed out that the API method scss:scss->css in SCSS doesn't seem to work with the same arguments as the function in the same name in Chicken's version of SCSS (which I only just found out about). In fact, it didn't work at all, following a redesign of SCSS's stylesheet data structure several versions and years ago. I've brought it up to date -- made it almost robust -- and after I make a few more fixes I'll put together a release of SCSS as well.

r6rs-thrift

I've just thrown together a release of a little project I started during this year's LibrePlanet conference, an implementation of the Apache Thrift framework in R6RS Scheme: r6rs-thrift. I thought it would be a fun, quick hack -- my assumption was that I could lift a lot of the type system and code generation stuff from r6rs-protobuf, my Scheme Protocol Buffers implementation from last year -- but it turned out that Thrift's a bit more complicated. Which is not to say that it wasn't fun to work on. But Thrift includes pluggable wire protocols, parameterized types, and some interesting variations on Protocol Buffers' vanilla structs (unions and exceptions). And there's not a whole lot (read: any) documentation, so I had to pretty much reverse engineer the serialization and codegen semantics by picking through the thriftc source code.

One interesting thing that came out Thrift's type system is that because you can effectively declare a type at the same time you're declaring a field of that type -- via, say, a container type as a field in a struct -- I had to add a layer of indirection to the type resolution system such that a descriptor for a user-defined type is declared (and registered) once and then used via an explicit "type reference" record everywhere else. Type references themselves can be parameterized to support Thrift's "generic" containers. I was initially worried about introducing that construct, but it ended up being reasonably discrete and quite powerful. I'm considering porting it back to r6rs-protobuf, just for, you know, funsies.

I was able to port the builder pattern that Protocol Buffers uses over to this implementation, which I thought was a neat little coup. Thrift is, you know, fine, but the mutability of the data types generated by the official compiler has always irked me, in part because it means there's no validation applied until you're about to marshal something over the wire. I suppose you could mount a philosophical defense to the effect that you don't really need an object to be well-formed until that point, but that presumes an awful lot about how you're planning to use your generated data structures.

At any rate: I made it; go get it.
gzochi

That was a quick four months! I've mostly had my head down developing this thing. I spent several weeks cranking out documentation, and then close to a month just working on the example game I wanted to ship with it, both on code clarity and on restructuring art assets. It's the first time I've ever shipped really comprehensive example code, and I'm very proud of the result, even if it has a decidedly retro flavor.

...Because I now have an 0.1 release that you can go download! It's actually very cool and hopefully easy to get started with (I also spent a week just futzing with Autotools scripts.)

As I pointed out on the GNU Guile mailing list, this was a project I started right after graduating from college, and it's what got me into Guile development in the first place: I had a much worse design in mind for how I was going to handle computing resource management back in 2003 (thread cancellation), which Guile didn't support at the time, so I committed to adding it. I've since learned almost ten years' worth of lessons about software architecture, and so this system doesn't capitalize on many features I was sure I needed way back when and thus set out to build from scratch -- dynamic user interface mark-up, e.g. -- but every detour was worth it. I think.

Now, if I could just remember what I was planning to do with a game development framework back in 2003...
gzochi

After releasing libRUIN a couple of months ago, I had some bandwidth to take on some new stuff, and I decided to revisit the project that had initially gotten me interested in user interface markup and GNU Guile in the first place: gzochi, my Free MMO game platform, ambitiously named after Xochipilli, the Aztec god of games and gambling.

My initial work on gzochi, while it took me down some interesting and fruitful paths (i.e., practically every Free Software project I've been working on for the past eight years), was probably all wrong. The system I'd envisioned involved a direct, first-class server-side model of the UI state of all connected clients, with practically nothing between that and the networking layer. I still want to use gzochi to explore things like dynamic UI deployment, but in the intervening years I've also learned that there's a layer for everything, and I wannt to keep the scope of each layer manageable.

So this time around I decided to decompose the problem a bit more aggressively, and I'm taking my architectural cues from a software project I'd worked with in a professional capacity, the Sun Game Server, aka Project Darkstar, aka RedDwarf Server, which takes the approach I had in mind insofar as providing a server container for games, but which makes no assumptions about the details of the games it hosts beyond providing some general "services" to them. To wit: There are message delivery services, a data retrieval service, and a task scheduling service; and "userland" game code is executed transactionally with respect to these services in order to hide the nasty details of concurrency.

The implementation of gzochi I'm currently pursuing steals the best parts of this architecture while replacing the server container bits with C code (with help from GNU Serveez) and replacing the game bits with Guile Scheme. I've got a working prototype of most of the system. Think this sounds interesting? Come aboard!
libRUIN

...version 0.2.0 is out! Holy cow, it's been four and a half years. There are a ton of improvements (do see the NEWS file if you're interested), but just as significantly, this release was made possible by improvements in and releases of a bunch of libRUIN's dependencies -- specifically GNU Guile and the SCSS and SDOM Scheme libraries.

And there were improvements (I hope) to my thinking about software development. If I can wax enlightened for a moment, here are a few observations / edicts that occurred to me as I was working on this release:

Avoid friction in code by getting the data model right.

I've found while working on libRUIN and other projects that code that's difficult to extend or refactor is often the result of an incomplete or incorrect data model. When I sketched out the initial architecture for the library, I think I was trying to keep the number of data structures it depended on low, both in the user-facing API and the internals. If data structure A has the same general shape as data structure B, I thought, why not just wedge A into this B-shaped hole? In retrospect, the additional size and complexity created by adding a few new struct typedefs was trivial compared to the disadvantages in legibility and flexibility brough about by inappropriate re-use of data structures. So: If your model needs a rich set of data structures to be accurate, so be it!

When a specification suggests a model, try that one first.

I spent a long time trying to figure out which parts of the CSS recommendation I could ignore by virtue of my only needing to render markup to a terminal. In doing so, though, I failed to pick up on the fact that a lot of the conceptual framework suggested by the recommendation is actually prescribed by it as well. For example, the "box model" isn't just a useful way of thinking about layout and rendering; it's really quite difficult to implement certain features of the recommendation without actually creating a first class representation of boxes. In retrospect this seems obvious, but at the time I was sure there was no way all of the formal complexity I saw in the spec could possibly be required for the work I was doing. The deeper I got, though, the clearer it became that the recommendation was the result of a long and fruitful two-way collaboration between the standards body and a number of successful "reference" implementations, and that ignoring the models derived from this relationship was actually making my life more difficult: Implementing support for things like the "white-space" property would be practically impossible unless I'd also baked the concepts of line boxes and inline content splits into my code. And I suspect a lot of successful specifications have in common the property that they are both describing requirements as well as the shortest path to meeting them. So: Until you've got a reason to do otherwise, let the specification guide your thinking.

Unit tests are worth it.

The original "test suite" for libRUIN was a set of XHTML documents and accompanying stylesheets from the W3C's CSS2.1 site. When I wanted to verify that a new feature was working or that I hadn't broken anything, I would feed these documents into a sample application and visually inspect the output to make sure it looked right. Obviously, moving to a suite of automated tests for each phase of the rendering pipeline made assessing the quality of the code much quicker. But I think the real benefit of unit tests is more subtle. By applying compile- and execution-time constraints to code, you control the way in which it can change. And by requiring that your code have entry points for unit tests to call, they enforce a layered, more modular architecture. Unit tests apply pressure on code that keeps it in good shape.

(This is probably all covered by a bunch of books and web sites about testing that I haven't read.)

That's all I've got. Take a look at libRUIN, and jump in if you can -- there's a ton more shit to do.
SCSS

I've just release SCSS 0.4.2, a bugfix release that builds upon the stability of 0.4.1. The bugfixes and API changes added here are a direct result of my making a lot of really exciting progress recently on a new, long-overdue release of libRUIN: The list of "I'm going to have to address that at some point"-type details -- for both SCSS and libRUIN -- is getting shorter and shorter.

Tarball is here. See also NEWS.
r6rs-protobuf

...version 0.2 is out! Downloads are here. I've added support for default values for optional fields and fixed some issues (mostly around enumerated types) that I discovered along the way.

One thing I noticed while writing SRFI-64 unit tests, though, was how neatly the `environment' procedure in the R6RS `(rnrs eval)' library works for testing code generation. My code generation routines return lists of datums representing expressions for creating the required bindings for a particular data type definition. `environment' allows for the creation of ad hoc "sandboxes" in which these binding expressions can be evaluated -- and in which test expressions can also be evaluated so that you can make assertions about the behavior of the stuff you bound. To wit:


(define test-env (environment '(rnrs)))
(for-each (lambda (expr) (eval expr test-env))
(protoc:generate-message my-message-def))
(test-assert (protobuf:message-builder? (eval '(make-my-message-builder) test-env)))


And you can create a new sandbox for every test, so that you don't have to worry about bindings generated by previous test cases polluting the environment of subsequent test cases that generate code for the same input definitions.
r6rs-protobuf

...version 0.1 is out! Check out the project page or go straight to the downloads. The thing's almost feature complete (haven't tackled "services" yet), but I decided to keep the version number low because the test suite is a bit anemic, and, of course, to account for bugs.

That was fun!

69 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!