Older blog entries for itamar (starting at number 15)

I just got back from the week long Zope3 coding sprintathon in Rotterdam. Lots of fun, very educational, and now I have even more obligations of code I need to write for free software projects :)

The sprint was an impressive commentary on the success of free software as a commercial enterprise, with around 25-30 developers from 15 different organizations. Unlike Zope2, which was in general developed by a single company - Zope Corporation - Zope3 is being developed in a series of coding sprints in addition to the individual ongoing contribution of developers. The sprints and development are done, in general, by programmers at companies whose business is based on Zope2.

There are 95 companies listed as Zope Solution Proviers on zope.org, but in practice there are probably many more, as well as even more companies who sites run on Zope. No doubt almost all of them are interested in Zope3, and quite a few are putting in some resources (e.g. giving workers leave for a few days to go to a sprint) to make it happen.

On the technical side, it's interesting to compare Twisted to Zope3. While they have different purposes - Twisted is a general purpose networking framework and libraries, Zope3 is a web application server, they are both fairly large Python frameworks.

Someone called the Twisted developers "abstraction astronauts", but compared to Zope3 Twisted is so concrete you could build bomb shelters with it. This has a price of course, Zope3 is quite slow and only does 1 request a second on my laptop, but it has not been optimized at all. Knowing the people involved I don't doubt their claims that they have clear paths for optimization, and I'm sure the final release will have decent speed.

In any case, the abstraction is a good thing. Zope3 is an inherently complex application, and the new design is quite superior to the kludge known as Zope2. And Zope2 is the best content-management oriented web framework I know of (the next person to point to their favorite PHP project gets a $10 voucher for Toys R Us where they can buy some more toys.)

Twisted has been running applications for most of its existence. For example, http://twistedmatrix.com has been running using Twisted's web server for about 2 years. Since this also includes user homepages and the #python wiki (MoinMoin CGI), this provides some real world testing under real conditions. In addition, a number of commercial and open source projects are using Twisted, some of which have already been deployed. Zope3 on the other hand has not been used for any real world usage, other than developing itself - not even a test server for the development project. This will soon be changing as a couple of projects will probably start using it once the alpha is out.

I am confident in the Zope3 source code quality however, beyond the design itself. The use of interfaces means there's a clean separation between the requirements and the implementation, which does of course suck in some places. In addition, there are a *lot* of tests. Twisted has 75,000 lines of code, Zope3 has 150,000. Twisted has 350 automated tests and about 15 tests that need to be checked by users. Zope3 has 2700 automated tests! This will mean refactoring to deal with real world requirements will be much easier.

All in all, I'm really looking forward to Zope3. I'd probably be jumping up and down going "GIMME GIMME GIMME I WANT IT NOW" except I'm trying to get out of doing content management website jobs.

Spending my time getting the Cog object database up to speed for use in a project I'm working on. I've gotten it to the point that developers only need to mark their classes as persistent and then mark them as dirty when necessary, and it Does The Right Thing.

Next task is rewriting the storage backend, probably using the Python bindings to bsddb. Long term features I want to add are a backup script, GC script, hooks for indexing stored objects, maybe rollback support, but I'll do those after I finish the rest of the contract.

As usual in Python, you need to work a lot less than you might expect - Python's pickle library already has all the hooks necessary for serialization to an object database. So Cog can concentrate on doing all the other parts. It's quite a nice package even as is, other than a few rough spots.

This used to be a post with me complaining about Linksys and Seth Bromberger. Seems like I was wrong, and he did find the vulnerability two weeks before I did, so I aplogize to him and Linksys.

I submitted the paper glyph and I wrote for Usenix '2003, 25 minutes before the deadline. I hope they accept it.

I added multicast support to Twisted. Or to be more accurate, the tests for the multicast code pass - I have no idea if it works in the real world, and probably won't until we get a Real User to use the code.

Together with the rewrite of the old UDP API, we now support (in theory :) another small corner of the networking world.

I realized the issue isn't just with exceptions - it can be with any operation (e.g. log rotation on a signal might be interrupting a log write method, and who knows what happens then).

Yet another benefit of event loops is that the problem is easier to solve. Instead of raising an exception (or rotating the log or whatever), you do this by registering a method to run in the next iteration of the event loop. Because you are forced to split up your tasks into small, self-contained, short running event handlers anyway, your program by design will be much easier to structure so that this is not a problem. So, instead of rotating log on signal, you would tell event loop to run log rotation method in its next iteration.

I'm going to have to change Twisted's rotate-log-on-signal code and possibly other signal handlers so they work that way, if they don't already.

Software Methodology

My father wrote a paper on software methodologies which might be of interest to people here, especially if you're interesting in agile programming and the like.

On Saturday I was at the Lightweight Languages Workshop at MIT, and here are notes about some of the talks. There's a webcast available from the website.

Erlang

First talk was Erlang, which is very interesting. It's a functional language, and designed for concurrency - you can launch tens of thousands of processes, and pass messages between them. And this is done fast - 2 microseconds to start a process. Apparently Oz shares a lot of the concepts here as well.

OS-level services in Scheme

Second talk was about the ways PLT Scheme (MzScheme?) is incorporating OS-level functionality into the language.

PLT Scheme OS-like capabilities that I found interesting:

• killing threads cleanly
• Indepent GUI eventspaces (e.g. independent regarding modal dialogs).
• custodian manages resources for threads so you can easily shut it down.
• enforce abstractions (like Java's private?) and then add hooks for bypassing it for debugging.

This is very useful for IDEs. They have example IDE that lets you run say, infinite loop in interpreter, without the GUI freezing up, and because you can kill threads you can cancel this operation. In addition, GUI you program via IDE runs in its own event loop so it doesn't interfere with IDE's event loop.

Apparently the author is spending a lot of time making the blocking OS level facilities not block, so he can do this sort of thing.

Asynchronous Exceptions in Python

The fourth talk was about raising exceptions in signal handlers in Python, and the problem this causes. Basically the issue is that in e.g. this code:

f = open("file")
try:
    s = f.read()
finally:
    f.close()

the file may never be closed, since a signal might raise an exception right after file is opened but before try/finally. The obvious solution is too disable the signal around this code, but this is TOO MUCH TYPING, and you MIGHT FORGET to reenable the signal. So what was suggested? Adding new keywords! They gave following example code:

block:
    f = open("f")
    try:
        unblock:
            s = f.read()
    finally:
        f.close()

where block/unblock do that to signals. I think this is terribly wrong. First of all, their own example was wrong, it should actually have been:

block:
    f = open("f")
    try:
        unblock:
            s = f.read()
    finally:
	block:
            f.close()
        unblock:
            pass

This is totally ridiculous - you need to wrap each block of code this way, it blocks all signals instead of just the appropriate one, and the syntax totally violates the standard behavior of existing keyword pairs in python, since they are on different indentation levels.

I think raising exceptions from signal handlers is just a messy idiom, and they just shouldn't be doing it. And given you don't need a keyword to solve the problem, you just can use the signal module, their suggestion is just badly implemented syntactic sugar, with side effects that extend beyond the problem they want to solve.

I might write up the rest of the talks tommorow.

Cory Dodt suggested paying Iraqi soldiers to surrender. This actually works quite well - lets say you have 100,000 Iraqi soldiers. If we give each 1 million US$ to surrender, that comes to $100 billion - half the projected cost of the war. Quote from #twisted:

or maybe just pay his army to surrender. it would cost less than shipping 250,000 troops over there, i'll bet you. the oil companies would probably chip in too
we could get everyone involved. like sponsoring a starving child in africa. except you're sponsoring an iraqi to surrender.
i wonder if we could get them to write their sponsors letters. "thank you for not blowing me up. thanks to your generous donation of 1 million dollars, instead of being dead, i am now an oil magnate in my native country."

Tommorow I'll post my summary of Little Languages Workshop (preview - Erlang, I ask stupid questions and get put down in public, suggested Python atrocities, and more).

The International ANSWER coalition organized the Oct. 26 anti-war demonstration. There were between 100,000 and 200,000 in Washington DC, and another 20-80 thousand in San Francisco. That comes to a total of about a quarter of a million people. I was at the DC one, and did my small part to help organize it in the NY branch, and it was very impressive indeed.

VoteNoWar.org is their next effort, leading up to a Peace Congress and another, hopefully larger, demonstration, once more in DC. Please help out - sign it, donate, volunteer with your local organizations, pass it on to all your friends.

Raph Levien has kindly retrieved my password for me, so I can post once more. Thanks!