Older blog entries for itamar (starting at number 19)

Hack of the day - Chopshop. Rips Shoutcast radio MP3 streams into separate MP3 files on your disk. It requires Twisted CVS (or 1.0.2 once released).

There are already existing programs that do this (Streamripper sounds nice, gotta steal that mp3 silence trick), but Chopshop is probably the shortest one - I wrote about 230 lines of code, half of it added to Twisted as generic Shoutcast protocol support.

Thanks to shlomif for the link to Neo-Tech, a hysterically funny cult. I highly recommend reading through the site (think of a cross between Heinlein's opinions as interpreted by a 16 year old, traditional rich right-wing American's political and economic agenda, and plain endearing wackiness.)

They call Bush a "closet secular humanist"! They explain how "The Zon/string-theory speculates that by orchestrating multi-dimensional quantum vibrations or gravity-unit ripples, technologically-advanced people could not only birth universes, but could design their initial conditions for optimum efficiency in developing invaluable conscious life yielding maximum returns on investment." They call Libertarians "left wing". They even explain what those mysterious Illuminati are up to.

In the immortal words of Pokey the Penguin: HOORJ!

I'm the first result of searching for "ontological proof" on Google.

The only way to deal with "logical proofs" of God's existence is VICIOUS MOCKERY!

Two weeks where I used computer a lot less, due to pain in wrists and hands. It's still not completely cleared up. I did feel well enough to finally release Cog version 0.3.

Cog is the Checkpointed Object Graph object database, providing semi-transparent persistence for large sets of interrelated Python objects. It handles automatic loading of objects on reference, and saving of modified objects back to disk. Reference counting is used to automatically remove no longer referenced objects from storage, and objects will be automatically be attached to the database if a persistent object references them.

It was originally written by Oliver Jowett, but I am now maintaining it.

I just got back from the week long Zope3 coding sprintathon in Rotterdam. Lots of fun, very educational, and now I have even more obligations of code I need to write for free software projects :)

The sprint was an impressive commentary on the success of free software as a commercial enterprise, with around 25-30 developers from 15 different organizations. Unlike Zope2, which was in general developed by a single company - Zope Corporation - Zope3 is being developed in a series of coding sprints in addition to the individual ongoing contribution of developers. The sprints and development are done, in general, by programmers at companies whose business is based on Zope2.

There are 95 companies listed as Zope Solution Proviers on zope.org, but in practice there are probably many more, as well as even more companies who sites run on Zope. No doubt almost all of them are interested in Zope3, and quite a few are putting in some resources (e.g. giving workers leave for a few days to go to a sprint) to make it happen.

On the technical side, it's interesting to compare Twisted to Zope3. While they have different purposes - Twisted is a general purpose networking framework and libraries, Zope3 is a web application server, they are both fairly large Python frameworks.

Someone called the Twisted developers "abstraction astronauts", but compared to Zope3 Twisted is so concrete you could build bomb shelters with it. This has a price of course, Zope3 is quite slow and only does 1 request a second on my laptop, but it has not been optimized at all. Knowing the people involved I don't doubt their claims that they have clear paths for optimization, and I'm sure the final release will have decent speed.

In any case, the abstraction is a good thing. Zope3 is an inherently complex application, and the new design is quite superior to the kludge known as Zope2. And Zope2 is the best content-management oriented web framework I know of (the next person to point to their favorite PHP project gets a $10 voucher for Toys R Us where they can buy some more toys.)

Twisted has been running applications for most of its existence. For example, http://twistedmatrix.com has been running using Twisted's web server for about 2 years. Since this also includes user homepages and the #python wiki (MoinMoin CGI), this provides some real world testing under real conditions. In addition, a number of commercial and open source projects are using Twisted, some of which have already been deployed. Zope3 on the other hand has not been used for any real world usage, other than developing itself - not even a test server for the development project. This will soon be changing as a couple of projects will probably start using it once the alpha is out.

I am confident in the Zope3 source code quality however, beyond the design itself. The use of interfaces means there's a clean separation between the requirements and the implementation, which does of course suck in some places. In addition, there are a *lot* of tests. Twisted has 75,000 lines of code, Zope3 has 150,000. Twisted has 350 automated tests and about 15 tests that need to be checked by users. Zope3 has 2700 automated tests! This will mean refactoring to deal with real world requirements will be much easier.

All in all, I'm really looking forward to Zope3. I'd probably be jumping up and down going "GIMME GIMME GIMME I WANT IT NOW" except I'm trying to get out of doing content management website jobs.

Spending my time getting the Cog object database up to speed for use in a project I'm working on. I've gotten it to the point that developers only need to mark their classes as persistent and then mark them as dirty when necessary, and it Does The Right Thing.

Next task is rewriting the storage backend, probably using the Python bindings to bsddb. Long term features I want to add are a backup script, GC script, hooks for indexing stored objects, maybe rollback support, but I'll do those after I finish the rest of the contract.

As usual in Python, you need to work a lot less than you might expect - Python's pickle library already has all the hooks necessary for serialization to an object database. So Cog can concentrate on doing all the other parts. It's quite a nice package even as is, other than a few rough spots.

24 Nov 2002 (updated 20 Mar 2003 at 05:15 UTC) »

This used to be a post with me complaining about Linksys and Seth Bromberger. Seems like I was wrong, and he did find the vulnerability two weeks before I did, so I aplogize to him and Linksys.

I submitted the paper glyph and I wrote for Usenix '2003, 25 minutes before the deadline. I hope they accept it.

17 Nov 2002 (updated 17 Nov 2002 at 05:36 UTC) »

I added multicast support to Twisted. Or to be more accurate, the tests for the multicast code pass - I have no idea if it works in the real world, and probably won't until we get a Real User to use the code.

Together with the rewrite of the old UDP API, we now support (in theory :) another small corner of the networking world.

Exceptions in signal handlers

I realized the issue isn't just with exceptions - it can be with any operation (e.g. log rotation on a signal might be interrupting a log write method, and who knows what happens then).

Yet another benefit of event loops is that the problem is easier to solve. Instead of raising an exception (or rotating the log or whatever), you do this by registering a method to run in the next iteration of the event loop. Because you are forced to split up your tasks into small, self-contained, short running event handlers anyway, your program by design will be much easier to structure so that this is not a problem. So, instead of rotating log on signal, you would tell event loop to run log rotation method in its next iteration.

I'm going to have to change Twisted's rotate-log-on-signal code and possibly other signal handlers so they work that way, if they don't already.

Software Methodology

My father wrote a paper on software methodologies which might be of interest to people here, especially if you're interesting in agile programming and the like.

10 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!