Older blog entries for titus (starting at number 99)

Deep Throat: someone gets it

It was nice to read this article in the Missoulian about why Deep Throat is so important. (Even though they're obviously a bunch of left-wingers down there and shouldn't be taken seriously.)

Im Juli

Just saw the movie Im Juli; 'twas pressed on me by a lab-mate. Quite enjoyable.

Apropos of our time

From Gore Vidal: It is not enough to succeed. Others must fail.

Via someone on advogato a while back: Parable of the Two Programmers.

These resemble my life at the moment.

twill-ism

Fixed a bunch of small issues with twill. Getting ready for 0.7.1. I'd like to bump up to 0.8, but without at least a minimal unit-testing framework I just don't feel good about it. The current testing framework is just acceptance testing: does everything run? I need to move to unit testing to do more in-depth testing of the trickier rules regarding precedence of form fields when searching, etc.

--titus

Comparison between Io 'Vector' and Python 'numarray'

(Got this off of PlanetPerl, which I read occasionally, via ziggy.) A comparison of Io 'Vector' and Python 'numarray'. Ziggy was amazed by the basic comparison, but does it say anything about Python? numarray is mostly a C extension module, I thought... If loops etc. were done in Python, I'm guessing that's where the slowdown is.

Planet (Former) Advogato

This is nice; advogato has always had a a very high-quality collection of people posting there, even if my recent posts drag down the average. It would be amusing were someone to, say, create a Planet (Combined) Advogato that combined "former" and "current" advogato into a comprehensive RSS feed, together with commenting abilities.

Come to think of it, I have the necessary technology all ready to go. Hmm.

--titus

Ruby on Rails & Perl

RoR apparently spooked the Perl people, too: see Catalyst. I'm curious about whether or not the MVC model works well for Web apps; I don't see how it fits. I must be missing something.

Creationism: God's gift to the ignorant

Dawkins speaks!

New Yorker review of Revenge of the Sith

I saw RoS a few days ago, and enjoyed it. If you can block out the dialog, the pictures are very pwetty...

The New Yorker review by Anthony Lane is pretty amusing. To quote:

The general opinion of "Revenge of the Sith" is that it marks a distinct improvement on the last two episodes, "The Phantom Menace" and "Attack of the Clones". True, but only in the same way that dying from natural causes is preferable to crucifiction.

and, of course, there's the cheap shot at Yoda's character:

Deepest mind in the galaxy, apparently, and you still express yourself like a day-tripper with a dog-eared phrase book. "I hope right you are." Break me a fucking give.

Actual programming

Added a bunch of stuff to twill, and integrated some patches that were sent to me by Paul McGuire (pyparsing author) and Joeri van Ruth. Having an automated test suite that tests twill itself sure is nice: I managed to break about 5 different things in one day, and the test suite picked 'em all out. I haven't gotten to the point where I'm doing unit tests directly on the Python code, though, because ... unit testing mechanize properly is a pain. It involves all sorts of web setup stuff. Ick.

Countdown continues: T = -7 days 'til I'm leaving town.

--titus

Miscellany

I'm busily preparing for a 7-9 wk absence; I'll be attending the Embryology course at Woods Hole for 6 weeks, starting June 11th. I'll still have e-mail access, but it sounds like the course is nasty & brutish in its abuse of one's time, so I expect time for coding to be minimal.

One thing I'll need to do while at the course is admin my bioinformatics servers: the sysadmin they hired (after they fired the competent guy) has just quit. Whee. That'll be fun.

In other news:

I'm trying to get the Quixote persistent sessions package "out the door", which basically means (a) checking it into the cafepy.com qxtras SVN repository, (b) writing some minimal docs, and (c) shoving the rest onto Mike Orr, who has been busy getting the stuff to actually work. He suggested naming it session2, so that's its new name...

sessions2 will provide persistent session stores for shelve, a file/directory setup, Durus, PostgreSQL, and MySQL. It's easy to add new stores; I hope to add ZODB, sqlite, and SQLObject stores when I get back. I'm planning to write it up in my Quixote tutorial before I leave.

I'm also hoping to get out the next version of twill. It's going to take a few hours of programming to fix the bugs and get the requested new features in, but it seems like it's fairly solid at the moment. I've also created a twill mailing list so that people can help each other out, if they're having problems. Doubt it'll see much action, but ya gotta try, you know?

Meanwhile, Ian Bicking is trash-talking twill. I have this to say: he's absolutely right. If you have your Web application implemented within a WSGI-compatible app framework and can use Paste to do your testing, that's the way I'd do it. You're also going to want to be able to stress-test the deployment, and sometimes you have to worry about legacy applications, and you might want to test browser <--> code interactions like cookies. Or (heaven forfend!) you might be testing a package not written in Python, or you might be working with people who don't really program but are writing "black box" tests. I think twill could be useful in all of these situations, and that's what I'm aiming it at. People also seem to find it a neat concept, and the comments I've gotten indicate general enjoyment.

Personally, I'd like to see WSGI record/playback middleware. Should be pretty easy to get something simple working...

Bio + Python

Met with Chris Lee @ UCLA the other day, and talked about Pygr, blastgres, and bioinformatics in general. Chris was interested in putting more effort into open source Python software for bioinformatics, and wanted to know if I had suggestions, info, etc. He also showed me some of the pygr stuff, which looks like a fantastic way to take at least some of the repetitive scripting out of bioinformatics.

It sounds like he's hoping to make a more general toolkit that will fill the niche that (IMO) BioPython should be filling but isn't. (I may sign on for that effort once I defend, which will hopefully be by October or November.) Chris is completely into Python and open source, and he's already "let go" of the code; it's good to see.

Books!

I'm eagerly awaiting the release of Bonehunters, the 6th Malazan Book of the Fallen. At this point I'm about 2 yrs ahead of the US publishers; I've been getting the paperbacks from bagabooks, an Amazon reseller that must import them from England directly. (Let me tell you: it's worth it... a great series, and it's just getting better!) I've got a whole list of UK books that I hope to grab sometime soon, but I doubt I'll have the time or $$ to get them while I'm gone over the summer.

I've been reading the Dresden Files recently, by Jim Butcher. They're about a practicing wizard in Chicago named Harry Dresden. The series is starting to get repetitious; hopefully the next book will introduce something fresh. Still, they're good enough for my regular pulp fiction fix.

I also read Hallowed Hunt by Lois McMaster Bujold: fun, but not as good as the Curse of Chalion, which I really enjoyed. I just finished The Fly in the Cathedral, about the splitting of the atom; and Tracy Kidder's Soul of a New Machine. Some good stuff.

--titus

Quixote sessions implementation

I coded everything up, and now have persistent session stores for Durus, PostgreSQL, shelve, and pickles into individual files in a directory. See this message for specifics, if you're (a) interested but (b) not actually on the quixote-users mailing list.

I boiled my problems with the current Quixote SessionManager down a bit further, and I'm writing them down mainly for future reference. I know it's boring and incomprehensible. Sorry. I know I'm being obsessive. Oh well.

So: there are precisely four entry points into the SessionManager. There's the begin request hook, the two end request hooks (success/failure), and the expire_session call. All other functions are called from those four.

However, the code isn't really written that way. A lot of communication between subfunctions goes on in member variables (the self.sessions dictionary), rather than via return values. This complicates life when considering persistence, because I wasn't sure how stringent to be with respect to transactions. The code appeared to allow modifications to sessions at several points, and these changes needed to be tracked.

Once I rewrote the code so that values were passed around on the call stack, it became much less confusing. Now the only real problem is that expire_session can be called by user code at any time. This means that expired sessions have to be held in limbo until the end of the request, when they can be deleted. (If they were deleted immediately, then a failed request would need to "undelete" them by aborting the transaction. In this case you'd have to have a specific transaction going for *each* request, which is ... ugly.)

The code for doing this isn't as pretty as it could be, but I'll be damned if I can spot an obvious place to simplify it without giving up thread safety or transactions.

Quite a bit of detail work needs to be done for this to be a polished solution to session persistence in Quixote, but I think all that work will be done in the individual session storage classes. I'm fairly happy with the high-level session management stuff, and I hope that other people who look at it agree ;).

--titus

Quixote 2.0 and session handling

So I added a bit on session handling to my Quixote 2 tutorial and then promptly lapsed into a state of confusion about sessions. Partly because I'm not too happy with that part of the tutorial, and partly because I don't grok the API used for session management in Quixote.

The big problem with Quixote, incidentally, is that it doesn't come with any built-in session persistence mechanism. So everyone has to roll their own. I've done this about 3 times now, and I'm getting sick of it.

I spent some time over the last week or so beating my head against the wall and have these observations to make:

  • the session manager API exposed by SessionManager unnecessarily mimics a dictionary. This makes subclassing it a bit annoying, because you have a ton of extra functions to check;

  • the dictionary thing is a subset of a bigger problem, which is that I *thought* I needed to implement a much more robust API than I actually did. There's all sorts of functions in there, but really only four of them are ever called externally, and only one of those can be called at any time during an HTTP session. That one (expire_session) is still annoying, but all sorts of assumptions can be made about the timing of the other three (start_request, finish_successful_request, finish_failed_request).

    Moreover, the existing API is way too complicated for a single class. SessionManager needs to be broken down into about four different classes so that I can understand each one in its entirety.

  • the transactional interface suggested by the Quixote SessionManager is a pain in the butt to use with any non-object database, because it requires maintenance of a separate db connection for each session. This is probably due to the Quixote developers (a) using a process model and not a thread model, for which this doesn't matter so much; and (b) their use of ZODB and/or Durus, both object databases, in their own work.

The upshot is that yesterday I decided to simply ignore the nasty bits of the existing SessionManager and write my own. So I did so. One minorly neat thing about the reimplementation is that it understands transactional considerations (commit/abort) out of the box; no more hard-to-grok logic in the actual persistence mechanism. More importantly, my reimplementation supports a very simple persistence API:

class SessionStore:
    def load_session(self, id, default=None):
        pass

def save_session(self, session): pass

def delete_session(self, session): pass

def has_session(self, id): return self.load_session(id, None)

I don't think a persistence API can get any simpler than that ;).

I've already implemented shelve-based and directory-based session stores, and everything's working fine. I'm going to wait 'til I have a Durus- and PostgreSQL-based implementation before releasing it.

Oh, and incidentally, twill can be used to test this stuff, so I can package together the session store implementation, a test app, and a bunch of test scripts. Among other things this will let people write their own session stores and test them...

--titus

24 May 2005 (updated 24 May 2005 at 08:26 UTC) »
On Scotch

I have five different kinds of single-malt Scotch in the house at the moment: Balvenie Doublewood, a small amount of Laphroaig (my 2nd favorite), some Glendronnach that my brother bought my father, a small bottle of cask-strength Cadenhead's Speyside, and a recently acquired bottle of Lismore. The Lismore is the cheapest: Trader Joe's had it available for $15, so I picked it up on a whim. It's also the blandest; it's a fine low-key evening drink, but it's not up to the standards of even the Glendronnach, which is far from my favorite. (My absolute favorite is Lagavulin, which even my wife likes. I haven't been able to find a cheap bottle of it, though.)

twill revisited

Got two bug reports immediately after the release of twill 0.7. Ain't programming grand? ;)

Random links

The New Yorker has an excellent article on Intelligent Design. Summary: "As the years pass, intelligent design looks less and less like the science it claimed to be and more and more like an extended exercise in polemics."

Forget Star Wars: Store Wars! Surprisingly good.

Editing Hunter Thompson sounds like quite the experience. I'm frankly amazed so many people put up with him!

Sean McGrath talks about Joel Spolsky's making code look wrong. This is the only one of Joel's essays I've found utterly absurd, thus far. (For example, an anonymous commenter brought up htmltext, which is a much better way of dealing with XSS attacks than inventing some ugly naming convention.)

--titus

twill release

Released twill v0.7. This release implements all of the basic functionality and hopefully will generate some feedback upon which I'll rest future development.

This version is largely a rewrite of the IPython-based previous version. It's based on pyparsing and cmd rather than simple string rewriting and IPython. cmd, in particular, was a real eye-opener: with one simple metaclass and a bit of fiddling, I got a full shell with readline support, built-in help, and tab-completion of standard commands. Wow.

(Michele Simionato paid me one of the nicest compliments I've ever received in Python-land: paraphrased, "I think that's only the third real use I've seen for a metaclass." ;;)

For people who have no idea what twill is -- and presumably this is the vast majority of readers! -- it lets you write scripts like this:

  # go to the /. login page
  go http://slashdot.org/login.pl

# fill in the form fv 1 unickname test fv 1 upasswd test submit

# ok, there's no such account ;). show error HTML. show

I'm using it to do things like test Quixote, clear mailman SourceForge lists, and check whether or not sites are up. It's an (IMO) invaluable way to write simple scripts for Web testing. It's not original, incidentally: Cory Dodt wrote PBP, and I stole his ideas shamelessly.

I am rather proud of the implementation, because it's one of the smallest and neatest bits of self-contained code I've written in a while. Very satisfying. (Don't burst my bubble if you disagree, please ;).

Got a couple things to write about through this week: gun nuttery; books, books, and more books; my Quixote 2 tutorial & session handling; a Quixotic review of CherryPy; and (ab)use of metaclasses. Probably more. But for now, I need some sleep...

--titus

90 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!