Older blog entries for titus (starting at number 123)

19 Nov 2005 (updated 19 Nov 2005 at 00:44 UTC) »
Unit tests save my butt

Just updated twill to the latest versions of the wwwsearch/mechanize code. Because twill reaches moderately far into the wwwsearch code, some of the internal changes that John Lee made to mechanize affected the functioning of twill adversely.

Conveniently, my unit tests caught many problems and I could iterate through & fix them one by one.

Dunno what I would have done without 'em... probably slapped a "beta" sticker on it and asked my users to test it for me ;).

Web testing links

Grig's recent roundup of Web testing tools missed a few. (OK, to be fair not all of them are "recently updated"!) I've put 'em all into the twill README. It's quite a list!

Good experiences with Python embedding

For you planetpython people,

Iago Rubio talks about how wonderful Python embedding can be...


Delusions of grandeur

Sparked by Ian Bicking's simple implementation of the twill language in JavaScript, as well as Robert Marchetti's Python-IE bridge project, PAMIE, I googled about and found a very recent article on PyXPCOM, too.

It should be relatively easy to build a common API layer for JavaScript, mechanize/mechanoid/zope.testbrowser, PAMIE, and PyXPCOM that supports the twill language. Then you'd be able to have a single twill script that runs in-browser & from the command line, and also manipulates PAMIE and PyXPCOM. Wouldn't that be nice?

Just a thought.


E-mail to my titus at caltech.edu address is failing. If, for some reason, you feel the need to contact me ... use titus at idyll.org.

For now I've had to set my Reply-To header to that, too. How frustrating; I'm sure I've lost e-mail, but I have no idea what, and I have no ability to fix the problem. I feel like a user. Argh.

(Nothing to see here. Continue about your business.)


A Plea for Python

Seth Vidal of fedoraproject.org has a request -- stop asking for PHP apps, and start using Python!

To quote,

With that in mind I'd like to make a request out to the python web programmers in the fedora-verse. We need some people who are willing to work on web applications either in zope or turbogears. Let's see some options and ways to progress. We have a lot of python programmers who can help audit the code and contribute sections as we build up our module-base.

Yo. What he said.

Interactive exploration of Web apps

One (apparently) whiz-bang feature of PBP, and now twill, is the ability to interactively browse your Web apps from the command line. For example,

>> go http://www.advogato.org/
==> at http://www.advogato.org/
current page: http://www.advogato.org/
>> showlinks

0. Articles ==> /article/ 1. Account ==> /acct/ 2. People ==> /person/ 3. Projects ==> /proj/ 4. Omnifarious ==> /person/Omnifarious/ 5. Read more... ==> /article/862.html 6. tarzeau ==> /person/tarzeau/ ... current page: http://www.advogato.org/ >> follow tarzeau ==> at http://www.advogato.org/person/tarzeau/ current page: http://www.advogato.org/person/tarzeau/ >> showlinks Links:

0. http://www.linuks.mine.nu/ ==> http://www.linuks.mine.nu/ 1. freshmeat.net/~tarzeau/ ==> http://freshmeat.net/~tarzeau/ 2. iBackup ==> /proj/iBackup/ 3. AmigaSHELL ==> /proj/AmigaSHELL/ ...

This inevitably receives the most "oohs" and "ahhs" the times I've shown twill to people sitting next to me. Apparently none of the other Web testing tools out there do this.

Anyway, I mention it because with the in-process WSGI testing patch, you can now browse your WSGI apps interactively without going through the Internet.

That strikes me as pretty nifty, if only in a Python-geek kind of way.

Go about your business. These are not the comments you're looking for.


13 Nov 2005 (updated 13 Nov 2005 at 20:28 UTC) »
twill & in-process testing of WSGI apps

Ian Bicking's Best of the web app test frameworks? sparked an interesting discussion (read down to the comments). Of particular interest to me was Ian's suggestion that twill (or, really, urllib2/httplib) be modified to send requests directly to a WSGI application without going through a TCP connection.

A few hours of hacking later, I've got a simple implementation that works (inside of twill). Briefly:

  • I replace the HTTPHandler.http_open method with a call to my own HTTPConnection class, "myhttplib.MyHTTPConnection".

  • myhttplib.MyHTTPConnection overrides the 'connect()' function so that for *specific* host/port connections only, a fake socket is created.

  • this fake socket, an object of type 'wsgi_fake_socket', intercepts the HTTP traffic and behaves like a WSGI server, calling the app object with the appropriate translated environment & catching the response.

  • this response is then passed back up to the HTTPResponse object, and all is copacetic.

This solution is generic to httplib. It should be easy to pop it into anything that uses urllib2 to talk via HTTP... which means that virtually any Python Web testing code can use this kind of thing to talk directly to any Python WSGI application.

OK, so does it work? Yes!

For example, here's a simple script testing my conference submission system over TCP:

% ./twill-sh -u http://issola.caltech.edu/collar/ tst2
==> at http://issola.caltech.edu/collar/
==> at http://issola.caltech.edu/collar/submit/
Note: submit is using submit button: name="view_status", value="view status"
Note: submit is using submit button: name="view", value="view paper"
1 of 1 files SUCCEEDED.

Here's the same script running. This time it's pointed at a host/port that's diverted to WSGI:

% ./twill-sh -u http://floating.caltech.edu/collar/ tst2
INTERCEPTING call to floating.caltech.edu:80
==> at http://floating.caltech.edu/collar/
INTERCEPTING call to floating.caltech.edu:80
==> at http://floating.caltech.edu/collar/submit/
Note: submit is using submit button: name="view_status", value="view status"
INTERCEPTING call to floating.caltech.edu:80
INTERCEPTING call to floating.caltech.edu:80
Note: submit is using submit button: name="view", value="view paper"
INTERCEPTING call to floating.caltech.edu:80
INTERCEPTING call to floating.caltech.edu:80
1 of 1 files SUCCEEDED.

GET/POST, redirects, cookies, etc. are handled as they should be.

If you want to play with the code, it's available via darcs:

darcs get http://issola.caltech.edu/~t/twill/
You can also just view myhttplib.py, which contains the entire implementation.

Some notes:

  • The code is ugly; I know that. There are a number of things I could do to it to make it nicer it, but at the moment I'm going to rest on my laurels ;). Feel free to critique and/or patch.

  • The 'make_environ' function is one of the two weakest links: it needs to behave like a "real" Web server with respect to filling out the environment dictionary, and that's tough. Right now I think it's handling cookies right, but not much else.

  • The other weak link is in the interplay between the read/write in the wsgi_fake_socket. I'm assuming an awful lot here... in any case, it should be possible to write a 2-way FIFO that properly mimics an open socket. (Then HTTP/1.1 connections would work, too.)

  • It should be fairly easy to pop in your own WSGI app, just for grins. You just need to modify myhttplib.wsgi_intercept appropriately, at any time before your first call to grab a URL.

Have fun,

RELEASE: twill 0.7.4

Following the dictat of my users... a new twill release!

Links: Cheese Shop entry, announcement, ChangeLog, and download.

There are lots of bug fixes -- cold filtered for a smooth browsing experience. The main innovation in this release is the unit testing: now, twill uses nose-based unit testing, and it also provides some simple commands to help users use twill to unit test their own apps.

Sample unit-test code:

    import os
    import testlib
    import twill.unit
    import twilltestserver
    from quixote.server.simple_server import run as quixote_run

def test(): # port to run the server on PORT=8090

# create a function to run the server def run_server_fn(): quixote_run(twilltestserver.create_publisher, port=PORT)

# abspath to the script script = os.path.join(testlib.testdir, 'test-unit-support.twill')

# create test_info object test_info = twill.unit.TestInfo(script, run_server_fn, PORT)

# run tests! twill.unit.run_test(test_info)

(See unit testing with twill for more info.)

Comments welcome...


a Perl-er learning Python

Interesting reading.


"David" posted about PyFLTK -- fantastic! I use FLTK from C++ and I've been waiting for the day when PyFLTK became usable. And it is!

(It's amusing to see how quickly this thread degenerated into a "but why not *my* favorite GUI?" argument.)


Inventive. But not very effective.

a I m A r I
n A b L o A
a G i I z L
x R e U a I
A n M c S


Web testing with twill

Moving towards version 0.8; just added a simple unit-testing structure that works via 'nose'. Running tests just got a whole lot easier...

Also spent a not-so-productive half-day working through a number of issues (ChangeLog). Now that I'm starting to use it for more complicated sites, I'm finding more problems. Whee...

Someone mentioned an interest in making it easier to integrate twill with unit testing frameworks. Now that I'm using 'nose', I am starting to get a better idea of what that might entail: fork() plus a kill() at the end is what I'm using. Hmm.

More O'Reilly publicity, this time via Michele Simionato's article on Web testing.

Oddly enough, it appears twill is written in Perl. Heh. Well, it's a 'P' language, at any rate...

I'm still working on what "peste e corna" means -- "pestilence and corn"? -- but people should be aware that my intense hatred of Zope stems from experiences with Zope 1. Or maybe Zope 2. Not sure. At any rate, I haven't looked at it for a loooong time. It's probably great now...


2 Nov 2005 (updated 2 Nov 2005 at 20:07 UTC) »
Training log software

My wife has been looking for training log software for Mac OS X. She's interested in finding something to keep track of when and how far she biked, run, and swam. So far she hasn't found anything that runs on her local computer -- mostly, there's a bunch of Web-based things, which she doesn't want to use.

She's not found anything obvious with google, but my suspicion is that there are some packages lurking out there under different keywords.

Ideas welcome -- I'll summarize here, if anyone has any ideas to send me.


Why should I care what color the bikeshed is?

Good question!

Geek joke

Question: Why should women marry open-source hackers?

Answer: Because open-source hackers are already used to dealing with badly-documented software.

(It was funny at the time...)

Bioinformatics lectures

Just gave a lecture on gene finding & will give another lecture on regulatory regions on Friday. E-mail me if you want the outlines or powerpoints.


Post-modern programming

I confess, I am a post-modern programmer. But Python makes it so easy...

twill publicity

Grig pointed out this short article at oreillynet.com: Web App Security Testing Using twill.

(who knew that writing documentation was such a good idea? ;)


As part of another project, I finally committed to a rewrite of cucumber (as I'd hoped to back in January). Announcing... cucumber2!

cucumber was a getattr-based object-relational mapping system for Python/PostgreSQL that used PostgreSQL's table inheritance to map objects into SQL tables without impedance mismatch.

cucumber2 is a more robust rewrite of cucumber. cucumber2 is based on new-style classes, metaclasses and properties, which means that things like inheritance (including multiple inheritance) work easily and directly.

cucumber has long been my pet project: an immensely useful little tool that I never got around to properly releasing. cucumber2 is much prettier in many ways, and I look forward to using it in several projects now that the package is cleaner.

At any rate, I'm making a very young cucumber2 public. For anyone who is interested, there's an overview available that's probably the place to start.

A few random musings:

  • First and foremost, metaclasses are wicked cool.

  • Guido's 2.2 changes are very, very nice and put lots of power in the hands of the developer.

  • Metaclasses are a very convenient way to do code generation. (I learned this trick from Daniel Arbuckle at a socal-piggies presentation.)

  • nose Just Works.

  • Strive for simplicity. It's in there somewhere, and when you find it, it's worth it.


114 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!