Ian Bicking's Best of the web app test frameworks? sparked an interesting discussion (read down to the comments). Of particular interest to me was Ian's suggestion that twill (or, really, urllib2/httplib) be modified to send requests directly to a WSGI application without going through a TCP connection.
A few hours of hacking later, I've got a simple implementation that works (inside of twill). Briefly:
- I replace the HTTPHandler.http_open method with a call to my own HTTPConnection class, "myhttplib.MyHTTPConnection".
- myhttplib.MyHTTPConnection overrides the 'connect()' function so that
for *specific* host/port connections only, a fake socket is created.
- this fake socket, an object of type 'wsgi_fake_socket', intercepts
the HTTP traffic and behaves like a WSGI server, calling the app object
with the appropriate translated environment & catching the response.
- this response is then passed back up to the HTTPResponse object, and all is copacetic.
This solution is generic to httplib. It should be easy to pop it into anything that uses urllib2 to talk via HTTP... which means that virtually any Python Web testing code can use this kind of thing to talk directly to any Python WSGI application.
OK, so does it work? Yes!
For example, here's a simple script testing my conference submission system over TCP:
% ./twill-sh -u http://issola.caltech.edu/collar/ tst2 >> EXECUTING FILE tst2 ==> at http://issola.caltech.edu/collar/ ==> at http://issola.caltech.edu/collar/submit/ Note: submit is using submit button: name="view_status", value="view status" Note: submit is using submit button: name="view", value="view paper" -- 1 of 1 files SUCCEEDED.
Here's the same script running. This time it's pointed at a host/port that's diverted to WSGI:
% ./twill-sh -u http://floating.caltech.edu/collar/ tst2 >> EXECUTING FILE tst2 INTERCEPTING call to floating.caltech.edu:80 ==> at http://floating.caltech.edu/collar/ INTERCEPTING call to floating.caltech.edu:80 ==> at http://floating.caltech.edu/collar/submit/ Note: submit is using submit button: name="view_status", value="view status" INTERCEPTING call to floating.caltech.edu:80 INTERCEPTING call to floating.caltech.edu:80 Note: submit is using submit button: name="view", value="view paper" INTERCEPTING call to floating.caltech.edu:80 INTERCEPTING call to floating.caltech.edu:80 -- 1 of 1 files SUCCEEDED.
GET/POST, redirects, cookies, etc. are handled as they should be.
If you want to play with the code, it's available via darcs:
darcs get http://issola.caltech.edu/~t/twill/You can also just view myhttplib.py, which contains the entire implementation.
Some notes:
- The code is ugly; I know that. There are a number of things I could
do to it to make it nicer it, but at the moment I'm going to rest on my
laurels ;). Feel free to critique and/or patch.
- The 'make_environ' function is one of the two weakest links: it needs
to behave like a "real" Web server with respect to filling out the environment
dictionary, and that's tough. Right now I think it's handling cookies right,
but not much else.
- The other weak link is in the interplay between the read/write in the
wsgi_fake_socket. I'm assuming an awful lot here... in any case, it should
be possible to write a 2-way FIFO that properly mimics an open socket.
(Then HTTP/1.1 connections would work, too.)
- It should be fairly easy to pop in your own WSGI app, just for grins. You just need to modify myhttplib.wsgi_intercept appropriately, at any time before your first call to grab a URL.
Have fun,
--titus