twill implements a simple scripting language and emulates a
command-line browser. With twill, you can visit URLs, follow links,
fill out forms, log in and log out, deal with cookies, and follow
redirects. Because twill is written in Python, you can also script it
completely from within Python. And, because twill accurately emulates
the HTTP behavior of a browser, it's good for both screen-scraping Web
sites you don't control, and testing Web sites that you do control.
Note that twill does have a big flaw in this regard: it cannot
understand JavaScript. This is why "browser drivers" like Selenium and PAMIE are required for
testing AJAX applications. However, there are tradeoffs. One
tradeoff is that twill can run in a completely automated manner and
has no browser dependencies, which makes it particularly suitable for
unit tests.
Unit tests are a useful type of testing because they're
generally simple, test discrete components of the source code, and
are automated. Because unit tests are automated, there is
little or no cost to running them, which means developers can test
their code after even minor changes. This can lead to a very fluid
style of development in which major refactorings are thoroughly tested
at each step.
Using twill for unit tests brings in a new problem, however: the setup
and teardown of the Web site. This is true not just for twill but for
any other HTTP driver, such as urllib2, webtest,
webunit, mechanize, mechanoid, and zope.testbrowser:
you still have to set up your development Web site to serve pages,
so that it looks and behaves like your real site. In practice, this
breaks down into multiple sub-problems:
- you have to start
another process; that process has to bind to a port and a hostname
(usually something like localhost:8080);
- your unit tests have to
wait for the Web site to start up, which can take a second or two;
- and then you have to shut the Web site down at the end of the
unit tests.
Each of these is its own tricky problem. (What if
the port is already bound? What if your server is slow today?)
Moreover, proper profiling and code coverage testing of the Web
application become considerably more complicated when you're dealing
with multiple processes and "real, live" Web applications.
This is moderately complex stuff to script, and it's pretty daunting to
get it all working in a unit test framework (at least for me!). Plus,
having a more complicated test setup than most of the individual tests
seems like a bad design choice. While actually deploying a test Web
site is necessary to properly test a Web app, for your unit tests it's
almost certainly overkill. But what other options are there, you might ask?
Well, after a a
suggestion from Ian Bicking, I implemented an in-process
test harness for WSGI applications, called wsgi_intercept.
Briefly, wsgi_intercept lets you mount any WSGI-compliant Web
application as a fake remote server. Calls to this server from enabled
HTTP drivers are intercepted and fed directly into the WSGI application.
This means that as long as the Web framework you're using supports WSGI
-- and most of them do -- you can test your application without going
through forking a new process, binding a socket, or actually exposing
the application to external input. But, because wsgi_intercept acts at
the level of httplib, your application looks and behaves like it does
when it is actually bound to a socket.
twill comes with full wsgi_intercept integration, of course -- and you
can find hooks and examples for all of the other Python Web testing
frameworks at the
wsgi_intercept page. So how does it work in practice?
Roughly speaking, you need to do the following:
- package your
application as a WSGI app object;
- build a function to return that app
object;
- hook that function up to a particular host/port
combination;
- run whatever twill scripts you want to run for the
test;
- teardown (remove the intercept, shut down the server, etc.)
Below, I walk through setting up nose-based unit testing of two simple
applications -- a Quixote
application, and a CherryPy
application. In both cases most of the complexity is in putting the
start/stop colls in the correct order.
In Practice: Testing a Quixote application using twill and wsgi_intercept
You'll need Quixote 2.3 for this, as well as nose 0.8.x and twill 0.8.2.
For Quixote, let's test the mini_demo application that
comes with Quixote 2.x. Unfortunately, Quixote doesn't (yet?) come
with a WSGI interface; however, there's an adapter available
here,
as wsgi_server.QWIP. So we'll need wsgi_server.py, as well
as Quixote 2.x and nose.
First, create a test file; I'll just call it 'test.py'. Let's start by
roughing out a nose unit test:
class TestMiniDemo:
def setUp(self):
pass
def tearDown(self):
pass
Now, the Quixote Publisher object we want to test is returned
by quixote.demo.mini_demo.create_publisher(). We need to
dynamically create a function that creates the publisher, wraps it as
a WSGI object, and then returns that same object each time it's
called:
_cached_app = {}
def build_app(_cached_app=_cached_app):
if not _cached_app:
publisher = quixote.demo.mini_demo.create_publisher()
wsgi_app = wsgi_server.QWIP(publisher)
_cached_app['app'] = wsgi_app
return _cached_app['app']
If we put this in the setUp function, we have what we need.
The only tricky bit is caching the application object. Why do we need
to do this? Well, the function passed into wsgi_intercept is called
once for each intercepted connection, but we only want to create
the WSGI app object once. By storing the app object in a dictionary
that persists for the lifetime of the dynamically defined function, we
essentially memoize the WSGI application object. (I'm not thrilled with
this particular approach: let me know if you have a better way of doing
this.)
OK, once we have this function, we need to install it to handle
requests to localhost:8080, and then we're ready. Our final
setUp function looks like this:
def setUp(self):
_cached_app = {}
###
### dynamically created function to build & return a WSGI app
### for a Quixote Web app.
###
def build_app(_cached_app=_cached_app):
if not _cached_app:
# create a publisher obj
publisher = quixote.demo.mini_demo.create_publisher()
# wrap
wsgi_app = QWIP(publisher)
# save
_cached_app['app'] = wsgi_app
return _cached_app['app']
# install the app at localhost:8080 for wsgi_intercept
twill.add_wsgi_intercept('localhost', 8080, build_app)
# while we're at it, stop twill from running off at the mouth...
self.outp = StringIO()
twill.set_output(self.outp)
The tearDown function is much simpler: we just need to remove
the intercept, and then clear the Quixote publisher object.
def tearDown(self):
# remove intercept
twill.remove_wsgi_intercept('localhost', 8080)
# clear out the publisher
quixote.publish._publisher = None
...and now we're ready for a test or two! I'll define two: one to test
the main page, and the other to test the link.
def test_welcome(self):
script = "find 'Welcome to the Quixote demo'\n"
twill.execute_string(script, initial_url='http://localhost:8080/')
def test_hello(self):
script = """\
follow link
find 'Hello world!'
"""
twill.execute_string(script, initial_url='http://localhost:8080/')
Briefly, these scripts both go to 'localhost:8080'; the first script
makes sure that it can find specific text on the front page, while the
second script tests the result of following the front page link to
a 'Hello world' page. (Longer scripts can go in their own file,
and execute_file can be used to run them.)
Putting it all together with the correct import statements -- you can
download the final file here -- and running
nosetests, you get:
% nosetests
..
----------------------------------------------------------------------
Ran 2 tests in 0.381s
OK
So everything works! Huzzah! (If you want to reassure yourself that
it's actually running the tests through the Web application
break a test by changing the 'find' statements to something else;
see, they really are being run. ;)
In Practice: Testing a CherryPy (2.1) application using twill and wsgi_intercept
You'll need CherryPy 2.1.1 for this, along with nose 0.8.x and twill 0.8.2 (the very latest).
For CherryPy, let's test the "Hello, world!" application that is
included in the tutorial code. The magic incantations to get a WSGI
app object out of CherryPy are not so tricky:
import cherrypy
from cherrypy.tutorial.tut01_helloworld import HelloWorld
# set up the root object
cherrypy.root = HelloWorld()
# initialize
cherrypy.server.start(initOnly=True, serverClass=None)
# get WSGI app.
from cherrypy._cpwsgi import wsgiApp
where 'wsgiApp' is the final application object we wanted.
Starting with a 'test.py' containing a simple framework for a nose unit test,
class TestHelloWorld:
def setUp(self):
pass
def tearDown(self):
pass
we can fill in the setUp function as before:
def setUp(self):
_cached_app = {}
###
### dynamically created function to build & return a WSGI app
### for a CherryPy Web app.
###
def get_wsgi_app(_cached_app=_cached_app):
if not _cached_app:
# configure cherrypy to be quiet ;)
cherrypy.config.update({ "server.logToScreen" : False })
# create root & set up the server.
cherrypy.root = HelloWorld()
cherrypy.server.start(initOnly=True, serverClass=None)
# get WSGI app.
from cherrypy._cpwsgi import wsgiApp
_cached_app['app'] = wsgiApp
return _cached_app['app']
# install the app at localhost:8080 for wsgi_intercept
twill.add_wsgi_intercept('localhost', 8080, get_wsgi_app)
# while we're at it, snarf twill's output.
self.outp = StringIO()
twill.set_output(self.outp)
and the tearDown function is virtually identical to the Quixote
example:
def tearDown(self):
# remove intercept.
twill.remove_wsgi_intercept('localhost', 8080)
# shut down the cherrypy server.
cherrypy.server.stop()
This application is a bit simpler than the Quixote mini demo, so let's
just build one test function:
def test_hello(self):
script = "find 'Hello world!'"
twill.execute_string(string, initial_url='http://localhost:8080/')
and when we run it, voila! it all works:
% nosetests
.
----------------------------------------------------------------------
Ran 1 test in 0.289s
OK
Conclusions and Caveats
This code is all still quite young, but it works pretty well
for me. It's important to emphasize that you do need to run
some tests on a live site -- twill can be used for sites without much
JavaScript, while Selenium is probably the way to go for anything more
complicated. Still, using twill and wsgi_intercept to run tests
in-process is relatively simple and straightforward, and I think it can
be a very useful component of your Web app development process.
There are several strong conveniences to testing Web applications in
this way. The biggest, for me, is the avoidance of all the complicated
setup stuff. A close second is that code coverage analysis, profiling,
and even debugging can all run within your unit tests, because everything
is in-process. And a third is that unit tests run this way seem to
run quite a bit faster, perhaps because there's no setup/teardown of
the Web server.
One major caveat: it's unclear how well the wsgi_intercept stuff will
work with threaded Web app servers. twill doesn't use threads, so
this isn't an issue in this case, but if you use a threaded HTTP driver
with wsgi_intercept, you could run into problems.
If you have any suggestions, corrections, or explications, please send
them on to me at titus@caltech.edu.
I'll acknowledge them appropriately, I promise! I would also be interested
in examples for other Python Web frameworks; right now I only use CherryPy
and Quixote myself.
--titus
Software Links
twill Web browsing language: Web site & docs; download twill-0.8.2.tar.gz directly.
Quixote Web application framework: Web site; download Quixote-2.4.tar.gz directly.
CherryPy Web application framework: Web site; download CherryPy 2.1.1 from SourceForge.
nose unit testing framework: Web site & docs; download nose-0.8.6.tar.gz directly.
Mike Orr's WSGI wrapper for Quixote: view wsgi_server.py here. (The file itself is included in the source distribution for this article.)
Source distribution for this article: darcs repository at the usual place,
http://darcs.idyll.org/~t/projects/wsgi_intercept-examples/, or
download wsgi_intercept-examples-latest.tar.gz.
CTB 2/06