Introduction: Testing Python Web applications using twill and wsgi_intercept

Posted 7 Feb 2006 at 17:49 UTC by titus Share This

One of the thorniest problems in GUI application development is how to test your user interface. Web applications, as a specific and somewhat limited example of a GUI, are no exception to this problem. However, there are several options for testing Web applications now available. One of my favorites is twill, a remote HTTP driver application that lets you script Web sites. (Disclaimer: I am the primary author of twill, so take my recommendation with a grain of salt ;).

twill implements a simple scripting language and emulates a command-line browser. With twill, you can visit URLs, follow links, fill out forms, log in and log out, deal with cookies, and follow redirects. Because twill is written in Python, you can also script it completely from within Python. And, because twill accurately emulates the HTTP behavior of a browser, it's good for both screen-scraping Web sites you don't control, and testing Web sites that you do control. Note that twill does have a big flaw in this regard: it cannot understand JavaScript. This is why "browser drivers" like Selenium and PAMIE are required for testing AJAX applications. However, there are tradeoffs. One tradeoff is that twill can run in a completely automated manner and has no browser dependencies, which makes it particularly suitable for unit tests.

Unit tests are a useful type of testing because they're generally simple, test discrete components of the source code, and are automated. Because unit tests are automated, there is little or no cost to running them, which means developers can test their code after even minor changes. This can lead to a very fluid style of development in which major refactorings are thoroughly tested at each step.

Using twill for unit tests brings in a new problem, however: the setup and teardown of the Web site. This is true not just for twill but for any other HTTP driver, such as urllib2, webtest, webunit, mechanize, mechanoid, and zope.testbrowser: you still have to set up your development Web site to serve pages, so that it looks and behaves like your real site. In practice, this breaks down into multiple sub-problems:

  • you have to start another process; that process has to bind to a port and a hostname (usually something like localhost:8080);
  • your unit tests have to wait for the Web site to start up, which can take a second or two;

  • and then you have to shut the Web site down at the end of the unit tests.
Each of these is its own tricky problem. (What if the port is already bound? What if your server is slow today?) Moreover, proper profiling and code coverage testing of the Web application become considerably more complicated when you're dealing with multiple processes and "real, live" Web applications.

This is moderately complex stuff to script, and it's pretty daunting to get it all working in a unit test framework (at least for me!). Plus, having a more complicated test setup than most of the individual tests seems like a bad design choice. While actually deploying a test Web site is necessary to properly test a Web app, for your unit tests it's almost certainly overkill. But what other options are there, you might ask?

Well, after a a suggestion from Ian Bicking, I implemented an in-process test harness for WSGI applications, called wsgi_intercept. Briefly, wsgi_intercept lets you mount any WSGI-compliant Web application as a fake remote server. Calls to this server from enabled HTTP drivers are intercepted and fed directly into the WSGI application. This means that as long as the Web framework you're using supports WSGI -- and most of them do -- you can test your application without going through forking a new process, binding a socket, or actually exposing the application to external input. But, because wsgi_intercept acts at the level of httplib, your application looks and behaves like it does when it is actually bound to a socket.

twill comes with full wsgi_intercept integration, of course -- and you can find hooks and examples for all of the other Python Web testing frameworks at the wsgi_intercept page. So how does it work in practice?

Roughly speaking, you need to do the following:

  • package your application as a WSGI app object;
  • build a function to return that app object;
  • hook that function up to a particular host/port combination;
  • run whatever twill scripts you want to run for the test;

  • teardown (remove the intercept, shut down the server, etc.)

Below, I walk through setting up nose-based unit testing of two simple applications -- a Quixote application, and a CherryPy application. In both cases most of the complexity is in putting the start/stop colls in the correct order.

In Practice: Testing a Quixote application using twill and wsgi_intercept

You'll need Quixote 2.3 for this, as well as nose 0.8.x and twill 0.8.2.

For Quixote, let's test the mini_demo application that comes with Quixote 2.x. Unfortunately, Quixote doesn't (yet?) come with a WSGI interface; however, there's an adapter available here, as wsgi_server.QWIP. So we'll need wsgi_server.py, as well as Quixote 2.x and nose.

First, create a test file; I'll just call it 'test.py'. Let's start by roughing out a nose unit test:

class TestMiniDemo:
   def setUp(self):
      pass

def tearDown(self): pass

Now, the Quixote Publisher object we want to test is returned by quixote.demo.mini_demo.create_publisher(). We need to dynamically create a function that creates the publisher, wraps it as a WSGI object, and then returns that same object each time it's called:

_cached_app = {}
def build_app(_cached_app=_cached_app):
   if not _cached_app:
      publisher = quixote.demo.mini_demo.create_publisher()
      wsgi_app = wsgi_server.QWIP(publisher)
      _cached_app['app'] = wsgi_app
   return _cached_app['app']

If we put this in the setUp function, we have what we need. The only tricky bit is caching the application object. Why do we need to do this? Well, the function passed into wsgi_intercept is called once for each intercepted connection, but we only want to create the WSGI app object once. By storing the app object in a dictionary that persists for the lifetime of the dynamically defined function, we essentially memoize the WSGI application object. (I'm not thrilled with this particular approach: let me know if you have a better way of doing this.)

OK, once we have this function, we need to install it to handle requests to localhost:8080, and then we're ready. Our final setUp function looks like this:

def setUp(self):
    _cached_app = {}

### ### dynamically created function to build & return a WSGI app ### for a Quixote Web app. ###

def build_app(_cached_app=_cached_app): if not _cached_app: # create a publisher obj publisher = quixote.demo.mini_demo.create_publisher()

# wrap wsgi_app = QWIP(publisher)

# save _cached_app['app'] = wsgi_app

return _cached_app['app']

# install the app at localhost:8080 for wsgi_intercept twill.add_wsgi_intercept('localhost', 8080, build_app)

# while we're at it, stop twill from running off at the mouth... self.outp = StringIO() twill.set_output(self.outp)

The tearDown function is much simpler: we just need to remove the intercept, and then clear the Quixote publisher object.

def tearDown(self):
    # remove intercept
    twill.remove_wsgi_intercept('localhost', 8080)

# clear out the publisher quixote.publish._publisher = None

...and now we're ready for a test or two! I'll define two: one to test the main page, and the other to test the link.

def test_welcome(self):
   script = "find 'Welcome to the Quixote demo'\n"
   twill.execute_string(script, initial_url='http://localhost:8080/')

def test_hello(self): script = """\ follow link find 'Hello world!' """ twill.execute_string(script, initial_url='http://localhost:8080/')

Briefly, these scripts both go to 'localhost:8080'; the first script makes sure that it can find specific text on the front page, while the second script tests the result of following the front page link to a 'Hello world' page. (Longer scripts can go in their own file, and execute_file can be used to run them.)

Putting it all together with the correct import statements -- you can download the final file here -- and running nosetests, you get:

% nosetests
..
----------------------------------------------------------------------
Ran 2 tests in 0.381s

OK

So everything works! Huzzah! (If you want to reassure yourself that it's actually running the tests through the Web application break a test by changing the 'find' statements to something else; see, they really are being run. ;)

In Practice: Testing a CherryPy (2.1) application using twill and wsgi_intercept

You'll need CherryPy 2.1.1 for this, along with nose 0.8.x and twill 0.8.2 (the very latest).

For CherryPy, let's test the "Hello, world!" application that is included in the tutorial code. The magic incantations to get a WSGI app object out of CherryPy are not so tricky:

import cherrypy
from cherrypy.tutorial.tut01_helloworld import HelloWorld

# set up the root object cherrypy.root = HelloWorld()

# initialize cherrypy.server.start(initOnly=True, serverClass=None)

# get WSGI app. from cherrypy._cpwsgi import wsgiApp

where 'wsgiApp' is the final application object we wanted.

Starting with a 'test.py' containing a simple framework for a nose unit test,

class TestHelloWorld:
   def setUp(self):
      pass

def tearDown(self): pass

we can fill in the setUp function as before:

def setUp(self):
    _cached_app = {}

### ### dynamically created function to build & return a WSGI app ### for a CherryPy Web app. ###

def get_wsgi_app(_cached_app=_cached_app): if not _cached_app: # configure cherrypy to be quiet ;) cherrypy.config.update({ "server.logToScreen" : False })

# create root & set up the server. cherrypy.root = HelloWorld() cherrypy.server.start(initOnly=True, serverClass=None)

# get WSGI app. from cherrypy._cpwsgi import wsgiApp _cached_app['app'] = wsgiApp return _cached_app['app']

# install the app at localhost:8080 for wsgi_intercept twill.add_wsgi_intercept('localhost', 8080, get_wsgi_app)

# while we're at it, snarf twill's output. self.outp = StringIO() twill.set_output(self.outp)

and the tearDown function is virtually identical to the Quixote example:

def tearDown(self):
   # remove intercept.
   twill.remove_wsgi_intercept('localhost', 8080)

# shut down the cherrypy server. cherrypy.server.stop()

This application is a bit simpler than the Quixote mini demo, so let's just build one test function:

def test_hello(self):
    script = "find 'Hello world!'"
    twill.execute_string(string, initial_url='http://localhost:8080/')

and when we run it, voila! it all works:

% nosetests
.
----------------------------------------------------------------------
Ran 1 test in 0.289s

OK

Conclusions and Caveats

This code is all still quite young, but it works pretty well for me. It's important to emphasize that you do need to run some tests on a live site -- twill can be used for sites without much JavaScript, while Selenium is probably the way to go for anything more complicated. Still, using twill and wsgi_intercept to run tests in-process is relatively simple and straightforward, and I think it can be a very useful component of your Web app development process.

There are several strong conveniences to testing Web applications in this way. The biggest, for me, is the avoidance of all the complicated setup stuff. A close second is that code coverage analysis, profiling, and even debugging can all run within your unit tests, because everything is in-process. And a third is that unit tests run this way seem to run quite a bit faster, perhaps because there's no setup/teardown of the Web server.

One major caveat: it's unclear how well the wsgi_intercept stuff will work with threaded Web app servers. twill doesn't use threads, so this isn't an issue in this case, but if you use a threaded HTTP driver with wsgi_intercept, you could run into problems.

If you have any suggestions, corrections, or explications, please send them on to me at titus@caltech.edu. I'll acknowledge them appropriately, I promise! I would also be interested in examples for other Python Web frameworks; right now I only use CherryPy and Quixote myself.

--titus

Software Links

twill Web browsing language: Web site & docs; download twill-0.8.2.tar.gz directly.

Quixote Web application framework: Web site; download Quixote-2.4.tar.gz directly.

CherryPy Web application framework: Web site; download CherryPy 2.1.1 from SourceForge.

nose unit testing framework: Web site & docs; download nose-0.8.6.tar.gz directly.

Mike Orr's WSGI wrapper for Quixote: view wsgi_server.py here. (The file itself is included in the source distribution for this article.)

Source distribution for this article: darcs repository at the usual place, http://darcs.idyll.org/~t/projects/wsgi_intercept-examples/, or download wsgi_intercept-examples-latest.tar.gz.

CTB 2/06


Updated article posted., posted 23 Mar 2006 at 04:55 UTC by titus » (Journeyer)

I've posted an updated version of this article here.

forgive me for digression, posted 27 Mar 2006 at 15:48 UTC by badvogato » (Master)

who's that famed physicist that you used to pick up at the airport, very much afraid of guarding the genius not being smashed by traffic? What do you know about Philip. Anderson? condensed matter theorist?

thank you., posted 17 Apr 2006 at 12:17 UTC by lkcl » (Master)

this is incredibly useful. am just starting to use mod_python, because zope was giving me a headache, and i need to start doing unit tests. now all i have to do is make my site WSGI compliant. argh :)

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

X
Share this page