Older blog entries for titus (starting at number 120)

A Plea for Python

Seth Vidal of fedoraproject.org has a request -- stop asking for PHP apps, and start using Python!

To quote,

With that in mind I'd like to make a request out to the python web programmers in the fedora-verse. We need some people who are willing to work on web applications either in zope or turbogears. Let's see some options and ways to progress. We have a lot of python programmers who can help audit the code and contribute sections as we build up our module-base.

Yo. What he said.

Interactive exploration of Web apps

One (apparently) whiz-bang feature of PBP, and now twill, is the ability to interactively browse your Web apps from the command line. For example,

>> go http://www.advogato.org/
==> at http://www.advogato.org/
current page: http://www.advogato.org/
>> showlinks
Links:

 0. Articles ==> /article/
1. Account ==> /acct/
2. People ==> /person/
3. Projects ==> /proj/
4. Omnifarious ==> /person/Omnifarious/
5. Read more... ==> /article/862.html
6. tarzeau ==> /person/tarzeau/
...
current page: http://www.advogato.org/
>> follow tarzeau
==> at http://www.advogato.org/person/tarzeau/
current page: http://www.advogato.org/person/tarzeau/
>> showlinks
Links:

 0. http://www.linuks.mine.nu/ ==> http://www.linuks.mine.nu/
1. freshmeat.net/~tarzeau/ ==> http://freshmeat.net/~tarzeau/
2. iBackup ==> /proj/iBackup/
3. AmigaSHELL ==> /proj/AmigaSHELL/
...

This inevitably receives the most "oohs" and "ahhs" the times I've shown twill to people sitting next to me. Apparently none of the other Web testing tools out there do this.

Anyway, I mention it because with the in-process WSGI testing patch, you can now browse your WSGI apps interactively without going through the Internet.

That strikes me as pretty nifty, if only in a Python-geek kind of way.

Go about your business. These are not the comments you're looking for.

--titus

twill & in-process testing of WSGI apps

Ian Bicking's Best of the web app test frameworks? sparked an interesting discussion (read down to the comments). Of particular interest to me was Ian's suggestion that twill (or, really, urllib2/httplib) be modified to send requests directly to a WSGI application without going through a TCP connection.

A few hours of hacking later, I've got a simple implementation that works (inside of twill). Briefly:

• I replace the HTTPHandler.http_open method with a call to my own HTTPConnection class, "myhttplib.MyHTTPConnection".

• myhttplib.MyHTTPConnection overrides the 'connect()' function so that for *specific* host/port connections only, a fake socket is created.

• this fake socket, an object of type 'wsgi_fake_socket', intercepts the HTTP traffic and behaves like a WSGI server, calling the app object with the appropriate translated environment & catching the response.

• this response is then passed back up to the HTTPResponse object, and all is copacetic.

This solution is generic to httplib. It should be easy to pop it into anything that uses urllib2 to talk via HTTP... which means that virtually any Python Web testing code can use this kind of thing to talk directly to any Python WSGI application.

OK, so does it work? Yes!

For example, here's a simple script testing my conference submission system over TCP:

% ./twill-sh -u http://issola.caltech.edu/collar/ tst2
>> EXECUTING FILE tst2
==> at http://issola.caltech.edu/collar/
==> at http://issola.caltech.edu/collar/submit/
Note: submit is using submit button: name="view_status", value="view status"
Note: submit is using submit button: name="view", value="view paper"
--
1 of 1 files SUCCEEDED.

Here's the same script running. This time it's pointed at a host/port that's diverted to WSGI:

% ./twill-sh -u http://floating.caltech.edu/collar/ tst2
>> EXECUTING FILE tst2
INTERCEPTING call to floating.caltech.edu:80
==> at http://floating.caltech.edu/collar/
INTERCEPTING call to floating.caltech.edu:80
==> at http://floating.caltech.edu/collar/submit/
Note: submit is using submit button: name="view_status", value="view status"
INTERCEPTING call to floating.caltech.edu:80
INTERCEPTING call to floating.caltech.edu:80
Note: submit is using submit button: name="view", value="view paper"
INTERCEPTING call to floating.caltech.edu:80
INTERCEPTING call to floating.caltech.edu:80
--
1 of 1 files SUCCEEDED.

GET/POST, redirects, cookies, etc. are handled as they should be.

If you want to play with the code, it's available via darcs:

darcs get http://issola.caltech.edu/~t/twill/

You can also just view myhttplib.py, which contains the entire implementation.

Some notes:

• The code is ugly; I know that. There are a number of things I could do to it to make it nicer it, but at the moment I'm going to rest on my laurels ;). Feel free to critique and/or patch.

• The 'make_environ' function is one of the two weakest links: it needs to behave like a "real" Web server with respect to filling out the environment dictionary, and that's tough. Right now I think it's handling cookies right, but not much else.

• The other weak link is in the interplay between the read/write in the wsgi_fake_socket. I'm assuming an awful lot here... in any case, it should be possible to write a 2-way FIFO that properly mimics an open socket. (Then HTTP/1.1 connections would work, too.)

• It should be fairly easy to pop in your own WSGI app, just for grins. You just need to modify myhttplib.wsgi_intercept appropriately, at any time before your first call to grab a URL.

Have fun,
--titus

RELEASE: twill 0.7.4

Following the dictat of my users... a new twill release!

Links: Cheese Shop entry, announcement, ChangeLog, and download.

There are lots of bug fixes -- cold filtered for a smooth browsing experience. The main innovation in this release is the unit testing: now, twill uses nose-based unit testing, and it also provides some simple commands to help users use twill to unit test their own apps.

Sample unit-test code:

    import os
    import testlib
    import twill.unit
    import twilltestserver
    from quixote.server.simple_server import run as quixote_run

     def test():
        # port to run the server on
        PORT=8090

         # create a function to run the server
        def run_server_fn():
            quixote_run(twilltestserver.create_publisher, port=PORT)

         # abspath to the script
        script = os.path.join(testlib.testdir, 'test-unit-support.twill')

         # create test_info object
        test_info = twill.unit.TestInfo(script, run_server_fn, PORT)

         # run tests!
        twill.unit.run_test(test_info)

 

(See unit testing with twill for more info.)

Comments welcome...

--titus

a Perl-er learning Python

Interesting reading.

PyFLTK

"David" posted about PyFLTK -- fantastic! I use FLTK from C++ and I've been waiting for the day when PyFLTK became usable. And it is!

(It's amusing to see how quickly this thread degenerated into a "but why not *my* favorite GUI?" argument.)

SPAM

Inventive. But not very effective.

X V A V P C
a I m A r I
n A b L o A
a G i I z L
x R e U a I
A n M c S

--titus

Web testing with twill

Moving towards version 0.8; just added a simple unit-testing structure that works via 'nose'. Running tests just got a whole lot easier...

Also spent a not-so-productive half-day working through a number of issues (ChangeLog). Now that I'm starting to use it for more complicated sites, I'm finding more problems. Whee...

Someone mentioned an interest in making it easier to integrate twill with unit testing frameworks. Now that I'm using 'nose', I am starting to get a better idea of what that might entail: fork() plus a kill() at the end is what I'm using. Hmm.

More O'Reilly publicity, this time via Michele Simionato's article on Web testing.

Oddly enough, it appears twill is written in Perl. Heh. Well, it's a 'P' language, at any rate...

I'm still working on what "peste e corna" means -- "pestilence and corn"? -- but people should be aware that my intense hatred of Zope stems from experiences with Zope 1. Or maybe Zope 2. Not sure. At any rate, I haven't looked at it for a loooong time. It's probably great now...

--titus

Training log software

My wife has been looking for training log software for Mac OS X. She's interested in finding something to keep track of when and how far she biked, run, and swam. So far she hasn't found anything that runs on her local computer -- mostly, there's a bunch of Web-based things, which she doesn't want to use.

She's not found anything obvious with google, but my suspicion is that there are some packages lurking out there under different keywords.

Ideas welcome -- I'll summarize here, if anyone has any ideas to send me.

Thanks!

Why should I care what color the bikeshed is?

Good question!

Geek joke

Question: Why should women marry open-source hackers?

Answer: Because open-source hackers are already used to dealing with badly-documented software.

(It was funny at the time...)

Bioinformatics lectures

Just gave a lecture on gene finding & will give another lecture on regulatory regions on Friday. E-mail me if you want the outlines or powerpoints.

--titus

Post-modern programming

I confess, I am a post-modern programmer. But Python makes it so easy...

twill publicity

Grig pointed out this short article at oreillynet.com: Web App Security Testing Using twill.

(who knew that writing documentation was such a good idea? ;)

cucumber2

As part of another project, I finally committed to a rewrite of cucumber (as I'd hoped to back in January). Announcing... cucumber2!

cucumber was a getattr-based object-relational mapping system for Python/PostgreSQL that used PostgreSQL's table inheritance to map objects into SQL tables without impedance mismatch.

cucumber2 is a more robust rewrite of cucumber. cucumber2 is based on new-style classes, metaclasses and properties, which means that things like inheritance (including multiple inheritance) work easily and directly.

cucumber has long been my pet project: an immensely useful little tool that I never got around to properly releasing. cucumber2 is much prettier in many ways, and I look forward to using it in several projects now that the package is cleaner.

At any rate, I'm making a very young cucumber2 public. For anyone who is interested, there's an overview available that's probably the place to start.

A few random musings:

• First and foremost, metaclasses are wicked cool.

• Guido's 2.2 changes are very, very nice and put lots of power in the hands of the developer.

• Metaclasses are a very convenient way to do code generation. (I learned this trick from Daniel Arbuckle at a socal-piggies presentation.)

• nose Just Works.

• Strive for simplicity. It's in there somewhere, and when you find it, it's worth it.

--titus

Tailor

Need to transfer between multiple version control systems? Try Lele Gaifax's tailor. With tailor, you can do things like convert CVS repositories into Subversion or darcs repositories. It's particularly useful for taking source from a non-distributed ("archaic" ;) source control system that you don't control and putting it under darcs. Imagine: your own version of Python, with your own patches kept under darcs control, constantly kept updated with the very latest Python patches. You could swap patches with other people, have different Python distros for different tasks, and lots of other things ... what a world!

simplicidade has good things to say about tailor, too.

epydoc and reStructured Text

While setting up my biolib project, I spent some time trying out code documentation generators. I discovered that epydoc groks reStructuredText, and was sold on it, especially once I figured out that I could link directly to the non-frames version. (The frames version clutters up the screen somethin' fearful...)

Today I added epydoc-generated documentation to session2, our replacement session management system for Quixote 2.x. I'm very happy with epydoc.

I might have used effbot's PythonDoc, which produces attractive and simple-looking Web pages, but it requires more specialized markup. Bleah. O well.

And pydoc and HappyDoc just don't measure up in terms of prettiness, sad to say; I used to use HappyDoc for some projects, but epydoc is muuuch nicer looking.

Quixote fixes

The current widgets.txt hasn't been updated since v1.x, so I did some work to fix the obvious errors. It's a pretty long, nasty document; it might be worth automating aspects of it, or just producing epydoc documentation for it...

Also found & fixed (I think) a bug in PTL's cimport.c that would allow imports of non-existent stuff to work, i.e.

from existing.somewhere import bogus

would set 'bogus' to None rather than raising an Import error.

Release: session2 v0.6, flexible persistent session management for Quixote 2.x

w/Mike Orr

session2 is our rewrite of Quixote's session management classes. It contains a new session manager and session class. It also contains persistent session stores for the MySQL, Durus, PostgreSQL databases, as well as session stores built around files in a directory and shelve databases.

Two major new things with this release, v0.6:

• Extended basic docs; in particular, added information on how to decouple your session store from your database.

• epydoc-based source documentation

You can download it directly, visit the Cheese Shop entry, or browse the source.

--titus

Astropix-of-the-day score

The neutron-star binary APOD references work done by me da.

And, on today's APOD, you can clearly see earthshine, which I worked on for a few years.

(Sadly, my sea urchin-based grad work has less likelihood of being on APOD...)

PRE-Announcement: TurboZCherryPloRails

Mike Watkins, via quixote-users. (I'd have picked a better name if I'd known someone was going to run with it ;)

Seriously, Mike's original post is the single most sober and realistic thing said about the recent proliferation of Web frameworks in Python. Ultimately, you've still got to write the application -- and for me, at least, that's always been the hard part.

--titus

p.s. I'm getting caught being snarky too much, I think!

socal-piggies, testing & twill

At our eighth meeting, I asked Grig for some advice on testing twill. (He recently reviewed twill on his blog, so I knew he was familiar with the issues.)

You see, I'd like to write unit tests for twill, but it's hard to think of exactly how to do it. twill is a testing tool, and because it interacts with real live Web sites it's hard to imagine how to thoroughly test it. I do have a simple Quixote application for testing twill's form handling capabilities, and with the most recent release of twill I am using code coverage analysis to ensure that I was testing all of the commands. Beyond that, I've had a tough time figuring out what to do.

Grig's advice was quite simple: build unit tests for those things that are unique to twill, and only those things. Since twill is built primarily on top of mechanize, ClientForm, and ClientCookie, he suggested that I avoid testing anything to actually do with HTTP and HTML handling because that's all contained in those packages. What I should test is logic surrounding those packages -- basically any logic unique to twill.

Duh. Yeah, it was pretty obvious once Grig said it ;).

biolib

I and pretty much everyone I talk to are unhappy with BioPython. I have my usual persnickety concerns to do with general coding style, oddball documentation, funky library APIs, and so on, but oddly enough many local people seem to agree with me this time.

I have several specific issues with BioPython. First, it's stuck back in the 1.6 / 2.0 dialect of Python, so the modules that I use most -- the NCBI interface -- don't make use of things like iterators. Very annoying. Next, it's also got some funky import stuff that doesn't interact well with Quixote. Finally, it inappropriately uses shorthand in the BLAST parser, so that you end up with BLAST output getting parsed into variables named 'sbjct' for some reason. Oh, and the EUtils interface is oddly written... and I think replacing the parsers they have in there with something based on pyparsing is maybe a good idea .

Mostly minor nits, I know, but the "bad code smell" is enough to make me think about starting a new project. Since the mailing list seems to have become a spam collector, and I don't see discussion anywhere else, I have been encouraged...

Anyway, we'll see. I'd like to combine some of the pygr coolness with some simple code from slippy and also motility. A couple local people are interested in joining in, so that's nice... The cool thing is that this would be a project actually directly connected to my research work, for once ;).

--titus