Older blog entries for titus (starting at number 161)

After a nice blurb or two about my lightning talk at PyCon '06, I thought I'd practice my screencast fu by recording (a version of) my lightning talk. Without further ado, here's a screencast of a rerun of my lightning talk. (Yeah, I know it kinda sucks; I'm still working on the process!)

This took a fair amount of investigatory work. I started out by using SnapZ Pro X, for Mac OS X, which produced hideously large files of generally low quality. After lots of wasted effort, I gave up and switched to pyvnc2swf. I knew (from Chad Whitacre's efforts) that it could work, I just didn't know how well it worked.

Well, the short and sweet of it is that pyvnc2swf is a fantastic piece of software, and with only a minor amount of geek fiddling, I recorded the screencast above. (I actually gave up on using pyvnc2swf directly on my Mac; the screencast was recorded over the network, which is a pretty nifty feature of pyvnc2swf...)

Audio was more of a problem. I ended up recording with Quicktime Pro at the same time as I was doing the screencast, then saved as AIF and re-encoded with LAME before overlaying it onto the SWF file. There were a few mishaps in this process, because QuickTime doesn't save as MP3 (!?!?) and the AIFF file it generated doesn't contain good header info. Amusing nonetheless.

wsgi_intercept & mock_http

While I was at PyCon, an old acquaintance from Reed, Marc Hedlund, contacted me about wsgi_intercept. Apparently he needed to build a mock HTTP server to test client HTTP code, and upon discovering wsgi_intercept thought two things: (a) this is perfect! and (b) hey, I know this guy! Weird, eh? Pity he wasn't at PyCon...

Anyhoo, he very kindly donated a simple mock HTTP example to the project, and I just checked that in. So now you can mock not only the TCP connection but also the entire server side. Heh.

twill and django

In addition to Michael Twomey's overnight hack using wsgi_intercept stuff to test Django, some other people appear to be using twill with django.

Googling about, I found this interesting discussion of unit tests and twill. That led to this file (it's a download link) which was probably not meant to be entirely public, but too late, google finds all...

I looked over the source to understand how they were using twill. A few comments and thoughts, some for them, some for me:

• twill's browser.get_html() doesn't actually return the tidy-ed or BeautifulSoup-ed HTML. It probably should, or it should at least have an option for it.

• twill.set_output(fp) will now endeavor to redirect all output to the given fp, so they don't have to do the (painful!) stdio remapping any more.

• I ended up writing my own test_follow_links for our MailOnnaStick app. Good idea. Should probably be in the std twill extensions lib at this point...

• I was also thinking of adding in hooks to require that page visits be successful, etc. Looks like they could make use of those; good to know.

• It would be cool to have random generators for forms that would let you fill in things like name, address, and zip fields with syntactically correect (or intentionally incorrect... hmmm...) input.

• Use coverage.py to look at the code coverage of your twill tests. That way, you can pretty easily figure out what parts of your Web app are completely missed by your Web tests.

Anyway, no good idea how to find these people, but I'd be interested in hearing their success/failure stories with twill.

--titus

Grig and I got a lot of positive feedback on our tutorial presentation, and I have a few thoughts to offer.

First, I had a cynical thought on why so many people signed up for the testing tutorial. The situation could be analogous to that of Christmas: some Christians go to church only once a year, at Christmas, to save their souls. Likewise, perhaps many people were thinking that they would be relieved of the need to test by coming to our tutorial.

Of course, the more optimistic take on things is that many people realize they need to test but they don't have a good idea of how to do it...

Second, several people (ok, two ;) commented that our tutorial was more technical, more involved, and higher-level than many (any?) of the other tutorials (!?!). I find this hard to believe, but, if true, there may be a demand for "entree-to-nuts", or maybe "soup to salad", not just "soup-to-entree", tutorials. (Evelyn Mitchell alluded to this in conversation with me, too.)

Third, the same people also mentioned that our tutorial was more technical than many of the other talks at the conference. I agree, in the sense that we presented stuff directly useful to & relevant to programmers. I've seen a few other comments that PyCon '06 was lighter on language internals and developer tool talks than before. This may just be another sign that Python is growing out of its gangly developer-only phase, but I think it results in a narrower conference. A lot of Python luminaries showed up and slouched around for a few days without giving talks. It'd be great to e.g. give a few python-dev people 15 minutes to talk about whatever they feel like.

Fourth, demonstrations are universally well-received even when they don't work all that well ;). So if you have the option, don't give a static presentation...

In other news, Robert Brewer has promised me a 100% solution to testing threaded Web apps, no doubt inspired by his own recent travails. So I'll be looking for that -- and now I know what you look like, Mr. Brewer...

--titus

p.s. Yes, I hear there was negative feedback too, and I'm somewhat perversely looking forward to getting the tutorial comments so that we have some explicit statements from people who attended the tutorial.

The ssize_t fix for 64-bit machines will let me read the entire human genome (3 gb) into memory in one string!

The AST checkin for the Python branch looks very interesting. There was some discussion about what kind of Python interface to offer in the AST BoF which was mostly (AFAIK) off-topic, but very interesting. Naively I suggested providing a C-only interface to it in advance of the Python interface, to which I think the response was that duh, the argument really was about what the publicly supported interface should be, because once they had such a C interface the Python interface would probably be obvious.

Thanks, bazcamp

Whichever pythonista brought along their wifi hub and opened up 'bazcamp' near my hotel room -- thanks! I owe you a beer...

Lightning Talk Coolness

Apologies for things I missed; I know that I forgot to write some stuff down.

Ben Collins-Sussman-Google gave a great talk on replacing himself with an IRC bot. (Ironically, google fails to find valid URLs for his stuff...)

py.execnet is an evil way to control remote machines to which you have SSH access. (Holger Krakenmaster is the name I remember for the author ;))

Two guys from Columbia presented Tasty, "a full featured, efficient tagging engine accessible entirely through a REST interface."

The spec module, presented by David Binger, looks pretty cool. Unfortunately google has failed me... links?

Lightning Talk feedback

Some people type too fast.

I'll post more about the lightning talk stuff when I get a chance to do a proper screencast.

Quotable Bram

From Bram Cohen's PyCon '06 keynote on Sunday:

"The first rule of networking is ""don't reimplement TCP, badly"" ." (my paraphrase)

"Python faithfully reproduces all of the Posix API's crapitude."

"I spend very little time haggling with the language." (on Python)

"Advanced continuation usage breaks my brain. So does try/finally, actually."

"rsync is the morning-after pill of file transfer protocols."

"The funny thing about introductory rates is that you can keep them up for a while." (Yes, he's speaking about credit cards...)

"So, you see, credit companies count it against you if you apply for lots of credit. So what you do is you stack up all of the credit offers that come in, and you send them all out on the same day. Then they don't see the other credit applications." (my paraphrase)

"The only platform Avalanche runs on is PowerPoint." (Steve Holden, discussing Bram's opinions on Avalanche.)

"Programmers in San Francisco don't really take vacations. They just wait for their current employer to implode, and then take some time off before finding a new job."

"If you try to bring a bunch of lead-filled balls on a plane in America, you often end up juggling for the security people. 'Yes, these hand grenades are for juggling!'" (my paraphrase)

--titus

I am at...

The George Bush Python Convention '06

Lotsa "George Bush" monikers -- Sr., I think -- in Dallas. Why not the conference, too?

Miserable plane experience. Apparently some f*cker ran through security in Houston, occasioning a whole-airport shutdown & attendant delay. We came in from LAX and had to shuttle over to the Continental Express lounge area, which was packed with travellers in purgatory. To compare that place to the pits of hell would be a compliment to hell: the dull scent of despair mingled with the sweat of impatient people sweltering in underperforming air conditioning, and the relentless turmoil of people was relieved only by occasional murmurs of boarding stewardesses looking for the rest of the flight crew.

Really, it was bad.

We declined the honor and skipped back to the reservation desk, where we took a jet to DFW rather than Dallas-Love, which meant a longer cab ride but got us to the hotel pretty quickly in the end...

The tutorial, it is over!

Our Agile Development and Testing tutorial went well. We've opened up our development Web site and put up our handouts (at the bottom of the site). Comments & feedback welcome!

We'll also be presenting a "highlights of the testing tutorial" on Sunday after lunch, for those who missed the tutorial but want to see Selenium and buildbot in action.

Software that got mentioned by various people before, during, or after our tutorial:

• Mousepose, a tool for demonstrations. (by David Goodger)

• bitten, a continuous integration tool linked into Trac. (by Matt Good)

• depgraph, for drawing dependency graphs. (someone in the audience -- sorry I didn't catch your name!)

The question that we didn't answer properly: why nose and not py.test.

And, finallyy, a note to self: when describing the use of Python wrappers to test C code, subversion is a fine example case.

A fictional but amusing LightningTalk title

"5 Minutes to a New Python Web Framework"

PyCon Quotes

"No matter how complex it is, it's just Python." -- Alan Runyan, Plone Keynote Speaker[0], explaining his sales pitch to organizations that worry about maintaining Plone.

"We have this concept of the J2EE liberation front..." -- Alexander Limi, Plone Keynote Speaker[1] (to great applause).

"PySVN has a bus factor of 1: there's one developer, and if he gets hit by a bus, the project dies." -- Brian W. Fitzpatrick, in his talk on cvs2svn.

--titus

While I did not know Bill Kent, his books sound very interesting.

A brief update

I'm in in the final throes of preparation for our tutorial at PyCon, next week. 42 attendees -- woot!

I spent the last few days hacking on our unit tests, fixing path-related issues and working on code coverage and profiling. I've also been adding a bunch of twill tests. We're up to 94% overall code coverage, and we're at a good resting point -- anything higher would involve testing very obscure corner cases that are better tested in other ways.

Miscellaneous observations:

• coverage.py is fantastic, but it's difficult to control from within a customised testing infrastructure. I spent an hour or two today ripping out the implicit loading and saving of the coverage cache, so that I can precisely control when coverage is done and don't have to worry about stale coverage results.

• I didn't realize until I started digging into it that nose, our unit test framework of choice, is based on unittest.py. It's pretty easy to get lost in all that code when figuring out whether or not you want to customize the unittest-based behavior or the nose-specific behavior.

• Andy Wingo's statprof is pretty cool, and it's very easy to use. I spent a bit of time hacking on it, and in the process fixed the two lines that rendered it incompatible with Python 2.3. I've also dealt with some output-related issues. Andy's OKed my redistribution of the module, so I'll slap the fixes/patches up on a darcs repository soon. (Watch This Space.)

I'm looking forward to releasing all of our code into the wild. We've done a lot of work; even if the application itself isn't where I want it to be, much of our testing framework is solid.

--titus

Via one of the Planets I read, I saw these pictures of China.

In response to an e-mail about it, one of my friends sent back these pictures of Mexico City.

Both sets of pictures are simply stunning.

profiling in Python: statprof, lsprof, ...

I've been looking into profilers for our tutorial application; this recent thread on python-dev is a good place to start. The situation is confused, to say the least. It's not helped by the near-complete lack of documentation for the various profilers!

I tried out statprof, Andy Wingo's statistical profiler for Python. It gave some interesting results suggesting that the majority of our application's speed problems lie in storage and retrieval. (Hmm.) Right now I'm trying to figure out the best way to integrate profiling into our testing protocols: should it be part of the unit tests? Or part of the extended "continuous integration" tests that are now part of our buildbot setup? And how do we turn it on and off easily for various tests?

I'm probably going to try out lsprof next. I'll try to document both lsprof and hotprof as I go.

Incidentally, it does indeed appear that using wsgi_intercept to run your Web site in-process lets you profile the Web app, as expected. This is certainly a problem you'd encounter with CherryPy, because CherryPy uses threads and you can't use statprof on code within threads.

Speaking of CherryPy and threads...

A fantastic cautionary tale about Web app testing harnesses and threads.

--titus

Well, it appears that people are using twill -- I got a few testy comments along the lines of, "why does twill 0.8.2. not work AT ALL"? So I fixed that (I ended up just including BeautifulSoup with twill) and released twill 0.8.3.

SvnReporter

Back When, I mentioned SvnReporter, a reporting program for subversion repositories. I just installed it and I'm happy to report that the mailing component works. I haven't tried the RSS feed component out -- I don't really have a need for it, although if Grig asks me to I'll add it -- but it's nice to receive informative e-mails on each commit.

My only comment is that it seems a bit over-engineered for my needs, but my needs are fairly simple and I didn't devote a lot of time to grokking the documentation, either. I just hacked and slashed the example file until it stopped complaining and e-mails started coming ;).

Installing the thing was a real bitch, though, through no real fault of Remy Blank's (the author). For reasons of decorators (and maybe more), SvnReporter requires Python 2.4. That's well and good, but it also requires subversion bindings. Unfortunately (for this purpose) I'm using Debian, and python2.4-subversion doesn't exist except in Ubuntu. I tried to compile the subversion Python bindings (...which requires the entire subversion source tree), but that failed for no good reason, twice. After the first monkeypatch for the APR_PATH_MAX problem succeeded, I hit a link problem. I quickly lost interest in going this route.

In the end I just sym-linked the Python 2.3 bindings into the Python 2.4 site-packages directory. Ugly and fragile, but it works.

One lesson that I'd like to repeat: if you want people to use your Python software, write it for Python 2.3. This would have been a 5 second install for me, rather than a 2 hr trek that I nearly abandoned.

setuptools

I'd like to echo grig on using setuptools, with perhaps a tad more specifics:

MAKE YOUR PACKAGE INSTALLABLE WITH EASY_INSTALL.

This amounts to:

• writing a distutils-compatible setup.py.
• publishing your package on PyPi.

Note how neither of these requires doing anything setuptools-specific? Yep.

If you feel like using setuptools.require and some of the other setuptools "magic", then you might be limiting your audience. But publishing a Python package via PyPi with a setup.py script is sufficient to make it easy_install-able.

Now, while we're griping about setuptools: yes, the --help stuff is silly and confusing. It would also be very, very nice to be able to say 'setuptools.require("Python>=2.3"), which currently doesn't work.

--titus

Whoops, screwed up some things; in particular, BeautifulSoup is needed to install 0.8.2. I've finally figured out how to test it properly, but I still need to work through the errors and do the testing. twill 0.8.3 will come soon.

On a related subject, does anyone have any HTML code that cannot be fixed with tidy, cannot be read (in tidy- or non-tidied- form) by HTMLParser, but can be parsed with BeautifulSoup? It's more difficult to test whether or not something is working properly if you can't test it directly.

twill, in-app

I recently added the ability to run twill scripts within the Web application to our tutorial app. The purpose is to run twill scripts for setup/teardown and testing of a Web app from within that Web app; as long as your Web app handles multiple simultaneous requests, it will work. Basically all you do is set up a page that retrieves twill scripts from a particular location and then executes them within that page. The output of the script is then retrieved and displayed on the page.

The big question is, why would you want to do this?

First, it lets you put live testing code in the actual application, e.g. like distributing Selenium tests with your Web app. (This is a pretty cool feature of Selenium.)

Second, programs like Selenium can use scripts written in the (much nicer IMO) twill language for both setup/teardown and some of the more boring non-JavaScript testing areas.

What other uses are there? Are we (Grig & I) on crack here?

--titus

Finally got out twill 0.8.2. In addition to the usual bug fixes and improvements, this version should be able to parse all but the very worst HTML pages. We'll see how that holds up under stress ;).

I've also added a very frequently requested set of features: return values from the simple scripting functions. While these are useless when using the twill language, they are useful to Python programmers who want to call the functions.

UPDATE: Just posted an advogato article on testing CherryPy and Quixote Web apps with twill using wsgi_intercept.

--titus

I've been down & out with chicken pox for the last week or so. Ick. Anyway, here are some random links I picked up:

An Interview with Mark Tilden is hilarious and worth reading, esp. if you read too much bad science fiction. I've met Mark a few times and he's a genuinely brilliant guy: take him seriously. (Well, as seriously as he takes himself. ;)

AskTog's analysis of the Scott Adams forum meltdown -- or, why a "publish" button may mean different things to different people.

Malcolm Gladwell's archive of New Yorker articles. Some brilliant stuff: in particular, Getting In, on the social logic of Ivy League admissions.

The story of the Idaho Integrated Breaching Shotgun: agile gun development?

Lesscode's "Maintainable Programmers". The tension between being able to find someone to hire, and hiring a good programmer.

Is Perl is too powerful? An interesting read, and, in some ways, a testament to the "one right way to do it" philosophy embraced by certain other languages.

And now... some software releases and HOWTOs.

New release of session2, persistent session management for Quixote 2.x

The Quixote Web app framework doesn't come with persistent session stores, so Mike Orr and I wrote our own and packaged it up for other people. This package includes a new session manager and session stores that use MySQL, PostgreSQL, Durus, shelve, and files in a directory to save sessions.

The latest version is 0.6.1; documentation is on the Web page at here; and you can download session2-0.6.1.tar.gz directly.

This version adds a function to clean out DirectorySessionStores (contributed by Graham Fawcett), and a rewrite of the tests into a unit test structure.

New release of wsgi_intercept, an in-process HTTP-to-WSGI tunnel.

wsgi_intercept is a mechanism that allows Web testing frameworks to call WSGI applications without going through the network. The tunnelling functions at a very low level, and it's a good way to test Web apps without actually having them bind a port.

This update fixes a stupid bug in application caching that rendered wsgi_intercept less than useful for unit testing.

The code in this package should be treated with care & ripped off rather than imported or re-used! You can download it at http://darcs.idyll.org/~t/projects/wsgi_intercept-latest.tar.gz, and a darcs repository is available at http://darcs.idyll.org/~t/projects/wsgi_intercept/.

Well, that's all for now.

--titus