Older blog entries for titus (starting at number 157)

While I did not know Bill Kent, his books sound very interesting.

A brief update

I'm in in the final throes of preparation for our tutorial at PyCon, next week. 42 attendees -- woot!

I spent the last few days hacking on our unit tests, fixing path-related issues and working on code coverage and profiling. I've also been adding a bunch of twill tests. We're up to 94% overall code coverage, and we're at a good resting point -- anything higher would involve testing very obscure corner cases that are better tested in other ways.

Miscellaneous observations:

• coverage.py is fantastic, but it's difficult to control from within a customised testing infrastructure. I spent an hour or two today ripping out the implicit loading and saving of the coverage cache, so that I can precisely control when coverage is done and don't have to worry about stale coverage results.

• I didn't realize until I started digging into it that nose, our unit test framework of choice, is based on unittest.py. It's pretty easy to get lost in all that code when figuring out whether or not you want to customize the unittest-based behavior or the nose-specific behavior.

• Andy Wingo's statprof is pretty cool, and it's very easy to use. I spent a bit of time hacking on it, and in the process fixed the two lines that rendered it incompatible with Python 2.3. I've also dealt with some output-related issues. Andy's OKed my redistribution of the module, so I'll slap the fixes/patches up on a darcs repository soon. (Watch This Space.)

I'm looking forward to releasing all of our code into the wild. We've done a lot of work; even if the application itself isn't where I want it to be, much of our testing framework is solid.

--titus

Via one of the Planets I read, I saw these pictures of China.

In response to an e-mail about it, one of my friends sent back these pictures of Mexico City.

Both sets of pictures are simply stunning.

profiling in Python: statprof, lsprof, ...

I've been looking into profilers for our tutorial application; this recent thread on python-dev is a good place to start. The situation is confused, to say the least. It's not helped by the near-complete lack of documentation for the various profilers!

I tried out statprof, Andy Wingo's statistical profiler for Python. It gave some interesting results suggesting that the majority of our application's speed problems lie in storage and retrieval. (Hmm.) Right now I'm trying to figure out the best way to integrate profiling into our testing protocols: should it be part of the unit tests? Or part of the extended "continuous integration" tests that are now part of our buildbot setup? And how do we turn it on and off easily for various tests?

I'm probably going to try out lsprof next. I'll try to document both lsprof and hotprof as I go.

Incidentally, it does indeed appear that using wsgi_intercept to run your Web site in-process lets you profile the Web app, as expected. This is certainly a problem you'd encounter with CherryPy, because CherryPy uses threads and you can't use statprof on code within threads.

Speaking of CherryPy and threads...

A fantastic cautionary tale about Web app testing harnesses and threads.

--titus

Well, it appears that people are using twill -- I got a few testy comments along the lines of, "why does twill 0.8.2. not work AT ALL"? So I fixed that (I ended up just including BeautifulSoup with twill) and released twill 0.8.3.

SvnReporter

Back When, I mentioned SvnReporter, a reporting program for subversion repositories. I just installed it and I'm happy to report that the mailing component works. I haven't tried the RSS feed component out -- I don't really have a need for it, although if Grig asks me to I'll add it -- but it's nice to receive informative e-mails on each commit.

My only comment is that it seems a bit over-engineered for my needs, but my needs are fairly simple and I didn't devote a lot of time to grokking the documentation, either. I just hacked and slashed the example file until it stopped complaining and e-mails started coming ;).

Installing the thing was a real bitch, though, through no real fault of Remy Blank's (the author). For reasons of decorators (and maybe more), SvnReporter requires Python 2.4. That's well and good, but it also requires subversion bindings. Unfortunately (for this purpose) I'm using Debian, and python2.4-subversion doesn't exist except in Ubuntu. I tried to compile the subversion Python bindings (...which requires the entire subversion source tree), but that failed for no good reason, twice. After the first monkeypatch for the APR_PATH_MAX problem succeeded, I hit a link problem. I quickly lost interest in going this route.

In the end I just sym-linked the Python 2.3 bindings into the Python 2.4 site-packages directory. Ugly and fragile, but it works.

One lesson that I'd like to repeat: if you want people to use your Python software, write it for Python 2.3. This would have been a 5 second install for me, rather than a 2 hr trek that I nearly abandoned.

setuptools

I'd like to echo grig on using setuptools, with perhaps a tad more specifics:

MAKE YOUR PACKAGE INSTALLABLE WITH EASY_INSTALL.

This amounts to:

• writing a distutils-compatible setup.py.
• publishing your package on PyPi.

Note how neither of these requires doing anything setuptools-specific? Yep.

If you feel like using setuptools.require and some of the other setuptools "magic", then you might be limiting your audience. But publishing a Python package via PyPi with a setup.py script is sufficient to make it easy_install-able.

Now, while we're griping about setuptools: yes, the --help stuff is silly and confusing. It would also be very, very nice to be able to say 'setuptools.require("Python>=2.3"), which currently doesn't work.

--titus

Whoops, screwed up some things; in particular, BeautifulSoup is needed to install 0.8.2. I've finally figured out how to test it properly, but I still need to work through the errors and do the testing. twill 0.8.3 will come soon.

On a related subject, does anyone have any HTML code that cannot be fixed with tidy, cannot be read (in tidy- or non-tidied- form) by HTMLParser, but can be parsed with BeautifulSoup? It's more difficult to test whether or not something is working properly if you can't test it directly.

twill, in-app

I recently added the ability to run twill scripts within the Web application to our tutorial app. The purpose is to run twill scripts for setup/teardown and testing of a Web app from within that Web app; as long as your Web app handles multiple simultaneous requests, it will work. Basically all you do is set up a page that retrieves twill scripts from a particular location and then executes them within that page. The output of the script is then retrieved and displayed on the page.

The big question is, why would you want to do this?

First, it lets you put live testing code in the actual application, e.g. like distributing Selenium tests with your Web app. (This is a pretty cool feature of Selenium.)

Second, programs like Selenium can use scripts written in the (much nicer IMO) twill language for both setup/teardown and some of the more boring non-JavaScript testing areas.

What other uses are there? Are we (Grig & I) on crack here?

--titus

Finally got out twill 0.8.2. In addition to the usual bug fixes and improvements, this version should be able to parse all but the very worst HTML pages. We'll see how that holds up under stress ;).

I've also added a very frequently requested set of features: return values from the simple scripting functions. While these are useless when using the twill language, they are useful to Python programmers who want to call the functions.

UPDATE: Just posted an advogato article on testing CherryPy and Quixote Web apps with twill using wsgi_intercept.

--titus

I've been down & out with chicken pox for the last week or so. Ick. Anyway, here are some random links I picked up:

An Interview with Mark Tilden is hilarious and worth reading, esp. if you read too much bad science fiction. I've met Mark a few times and he's a genuinely brilliant guy: take him seriously. (Well, as seriously as he takes himself. ;)

AskTog's analysis of the Scott Adams forum meltdown -- or, why a "publish" button may mean different things to different people.

Malcolm Gladwell's archive of New Yorker articles. Some brilliant stuff: in particular, Getting In, on the social logic of Ivy League admissions.

The story of the Idaho Integrated Breaching Shotgun: agile gun development?

Lesscode's "Maintainable Programmers". The tension between being able to find someone to hire, and hiring a good programmer.

Is Perl is too powerful? An interesting read, and, in some ways, a testament to the "one right way to do it" philosophy embraced by certain other languages.

And now... some software releases and HOWTOs.

New release of session2, persistent session management for Quixote 2.x

The Quixote Web app framework doesn't come with persistent session stores, so Mike Orr and I wrote our own and packaged it up for other people. This package includes a new session manager and session stores that use MySQL, PostgreSQL, Durus, shelve, and files in a directory to save sessions.

The latest version is 0.6.1; documentation is on the Web page at here; and you can download session2-0.6.1.tar.gz directly.

This version adds a function to clean out DirectorySessionStores (contributed by Graham Fawcett), and a rewrite of the tests into a unit test structure.

New release of wsgi_intercept, an in-process HTTP-to-WSGI tunnel.

wsgi_intercept is a mechanism that allows Web testing frameworks to call WSGI applications without going through the network. The tunnelling functions at a very low level, and it's a good way to test Web apps without actually having them bind a port.

This update fixes a stupid bug in application caching that rendered wsgi_intercept less than useful for unit testing.

The code in this package should be treated with care & ripped off rather than imported or re-used! You can download it at http://darcs.idyll.org/~t/projects/wsgi_intercept-latest.tar.gz, and a darcs repository is available at http://darcs.idyll.org/~t/projects/wsgi_intercept/.

Well, that's all for now.

--titus

The 'configure' thread started by my little rant about libhid has ended, in two ways. First, some great comments and suggestions by various people -- this is one of the reasons why I like advogato so much! Second, the libhid people (person?) are making unofficial debs available.

The only response I have to the thread -- sorry for taking so long, guys -- is that I don't buy the argument that non-portability of configure scripts should prevent their checkin to the source repository. After all, these are the configure scripts they're distributing with their releases; even if they're not completely portable, they're obviously portable enough to be useful to the majority of their users. (...or else they'd have chosen a different build system.)

BTW, david, the e-mail address listed on www.freetype.org doesn't work...

RadioShark stuff

My radioshark adventures continue. Now I've got a simple show definition class, scheduling script, and basic nice interface to the Radio Shark, all for recording radio shows to MP3. It's ugly code -- consider yourself warned ;).

Great Legal Jokes (no. 94)

heh.

Tutorial stuff: wiki page, and screencasting software

If you're attending our tutorial, please go ahead and comment on our wiki page. Right now that page just contains our tentative outline; and, FWIW, it seems like the tutorial is going to have a substantial Web testing component.

As Grig said on the wiki page, we're thinking about screencasting components of the tutorial. Does anyone have any suggestions on what software to use? vnc2swf is an obvious choice; is there anything else cheap or free, and hopefully cross-platform?

Let me know -- thanks!

Interesting twill discussions

It looks like the next twill release will be extraordinarily tolerant of screwy HTML. Check out the thread. Basically, in addition to the tidy stuff, I'm going to add BeautifulSoup parsing (which is already in mechanize -- I've just got to hook it in) and a patch to ClientForm to ignore parse errors.

--titus

1. I hate automake/autoconf/aclocal/autoscrewitallup. CHECK THE RESULTING CONFIGURE SCRIPT INTO YOUR DAMN SOURCE REPOSITORY. That way I don't have to spend 5 hours tracking down EXACTLY which version of these utilities you used so that I can check your source code out and compile it.

   (With reference to libhid.)

2. If you are making a library for Python application developers, DON'T USE FEATURES SPECIFIC TO PYTHON 2.4. Python 2.3 is the version available on Mac OS X 10.4, and that's the largest installed base of Python in existence. More to the point, I don't care about your whizbang application library if by using it I penalize all Mac users.

   (With reference to lots of packages.)

The GUI Testing Tarpit

Brian Marick speaks about his solutions for GUI testing (by which he seems to mean "Web testing"). While I think his approach has some merit, I'm mildly skeptical about it, but I don't really have anything intelligent to say in response. More anon, presumably.

In other news, Jeremy Jones suggested that a Web unit testing framework be included with Python. Here again I'm skeptical, but only because I don't like the general approach.

Even if I did think it was a good idea, which package? twill can't really be considered a unit-testing framework, and it's too sizeable anyway. webtest is nice, as is webunit; I guess I'd go with webtest, because it's really simple.

I do think mechanize should be considered for inclusion in the standard library, however -- or at least ClientForm. They're very useful and (apart from the mechanoid fork) there's nothing else even remotely approaching their level of functionality.

SvnReporter looks cool

http://www.calins.ch/software/SvnReporter.html: to quote,

"""

SvnReporter generates various reports in response to commits happening in a Subversion repository. It is intended to be called from the post-commit hook.

Two types of reports are supported: single-commit and commit list reports. The former generate reports relative to a specific revision only, and are typically used to generate post-commit mails. The latter generate reports relative to a list of commits, e.g. an RSS feed or a web page showing the latest commits.
"""

I'll install it soon and report more.

RadioSHARKage

So, I decided to give my wife a tech solution for our 2nd wedding anniversary. (This would be more dangerous were she not as gadget infested as I...)

The problem: she would like to be able to timeshift and/or download certain radio shows to her iPod. Most of these radio shows (CarTalk and Wait, Wait, Don't Tell Me! are the two big ones) are either for-pay only or aren't actually available at the moment.

My solution: the Griffin radio SHARK.

The Radio Shark first caught my attention when Declan Fleming wrote about using it for radio capture and self-podcasting. Then I noticed that Michael Rolig got it working under Linux. So, I went and bought one, with the ultimate goal of hooking it up to our MythTV box and setting up automatic recordings and exporting them to podcasts.

It worked right out of the box on my iBook, of course: I plugged it in, installed the software, and was immediately listening to live radio.

Linux, as usual, proved a tad trickier. Here's a rough outline (to be gussied up as I play more):

1. Plug in the Radio Shark and verify that it's seen (using lsusb).

2. Spend several hours playing around with libhid, only to figure out that Michael Rolig is using the latest Subversion checkout of libhid and I can't figure out how to compile it. (See my complaint about autof*ck, above).

   Ultimately,

   1. grab the latest release of libhid;
   2. patch it with the Debian stuff;
   3. monkey-patch in the interrupt_write function;
   4. rebuild

   (Shockingly, this worked!)

3. Transliterate shark.c into Python (shark.py).

4. Install ecasound.

5. Insert the audio kernel module (linux 2.4.28) so that the Radio Shark is attached to a sound device (in my case, /dev/dsp2).

6. Use the appropriate ecasound magic to listen:

   ecasound -D -f:s16_le,2ch,48000,inter-leaved -i:/dev/dsp2 -B:nonrt -z:db -b:4096 -o:/dev/dsp
   

7. Use the appropriate ecasound magic to save:

   ecasound -D -f:s16_le,2ch,48000,inter-leaved -i:/dev/dsp2 -B:nonrt -z:db -b:4096 -o:radio-recording.mp3
   

8. Scrape the rss-maker.sh and urlencode.sed stuff off of Declan's page. Run them and put the files in the right place.

9. and... voila, it all works! iTunes recognizes the podcasts and it all Just Works on the client side.

The final file size is approximately 30mb/hr, after postprocessing with

lame -S --lowpass 6000 -q 2 --vbr-new -V 8 in.mp3 out.mp3

Ultimately I'd like to start using an official libhid version, but that will have to wait for a new libhid release. I'm also hoping to use libhid to get our Streamzap remote working, but that's waiting on a different bug...

cheers,
--titus

I'm surprised at how popular our tutorial is. Did we hit on some magic buzzwords or something!?

Job posting

Someone contacted me via socal-piggies (I guess "Python los angeles" will find us?) and asked me if I wanted to apply for a job. I ended up getting a complete job description out of her -- I was doubtful that "proficient in Python" was sufficient, although that's all she sent me at first! -- so here it is.

The ideal candidate will have a strong expertise in search technologies,
distributed systems, Unix/Linux architecture with exceptional skills in
python and shell scripting. Individual will also have some experience in
object-oriented web development and system administration.

 Requirements:

 .       Fluent in Python
.       Fluent in C++ programming language
.       Fluent in shell scripting
.       Fluent in distributed computing and multi-tier application
architectures
.       Fluent in UNIX/LINUX architecture and OS principles
.       Experience with ORACLE
.       Bachelor's degree (minimum)

The web site is somewhat non-descriptive: carreraagency.com. Contact sandra.bunn at the obvious domain name if you're interested.

(I pointed her towards the Python jobs board, too.)

--titus

Courtesy of some prodding from some irish guy.

At the top of your setup.py (you do have one, right?) add:

 ### ez_setup.py and 'use_setuptools()' will automagically download and
### install setuptools.  The downside to this code is that then
### you need to include ez_setup.py in your distribution, too.

 try:
   from ez_setup import use_setuptools
   use_setuptools()
except ImportError:
   pass

 ### this is the critical line
from setuptools import setup  # instead of the 'distutils.core' setup

 ### also import Extension, etc -- anything else you need -- from setuptools.

 

then you can make eggs with the bdist_egg command. Try:

% python2.3 setup.py bdist_egg
% python2.4 setup.py bdist_egg

to build eggs for each version of Python you have installed. The eggs will end up in build/. If you're distributing precompiled binary code, you'll need to make an egg for each platform/Python version.

Anyway, hope that helps!

--titus

p.s. I should probably write a "5 minute guide to distributing Python software properly". Lessee... (1) make setup.py; (2) add it to PyPi; (3) post to c.l.p.a. ;) Oh, and (1a) run cheesecake!