Older blog entries for titus (starting at number 111)

socal-piggies, testing & twill

At our eighth meeting, I asked Grig for some advice on testing twill. (He recently reviewed twill on his blog, so I knew he was familiar with the issues.)

You see, I'd like to write unit tests for twill, but it's hard to think of exactly how to do it. twill is a testing tool, and because it interacts with real live Web sites it's hard to imagine how to thoroughly test it. I do have a simple Quixote application for testing twill's form handling capabilities, and with the most recent release of twill I am using code coverage analysis to ensure that I was testing all of the commands. Beyond that, I've had a tough time figuring out what to do.

Grig's advice was quite simple: build unit tests for those things that are unique to twill, and only those things. Since twill is built primarily on top of mechanize, ClientForm, and ClientCookie, he suggested that I avoid testing anything to actually do with HTTP and HTML handling because that's all contained in those packages. What I should test is logic surrounding those packages -- basically any logic unique to twill.

Duh. Yeah, it was pretty obvious once Grig said it ;).

biolib

I and pretty much everyone I talk to are unhappy with BioPython. I have my usual persnickety concerns to do with general coding style, oddball documentation, funky library APIs, and so on, but oddly enough many local people seem to agree with me this time.

I have several specific issues with BioPython. First, it's stuck back in the 1.6 / 2.0 dialect of Python, so the modules that I use most -- the NCBI interface -- don't make use of things like iterators. Very annoying. Next, it's also got some funky import stuff that doesn't interact well with Quixote. Finally, it inappropriately uses shorthand in the BLAST parser, so that you end up with BLAST output getting parsed into variables named 'sbjct' for some reason. Oh, and the EUtils interface is oddly written... and I think replacing the parsers they have in there with something based on pyparsing is maybe a good idea .

Mostly minor nits, I know, but the "bad code smell" is enough to make me think about starting a new project. Since the mailing list seems to have become a spam collector, and I don't see discussion anywhere else, I have been encouraged...

Anyway, we'll see. I'd like to combine some of the pygr coolness with some simple code from slippy and also motility. A couple local people are interested in joining in, so that's nice... The cool thing is that this would be a project actually directly connected to my research work, for once ;).

--titus

twill v0.7.3

I just released a minor update to twill, twill v0.7.3. This update fixes a few minor problems, adds a few new commands, and also contains a script for stress- and performance-testing, 'twill-fork'.

See the announcement for specifics on the release, or read the docs. You can also just download it...

--titus

A Users Manual for the Universe

Here. Cute ;).

Sourpuss & unfair gripe of the day

It's hard to take the xerces-c project seriously with build instructions like this.

I particularly like the way they make you set an environment variable, although the "runConfigure" script is pretty astounding, too.

(Not that I like the default autoconf/configure much, but it's pretty standard; I have to re-read the instructions for xerces-c everytime I build it, this way.)

WSGI support for Trac

Took the existing patch for adding WSGI support to Trac and added a simple SCGI server and a dumb CGI server to wrap the WSGI app.

Here. Works for me, so far...

trac, incidentally, is pretty cool. Small, neat, to the point. And pretty.

DoS attack

Finally got the chance to investigate some intermittent Web server downtime. Turns out that it was neither name service nor Apache config problems, but instead a DoS SYN flood. This guide was useful; I got the site back up and running in no time flat. I'm still getting lots of connections, though. If it escalates I'll start using iptables to block.

I'm not sure which site they're after. Perhaps wrightflyer.org?

twill additions

Ed Rahn sent me a nice patch adding run and runfiles functionality to twill. These enable the execution of straight Python statements and other twill scripts from within twill. He also submitted something enable $variable substitution, which I'm still a bit iffy on; people tend to expect "this $variable" to be substituted, which doesn't work yet. Adding that could be a bit of a mess.

Ed's patch catalyzed a namespace reorganization and the addition of setglobal and setlocal commands, as well.

All in all, it sounds like a busy weekend -- but it wasn't ;). Lotsa sleep. hooray!

--titus

Why I Like Darcs

Gave a presentation on darcs, the distributed revision control system, at the SoCal Python Interest Group meeting yesterday. The presentation is available here.

twill news

Lambda the Ultimate, the Programming Languages Weblog, had a post about twill. I am not worthy. It was cool to see it there, though!

twill is in a nice, happy place. Most of the work I plan to do on it has to do with increasing its applicability to e.g. thread and load testing. Very little functionality needs to be added; if anything, I need to help out with mechanize. twill is, after all, just a thin wrapper around mechanize...

Speaking of mechanize, what's up with mechanoid? It took me about 5 minutes of browsing to figure out that this page was where I wanted to go, not the PyPI-referenced page. I still haven't figured out why the guy forked mechanize, either.

back to the crib,
--titus

twill 0.7.2

Released twill 0.7.2. May it live long and prosper.

It's mainly a bugfix+docs release, with one or two minor new features.

Trac

Installed Trac for the aforementioned Caltech Software Carpentry class. Pretty cool stuff. A fairly simple, straightforward project management system. It's also pwetty.

Naturally I had to get frustrated with something, so I spent half an hour hacking in SCGI support (it comes with FCGI and CGI support only). I'm happy to send out patches, but I'm not going to submit 'em to the Trac project until I have some more time to make a clean set. (Or somebody asks.) I found a note that Trac is going to add SCGI support with WSGI support sometime around 2.0; right now it's at 0.8.4. Hmm. That might be a worthwhile hour or two of my time...

I also spent an hour or so installing a hacked version of Trac that contains support for a darcs backend instead of subversion. Works really well so far! The tricky bit was noticing that trac actually contains provisions for running directly from its source directory. This is good because it has a number of moving parts which would be difficult to otherwise configure without simply installing it from scratch in some new location.

chillin',
--titus

Miscellany

Spent much of Tuesday trying to get the lirc_streamzap module working. No luck; it segfaults. The situation is better than it was -- LIRC didn't even nominally support the StreamZap until relatively recently -- but it sucks balls that I can't get it working. Ah well, less whining, more hacking!

WinMyth rocks, even if it's still a bit slow & buggy. Good job! (I've been using MythOnMacOsx for a while now; I'm trying to get my wife hooked into our Myth setup with her WinXP box.)

Updated twill, my Web-scripting-for-testing tool, to fix a few problems. The biggest problem was that the extend_with command didn't work to import commands into the shell, although it worked fine for script execution.

And, in other news, working on a Trac install for the Caltech Software Carpentry class. More on that once it's all set up...

All of this is occurring in a general background of sadness for the victims of Katrina. I wasn't nearly this sad about 9/11, for some reason; perhaps because I think the misery was more extended in this case. Not sure. But when I read about 100 people who drowned while waiting for emergency evacuation... I can't help but get pretty upset. sigh. My wife may end up going out there to help the Red Cross with shelter ops, so we're holding off on donations 'til we see if we have to buy a plane trip. I'm feeling moderately useless myself ;(.

peace out,
--titus

Fun with docs

Cartwheel passes data to and from 3rd-party binaries. Generally, that data is in the form of sequence. So, when talking about security, I felt it necessary to add this proviso:

...if someone can encode an attack in valid IUPAC DNA or protein sequence, then it may get through...

I wonder if it's possible to write a buffer overflow for BLAST in DNA sequence? That'd be pretty neat. I'm tempted to offer a bounty, but I don't have any money to offer. Hmm. Perhaps I could offer to sequence part of someone's genome? ;)

Anyway, a first stab at docs is complete. Let's hope they're not too rudimentary to be of use.

twill

OK, I'm going to start working on this soon. There's at least one outstanding bug that Michele Simionato has been waiting for me to fix...

--titus

29 Aug 2005 (updated 29 Aug 2005 at 18:44 UTC) »
Reflections on documentation

Over the last year or two, I've become a convert to the (rather obviously good) practice of documenting my packages. I'm not talking about the source code itself, which is more-or-less documented depending on the when I wrote it and the language I wrote it in -- e.g. Python requires fewer docs, C++ more. I'm talking about documenting APIs, developer setup, compilation tricks, and installation procedures. Writing tests is also a form of documentation, and I've been doing that a lot more, too.

Now, when I say "become a convert" I mean that I'm actually doing it now. Given the time involved, this isn't trivial: even writing sketchy documentation is a time-suck. It's also way less fun than hacking on new features. The flip side is that people are more likely to actually interact with your software if you document it. And, for me, that's one of the main points behind writing open source software...

There are some less obvious benefits, though, that add even more value than use. The main one is that writing installation instructions for other people gives me time to reflect upon just how crufty the install actually is for some of my packages. Cruft is usually easy to fix, once identified -- but I tend to work in my own little cocoon of already-installed programs, and it's hard to identify the cruft without leaving that cocoon. Another benefit is that I get to document some of the stupidities that are on my TODO list, and after thinking about them I often reprioritize my development to fix the worst ones first.

I'm sure there are other benefits as well, but those are the two that I'm really noticing.

I bring this up because I spent a few hours writing some Cartwheel install instructions. A guy down at UT San Antonio, Michael Edwards, has spent a week trying to get everything working. His e-mails inspired me to spend some time documenting, and as a bonus he's also found a few out-and-out bugs in the process. The documentation is still a work in progress, but I'm kind of amazed at just how much software I had to get working in the first place... here's a temporary link, just for posterity's sake.

Anyway, back to documenting...

--titus

Testing X11 apps

cxtest popped up in a /. interview with someone from Crossover Office a while back, and I duly recorded it as something to try. I just checked it out from CVS, built it, and tried it out. Their README instructions worked perfectly, despite their dire prognostications, and I got a recorded script of me testing out xmessage.

Very cool.

I wonder how well it scales? I will have to try it out with my (so far untestable) GUI... Combined with Web testing a la twill and C++ and Python code coverage analysis, I may finally have a soup-to-nuts testing setup available for my application suite.

--titus

I'm back.

I spent most of June and July (6 weeks total) at Embryology Boot camp in Woods Hole, MA; came back to California after stopping off in NY and Minnesota to visit parents & relatives; and then went to Antibes, on the Cote d'Azur / French Riviera for a week with my in-laws. I've been back just over a week, and I spent most of that week dealing with long-neglected tasks and restarting my research work.

Embryology Boot Camp

The Embryology course was really amazing. It was an intensive 6 wk intro to animal development; we covered worms, flies & other arthropods, four vertebrates (chick, mouse, Xenopus, and zebrafish), ascidians, sea urchins, several annelids, ctenophores (comb jellies), cnidarians, clams, other lophotrochozoans, and more clams. The format was 2-4 hours of lecture a day, interspersed with 6-12 hours of lab work, 6 days a week; each day usually stretched from 9:30am 'til 2am. Drinking -- occasionally heavy drinking -- was part of the course (par for the course? ;), as was late night swimming and almost weekly parties.

The instructors were all excellent. Each lecture was typically given by a different professor, and the professors were among the best in their field. We usually had an hour-long Q&A session after each lecture, so we could really get into their research. And in general the lectures were coordinated with the labs, so immediately after hearing about e.g. chick development, we could go into the labs and work with chick eggs.

The lab part of the course was stunning: each group of teaching assistants brought reagents and technique handouts, and were available throughout the lab period to answer questions and help with techniques. I've never had so many expert biologists waiting hand and foot on me ;).

Zeiss brought several million dollars worth of microscopes to the course, too, so I could play with scopes that I simply don't have daily access to even here at Caltech. They did keep on breaking their own scopes by swapping out hardware, but that just means that I'm now somewhat expert at troubleshooting their microscopes!

The best part of the course was the other students. There were 25 students total, and I got to know the other 24 students really well. It was definitely a boot-camp atmosphere, with the attendant intimacy and group dynamics. Good stuff. I expect to be seeing 'em all again over the next few years.

It's hard to really explain how great this course was, but that's enough gushing for now ;). I'll post some pretty pictures from there over the next few weeks.

Antibes and the Cote d'Azur

If you've been there, you understand. If you haven't, go. The Nice/Antibes/Cannes area is fantastic. We stayed at the Three Palms, which I highly recommend.

For-work & for-fun programming

I'm slowly catching up on stuff. There are some twill issues that need to be dealt with. I'm also working on updating the Cartwheel documentation to fix the problems that an installer has been reporting. And sloooowly but surely I'm picking up the pieces of my research from before the summer: PhD or bust!

--titus

102 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!