Advogato blog for titus

E-mail is getting really unreliable

Mon, 12 May 2008 08:05:03 GMT

I've been hit by a few different e-mail-related problems over the last few months, and it's becoming intensely frustrating.

Some servers seem to randomly drop messages from me, for no obvious reason; at least, people don't get one message and then do get another, a day later. gmail may be at the center of this but it's not clear. (Changing my public From: address to ctb@msu.edu may have contributed to the lossage rate, but cannot be the only cause.)

Yahoo at first decided to mark messages as spam, then drop messages, and now likes me again.

As I write, caltech.edu seems to be bitch-slapping a select subset of my messages. Either they're grey-listing and waiting to forward some messages on, or they've dropped some messages entirely.

My daily e-mail load has grown to the point where I appear to be hand-deleting real messages from my inbox when my eye gets them confused with spam or automated notices. I know of at least two messages that I've deleted in the last month -- I found them in my received-not-spam folder (where I save all incoming messages) but have no recollection of having read them. Before this month, I can't recall having accidentally deleted two e-mails in the last 5 years. And it's not like my spam filter has a high false negative rate: I barely get any spam at all in my inbox.

Now I'm trying to start sending messages to people in my lab, and some of them are not responding or acknowledging the messages. Maybe they're not getting them. Maybe they're out to lunch. Maybe they don't like me, or authority, or something. I don't want to hassle people who don't want to respond, but I also want to make sure they got the $!%#$#$ message! Ah well, I will be physically there soon...

All of this goes to say that for a variety of reasons -- increasing amounts of e-mail, increasing amount of SPAM e-mail, increasingly random and annoying anti-spam measures implemented by the big inbox providers -- e-mail has become unreliable enough that I have to think about it. Even worse, the number and variety of anti-spam measures in play mean that neither I nor the receiver may have any ability to affect the spam filter that is dropping e-mail (translation: I don't know what to do to make things more reliable).

Grr.

I think the time is coming where a reputable SMTP forwarder could make some $$; I'd be willing to pay a $ or $$ for a bonded SMTP provider! Anything like that out there that actually works?

--titus

p.s. In a largely unrelated side note, the number of blog comment spammers attempting to post to my blog continues to hit record highs on a daily basis. I've never approved a spam comment -- yet they continue to try. It's mind-bogglingly stupid and it just goes to show that stupid behavior will continue indefinitely if it's approximately zero-cost to the commit-ee. Grr x 2.

So, err, drop me a line if you wrote a witty comment that didn't get posted, and if I don't accidentally delete your e-mail I will approve your comment.

pygr gets some summer love

Fri, 9 May 2008 18:03:44 GMT

(pygr is a neat bioinformatics framework in Python.)

After some commenters on my last post seemed happy to hear that pygr was the focus of some summer work, I realized I had only discussed the pygr summer work in a post to the biology-in-python list.

Whoops.

So, here's the scoop: not only is pygr the focus of Rachel McCreary's Google Summer of Code project, but Jenny Qian will be using pygr to build an ENSEMBL interface, also as part of the Google Summer of Code.

That's not all!

In addition to Rachel and Jenny (under the sterling mentorship of Chris Lee, Robert Kirkpatrick, Namshin Kim, and myself) I have two MSU students working with me over the summer, Alex Nolley and Marie Buckner. They'll both be working with pygr-related things, although like Jenny their efforts may end up being more on ways to use pygr than on pygr's code itself.

I also have a grad student or two that may drop in on pygr, if only to use it for something research-y.

So all in all, pygr will get a lot of love this summer. Hopefully we can polish the code and documentation and tutorials to the point where the learning curve is as minimal as it can get, and this fabulous package will become readily available to many others...

Why am I personally putting so much effort into pygr? Well, I've been using it more and more over the last few months, and (somewhat like scipy) it's transformed my work by turning annoyingly difficult data organization problems into trivial Python transformations. I can literally throw together a custom genome browser in a matter of hours -- I've implemented two or three already, for different projects -- and it has enabled several new research program. pygr seems to be one of those rare packages (kind of like Python itself) that is not only functional and effective but presents a unified and coherent intellectual interface. pygr is the only good middleware layer I've seen for sequence intertwingling in bioinformatics. It's not that mature yet, but it has serious promise, and I'm hoping to get in on the ground floor, so to speak :).

cheers,

--titus

Dear Lazyweb: JavaScript "imagemaps" and/or image subselection?

Wed, 7 May 2008 22:05:07 GMT

Dear Lazyweb, help!

I'm embarking on a number of summer projects in my new lab at MSU, and several of them focus on using pygr to do cool genomic stuff. In particular, I'm planning to build a personal genome annotation system that will let people run their own full genome Web sites and annotate the genomes with private information such as Solexa data, cDNA/EST projects, ChIP-seq, cis-regulatory reporter constructs, ncRNA predictions, etc. etc. (If you're interested in this sort of thing, get in touch -- it will, of course, be open source and open development, albeit in Python :)

As I've been thinking more about how to do the display side of things, I've been running headfirst into a serious lack of knowledge. I would like to make an interface that looks somewhat like your standard genome browser/GMOD/UCSC interface, such as this UCSC view of the chicken genome. I already have the basics of that view working; for example, see this simple example and a group-feature example. But I'd like to add more - a LOT more -- interactivity.

Ideally I'd like to be able to draw simple objects (squares, rectangles, lines) on some sort of canvas and then use JavaScript and AJAX to pop up windows and display bits of information. But I don't really know this space of functionality very well.

So I'm turning to the lazyweb.

Are JavaScript+image maps the right way to go (for example, this, this, and this)? Do they work well with multiple browsers? Or are there good JS libraries for drawing images on the fly in the browser? Is SVG a good thing to look at? Were you stuck with this task, what would you use?

The most important things for this project are, in order of importance:

basic functionality (JS image maps seem fine for this)

cross-browser functionality

selection (e.g. GMOD RubberBandSelection)

flexibility: reordering and redrawing of images.

Your thoughts are much appreciated! Please drop me a line or comment, whichever is most convenient. I'll summarize the options.

thanks,

--titus

p.s. I'm perfectly fine with "Google this, dumby!" I just don't have much in the way of google keyword knowledge in this area...

Eating your own dogfood (but only eating half the bowl)

Sun, 20 Apr 2008 20:07:25 GMT

So I'm pretty bullish on testing for maintenance reasons. It was nice to see how well it worked out for me when a user recently reported a problem with Cartwheel.

This is what happened: third-party package (LAGAN) that the user was running through the Web interface depended on certain command-line behavior from 'sort'. Now, I wasn't aware the the command-line arguments to sort were still evolving, but apparently they are -- my latest Debian upgrade removed some options (the '+1' behavior) in favor of '-k 1'. In any case, I did this big upgrade of many packages, and didn't realize that this third-party program was now broken. (More on that later.)

The user reported weird results, so I went and verified that he'd set everything up properly and that this was in fact a real problem. Then I ran the Cartwheel automated test suite. Voila! Problem was instantly pinpointed in a reproducible manner.

I fixed the program (editing Perl, ick), re-ran the tests, and then re-ran the user's analyses. Tada, done.

OK, so, great, the tests pinpointed the error for me after the user had found it.

Why did I have to wait for a user to report it?

Because I wasn't running the tests under continuous integration on my compute server.

Why not?

Can't think of why.

What would you have done differently?

I would have made sure all my tests were passing on my compute server after I upgraded the thing, i.e. not been a schmuck.

What have we learned?

Tests are only useful if (first) you write them -- that's half the battle -- and (second) you run them. Oops.

More generally, it was fun to note that by putting a fairly high-level functional test on the batch-processing backend, I discovered a bug several levels down in my software stack -- a problem lying between a third-party package and a system utility. Unit tests wouldn't have found this bug, unless the third-party package had them (don't think so) and I was running the third-party package unit tests (good grief...)

OK, back to work.

--titus

John Ringo is a caricature of a wingnut

Sat, 19 Apr 2008 10:09:32 GMT

I read a lot of total crap, and one of my recurring crap authors has been John Ringo. He's a total nutjob politically, but he writes good battle scenes and is an enjoyable read once you cut through the nonsense. Still, I'm having a tough time getting through the opening chapters of The Last Centurion. In this book, Ringo constructs a near-future world where Hillary Clinton is president, global cooling is the problem, and the chemicals from processed food and big farming are life saving.

Let's take those one at a time.

One of Ringo's favorite tropes is that the left, and the Clintons especially, are what's wrong with America. It's hard to convey the dripping scorn with which he discusses these topics, but it involves a lot of naughty words. In this book, Hillary Clinton (or a straw woman facsimile thereof) is president through the Big Chill and the simultaneous deadly bird flu outbreak, and she makes every mistake possible. While Hillary Clinton is not my favorite politician, it's worth noting that our current president (who can do little wrong in Ringo's eyes) has actually made almost every mistake possible, and this makes Ringo's text unbearably difficult to read. If Ringo is hoping to even tell a good story, much less sway anyone's opinion, he'd be better off with less in the way of textual histrionics.

Another one of Ringo's tropes is that the global warming hypothesis is nonsense. Not only does he mention this frequently, but he literally pauses in the middle of his books to deliver four page diatribes on the subject. In this latest book, Ringo makes the next big climate change event a major solar COOLING, which has predictable effects on the food supply. Now, I'm a scientist and a lefty, and I've even worked on science relevant to climate change, so presumably (by Ringo's criteria) I am unfit to comment, being moderately knowledgeable. But when your social commentary depends entirely on fiction, it loses any relevance and becomes a distraction.

The most interesting novelty in this book (which presumably will become another abortive series, to join the ranks of his other five unfinished series?) is the device where American lives are saved by having eaten so many processed foods. As far as I can tell, the idea is that eating processed foods conveys resistance to chicken flu, and this leads to a dramatically greater survival rate in America. I'm not sure why this device is in the book, unless it's another imaginary nail in Ringo's imaginary coffin of liberalism. Whyever it's there, it's entertainingly stupid -- there's plenty of evidence that weird, random chemicals do weird, random things to your DNA, and that's one reason why cancer is so prevalent. There's no reason at all to believe that these chemicals would somehow "cancel out" bird flu. But what do I know? I'm just a molecular freakin' biologist...

Combine all that with Ringo's inimitable writing style in which no breasts are too big, no hero goes unfucked by multiple (large-breasted) women, and no terrorist goes unpunished, and these books are truly a piece of work. I do not, however, mean "of art". In fact, this last book is so outlandish that I'm actually becoming a bit suspicious of Ringo's sincerity. It's hard to read such complete and utter crap without thinking that perhaps the author is secretly making fun of the very viewpoints he is espousing. But it's been a consistent trend towards lunacy thus far, so I'm inclined to believe that he's actually somewhat sincere.

Anyway, here's my judgement: Ringo's latest book is masturbatory fodder for hard right wingers, and it's becoming increasingly difficult to enjoy his books if you're not actually lobotomized. Luckily that ensures him an 18% market.

--titus

Threading and subprocess

Sat, 19 Apr 2008 09:18:54 GMT

I'm having a long-running discussion with some people about threading and why using threads with simple subprocess calls is almost certainly an overcomplicated (== BAD) use of threads. Everyone seems to think I'm wrong (at least, there's either deafening silence or straight out argument ;) and I think I finally figured out why.

The task at hand: use subprocess to run some command (say, 'ping') a bunch of times. Because the command is I/O bound, you want to run the commands in parallel. Should you use threads to do this? Is it necessary in order to achieve good performance?

Well, consider these two examples ('common.py' is down at the bottom; it just contains the list of IP addresses to ping, and a function to call subprocess.Popen).

nothread.py:

from common import IP_LIST, do_ping

z = []
for i in range(0, len(IP_LIST)):
   p = do_ping(IP_LIST[i])
   z.append(p)

for p in z:
   p.wait()

thread.py:

import threading
from common import IP_LIST, do_ping

def run_do_ping(addr):
   p = do_ping(addr)
   p.wait()

###

# start all threads
z = []
for i in range(0, len(IP_LIST)):
   t = threading.Thread(target=run_do_ping, args=(IP_LIST[i],))
   t.start()
   z.append(t)

# wait for all threads to finish
for t in z:
   t.join()

Both of these work fine, and in both cases are easily modifiable to retrieve the output, exit status, etc. of the ping command. (In the threaded example you have to keep track of 'p' in 'run_do_ping' to retrieve this kind of info, and I wanted to keep things as simple as possible.)

They also run in about the same amount of time, although the non-threaded one is quicker by a few milliseconds for me. I think this is because thread starts & joins are extra overhead.

The key misunderstanding in the discussion seems to have been that the examples at hand were using subprocess.call, which blocks waiting for the subprocess to exit, i.e. equivalent to using this code in nothread.py:

for i in range(0, len(IP_LIST)):
   p = do_ping(IP_LIST[i])
   p.wait()

Here the pings would execute serially rather than in parallel, with the obvious performance problem :). However, you can bypass this effect of subprocess.call by using subprocess.Popen, which creates a new process that executes in parallel with the calling process.

So, for this simple use of subprocess -- running a shell command and gathering the output -- which is "better"? I think 'nothread.py' is better because it is simpler, shorter, clearer, and less complicated. Of course, as soon as you start doing more complicated stuff like reading the streams of information coming out of the subprocess commands, the threaded version may well have its advantages. But that's not the case here, I think.

Comments welcome.

--titus

common.py:

import subprocess

IP_LIST = [ '131.215.17.3',
            '131.215.17.4',
            '131.215.17.5',
            '131.215.17.16',
            '131.215.17.17',
            '131.215.17.18',
            '131.215.17.19',
            '131.215.17.24',
            '131.215.17.25',
            '131.215.17.31']

cmd_stub = 'ping -c 5 %s'

def do_ping(addr):
    cmd = cmd_stub % (addr,)
    return subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)

Some new terminology?

Tue, 15 Apr 2008 18:03:34 GMT

In some discussions with a moderately new Python programmer who seems to value complexity over simplicity, I may have coined a new term:

"Penis size" style of programming -- the (mistaken) belief that the
more advanced programming language features you use, the more
impressive your code will look.

I think it's a fair generalization to say that experienced programmers value simplicity over complexity, all other things being equal.

A search for "penis size programming" came up with this link, which is entertainingly apropos.

--titus

p.s. I originally used "dick size", but now that I'm a professor, I have to be decorous, right?

Google Highly Open Participaton Contest -- another notch in the source code!

Wed, 9 Apr 2008 08:03:22 GMT

Pavel Vinogradov <fastnix> has been keeping me updated on an issue he discovered while testing TCMalloc with Python as a Google Highly Open Participation (GHOP) task, task 105.

Briefly, Pavel discovered a situation in which replacing the Python memory allocator with TCMalloc resulted in really bad performance. The latest is that there appears to be a bug or gotcha in TCMalloc with glibc, where TCMalloc does a poor job in cases where mremap can be used by glibc. The TCMalloc folk are going to look into it more, I gather. (See google-perftools thread here.)

Anyway, this was a situation where we just threw the task at the students to see if anything interesting would pop out -- not expecting much of anything other than a learning experience for the student -- and yet through some simple-yet-dogged testing, Pavel really contributed something.

Awesome stuff!

There have been several real success stories to GHOP. I need to write them down, sigh... my kingdom for some time :)

--titus

Google's AppEngine OneThousand

Tue, 8 Apr 2008 08:03:17 GMT

At Google Campfire One, v 2.0 -- introducing AppEngine.

IT'S FREEZING. The cider ran out. Brr.

Deploying Web apps is annoyingly difficult. Technical hurdles, etc. Need machines. Blech. Costly.

AppEngine solves all these problems. Runs web apps, handles app lifecycles, apps are run on Google infrastructure can make use of GFS, auth etc. etc. etc.

Components

Scalable serving infrastructure
Python runtime
Soft Dev Kit
Web-based admin console
Data

Config is in YAML, with mapping done by regexp.

from google.appengine.ext import webapp, looks like Java to me.

Naah, that was mean. It's python.

Oooh, WSGI built in.

def get, def post -- looks like web.py. why do people do that?

Using introspection, building SQL-like GQL, to drop stuff into/suck stuff out of database with proper names.

Django templates.

Single-command deployment.

Scalable serving infrastructure: when app pushed, pushed to multiple fault tolerant servers. Any one may fail, but request will always go through. THE ROADS MUST ROLL.

All Python runtime and many third-party libraries available. (How do they deal with security? What about SQL and ORMs?)

SDK: releasing for Linux etc.

Web-based admin console. Standard stuff, stats collected in "near-real time".

Data store. BigTable. Yah. Horizontally distributed fault-tolerant system.

No joins in GQL?! Rationale: joins may need to work across computers and individual ram capacity.

Send e-mail, make HTTP reqs, auth with Google accounts, use a variety of frameworks.

Went to get cider, they replenished. There are also meatballs wrapped in dough (!?!) It is still FREEZING.

Guido gets up. He's on the AppEngine team.

All about making tools for developers. But hates root password. Thinks AppEngine solves this.

"First time Google lets other people run stuff on their servers."

Isn't this a support nightmare? Software versions etc?? Well, can upload your own frameworks.

Stdlib emasculated in three ways: writing to the file system is forebidden; cannot talk directly to the network (urlfetch & mail sending API); no threads (chuckle, I think GvR not so secretly hates threads).

Python is not the only language (you heard it from Guido). Perl? COBOL? Assembly?

Stuff about admin infrastructure.

Lots of nice error logging/tracking stuff.

Data viewer. Interactive query. Nifty. (Hmm, how do you upload bulks of data??)

Version control built in. Yay.

Is testing built in???

Adding a domain...

Host limits: 5mn page views a month for a well written application is free.

Over and out. I'm freezing. Still.

--titus

2008 Summer Internship at Disney Animation

Sat, 5 Apr 2008 01:03:15 GMT

Please send this on to anyone who might be interested...

Disney Animation has an opening for a summer intern to work on a testing project under the supervision of Paul Hildebrandt and Dr. C. Titus Brown. The ideal candidate will have experience with a dynamic language supporting introspection (Python preferred) as well as experience developing unit, functional, and regression tests, but the ability to learn quickly is the only requirement. The work will consist of building a new testing tool for aiding test automation on large existing code bases. We expect to release the tool under an Open Source license.

Please send your CV and a short personal statement to da-testing-2008@idyll.org, c/o Paul Hildebrandt and C. Titus Brown.

BA/BS requested, MA or PhD in-progress desired.

Housing stipend provided; competitive hourly wage.

2 months commitment minimum, 3 months preferred.

Opportunity for follow-on work or continued employment.

Disney Animation is located in Burbank, CA.

I'll post what I can of the specific project proposal when I have a cleaner version.

--titus

Advogato blog for titus

E-mail is getting *really* unreliable

pygr gets some summer love

Dear Lazyweb: JavaScript "imagemaps" and/or image subselection?

Eating your own dogfood (but only eating half the bowl)

John Ringo is a caricature of a wingnut

Threading and subprocess

Some new terminology?

Google Highly Open Participaton Contest -- another notch in the source code!

Google's AppEngine OneThousand

2008 Summer Internship at Disney Animation

E-mail is getting really unreliable