Advogato blog for oubiwann

Twisting the Planet

Fri, 2 May 2008 01:03:46 GMT

As Steve blogged the other day, we've been jamming on some Twisted lately. But it's not the kind of thing you usually hear from us. We're not doing something esoteric and mind-blowing. We're doing something much harder: working out how to bring Twisted to the masses.

The motivation for this is philanthropic: we believe in Twisted's goodness :-) As Allen Short paraphrased on IRC the other day after listening to MIT entrepreneur Raffi Krikorian "it sounds like he's saying Twisted makes you smarter." Humor aside, Allen is right. Twisted does make you smarter: with increased familiarity and experience, you start thinking outside the box. Twisted helps you become a more creative problem solver.

In particular, we're reviewing the "Teach Me Twisted" open space session we had at PyCon. A bunch of you showed up for it, and the energy in that room was just phenomenal. 30 minutes after the session, people were still talking excitedly about what they were learning or how they were using Twisted or just sharing their love for the code :-)

For those of you that missed it, Steve Holden was the headliner while Alex Martelli played impromptu co-star. The humor and enthusiasm from these two was just incredible. Glyph, Itamar, and Chris played educators while JP, Zooko and I handled one-on-one questions in the audience. There were more players, but you get the point: it was a highly dynamic, lively and fun experience. Folks were so jazzed that conversations that night lasted long into the wee hours of the morning.

After almost two months' worth of post-PyCon follow-up, we're finally getting around to comparing notes. My biggest concern is for the absolute new-comer and the lack of intuitive and useful metaphors that would help aspiring Twisted users grasp the event-driven concepts of our code quickly. Steve and I are both interested in establishing a Proper and Good motivation for using Twisted. My girlfriend, who has a Masters in anthropology, was also there. Thanks to her insight and background, she has a completely different perspective of the community (and the new-comer dynamic at the session that night) and has some completely unique ideas for crafting a new generation of tutorial materials.

We're just getting started, but it's quite exciting. We expect to have more thoughts to share on the matter... in the form of materials as well as potential news items.

One last parting thought: despite the rumors and well-earned reputation to the contrary, Twisted coders are not exclusionists: everyone's invited to the party. We're just trying to make it easier to get there :-)

Technorati Tags: community, pycon, education, programming, twisted

Synthesis Studios Twisted Interview

Thu, 24 Apr 2008 06:04:39 GMT

This is a re-blog, in case some of you aren't subscribed to http://labs.twistedmatrix.com/ -- we've got a new episode of the Twisted Show up, and it was a excellent interview with Synthesis Studios. Go check it out!

Technorati Tags: community, entertainment, geeks, programming, software, twisted

TSF Founding Sponsor Round

Tue, 15 Apr 2008 22:04:16 GMT

Earlier today, I posted to the Twisted mail list and to our Labs blog about the deadline for the Founding Sponsors (individuals and companies/organizations) for the Twisted Software Foundation, so I thought now would be a good time to give a quick status report :-)

Things are going well. We're still working with 4 major donors (some of whom are surprising!), with an additional 3 outliers who may or may not donate. We're hoping to have all that confirmed and settled within the next two weeks. For the full list of current sponsors, please see the Founding Sponsors page.

So what does all this mean? It means that Twisted has a continually-increasing chance of meeting your needs and exceeding your expectations :-)

Sponsors will have access to a private email list that is managed by the TSF, and this will be the primary forum where desired features and issues to be addressed will be discussed. The donations help us address resource issues and the collective voice of the sponsors will help provide a focus on important topics.

Not only do the current sponsors have their logos+links on the front page, but Founding Sponsors also get this on a dedicated page on the Twisted site in perpetuity. We've got some amazing Google Juice (search), so this works out well for all. We've also been approached by sponsors who are using it as a means of recruiting Python and Twisted talent in their shops. There are all sorts of creative ways that this can be of benefit :-)

If you haven't stopped by the TSF pages, give them a look and see if it's something you or your organization could be interested in.

Update: Thanks to feedback from Grig Gheorghiu, we now have two domains that direct to the TSF page:

Be aware that the DNS for those may still be propagating when you click on them :-)

Technorati Tags: tsf, twisted

Twisted on Nexenta/OpenSolaris

Thu, 10 Apr 2008 04:04:39 GMT

A few hours ago, I had the chance to install Nexenta/OpenSolaris in Parallels. The install was pretty straight-forward and quick. Gnome isn't on the .iso, but this was easily addressed with a/etc/apt/sources.list update, and an apt-get (a ~450MB dependency download and install). With Gnome was up and running, I was amazed at its responsiveness: Gnome on Nexenta seems to be much snappier than Ubuntu 7.10. This is the first time I've seen something I could use instead of Ubuntu, and that's saying a lot.

Next up was getting the Twisted and Divmod code installed. This required the following additional package installs:

sudo apt-get install python2.4-zopeinterface
sudo apt-get install python2.4-profiler
sudo apt-get install python2.4-pyopenssl
sudo apt-get install python2.4-crypto

With that done, I ran trial twisted, and watched the tests zip by. The end result? Only 1 failure and 2 errors; that's a pretty significant improvement over Twisted in Solaris 10. The failure was actually a little bit weird: the test can't find /dev/tty, however the device does exist (and I can open it from the python prompt). The two errors came from the UDP "multi listen" test, and were are result of the test timing out.

If I can get those two tests sorted out, I'll start testing the Divmod code. If all goes well, this could very well end up being my new development platform.

Technorati Tags: divmod, gnome, python, solaris, twisted

The Problem with and Solution to Google's App Engine

Tue, 8 Apr 2008 21:04:14 GMT

I know everyone is all aglow with the new web development offering from Google, but let me do the unpopular thing and put some things into perspective: there are limitations.

In fact, the limitations that exist will prevent me from using App Engine with all of my projects, save one (that one being a very simple web site). First, the limitations that prevent me from using App Engine (from one of their FAQs):

Sockets are disabled with Google App Engine
The system does not allow you to invoke subprocesses, as a result some os module methods are disabled
Threading is not available

This means that I can't write a deferred wrapper for their data layer, I can't use Twisted for such things as XML-RPC or AMP-based communications, and I can't use an async templating system (like Nevow). I'm stuck with CGI and blocking code. And for all but the simplest projects, that's a big "No Thank You" from me.

This doesn't mean that I won't use it -- I will. I have one project that this will be perfect for... but it's for someone else, not me.

However, these limitations are actually good news :-) Here's the silver lining:

As Glyph as alluded to in his recent blog post (and in our tweets), we've recently completed a massive week-long BizDev Divmod sprint in Boston. One of the results of this is based on community feedback we've had over the last year, and which culminated at PyCon 2008 in Chicago with multiple requests for particular services from The Twisted Company. That result is a set of tools, features, and management options folks will be able to use with our software (app server, smart object db, network services, etc.). People really want to start using our stuff in cloud/grid computing environments. They need support for multiple and diverse network services, inter-store communications, massive deployments, etc. Two months before PyCon, we started working on tickets to support this, and we're making excellent progress toward providing the requested features.

We're still unclear as to which parts of this will be open source, as that will be driven by a combination of business and community demand. Regardless, Google's lack of support for this stuff has (for now) left the field wide open for us. And that, folks, is a big "Thank You Google!" :-)

Technorati Tags: business, community, divmod, google, grid, mantissa, networking, nevow, cloud, pycon, python, services, software

We're in the Kitchen, Cookin Ur Mealz...

Fri, 28 Mar 2008 05:05:05 GMT

As Glyph just said, our community/development site is "reloaded" with a fresh look and the beginnings of some new structure. This comes as a result of many interdependent activities in the offing here at Divmod, most of which are still in the oven (and we wouldn't want to ruin the surprise now, would we?).

At PyCon, many of you approached us about more than the Twisted stuff we've been working on, and we had some good conversations. We've listened to all of you and have been making Herculean strides to provide a clearer view into what we do and how it can help you. We've got a long way to go regarding site content improvements and enhancements to documentation, both of which are genuinely at the top of our list right now. Yes, we've got a vision; but that is truly nothing without the continued interest and support of curious and creative folks like you, who want to architect extraordinary software.

There's been an extraordinary amount of energetic development, conversation, brainstorming and contribution that's been happening on IRC, at our offices, and in the code base, by employees and community members, at work and at play -- it's terrific to be a part of something so genuine and organic (the rumors of inorganic overlords have been greatly exaggerated).

Come take a look! ... And stay for a while :-)

P.S. If there are any CSS-on-IE Super Freaks in the House, will you please stand up? We're trying to work on some styling oddities that exist on the site and we could use some help!

Update: Paul Hummer has been amazingly helpful and got us up and running on IE when he had spare moments today. Thanks Paul!

Technorati Tags: business, community, design, divmod, mantissa, nevow, python, web, work

Genshi in Nevow, Revisited

Thu, 27 Mar 2008 11:15:08 GMT

About a year ago, I blogged about using Genshi in Nevow. It was purely an exercise in curiosity on my part: I wanted to know how flexible Nevow's template parsing was. Could I just drop something else in there that was completely different? The answer was yes and no :-)

The "no" part of that was discovered by a kind reader, Karl Bartel, who found that the Genshi templates in Nevow were one-time wonders: after the first render, data was not refreshed. I didn't have time to take a look at why, since Zenoss was occupying so much of my time then. Now that I'm rocking out at Divmod, this sort of thing is more or less my business now :-) What's more, a friend recently made a deeply heart-felt plea to get Genshi working on Nevow, so I took another look tonight.

Here's the "yes" part of that answer :-) This only represents about an hour or so of work, so this hasn't been tested very thoroughly, but there's a trivial way to get last year's old, static code working such that Genshi templates are dynamically rendered in Nevow. Here's what I did to make it work:

preserve the original template data in a temp variable
reassign to self.template after Nevow's load() method has been called
clean Nevow's template cache

That's it :-)

If you want a single template to be able to be processed by both Nevow's templating machinery as well as Genshi's, you're going to have to make some sacrifices in efficiency. However, if you want a pure-Genshi solution for Nevow, you should be able to use this code to get something up in no time.

In addition to the bullet points above, I also did the following:

added some "method filters" so that Nevow stuff didn't get called by the Genshi loader
checked to see if the python attributes available to the Genshi templates were callable and if so, called them

As usual, leave a comment if you run across any problems.

Technorati Tags: community, genshi, howto, python, software, templating, twisted, web

Twisted on Solaris: Fail

Mon, 24 Mar 2008 04:06:02 GMT

No, this isn't the fail blog, but right now it ain't the supper happy dev blog, either. Let's do the Good first:

Good

To mis^H^Habuse Wind In His Hair's quote from Dances with Wolves, here's what I say to Sun: Do you see that I am your fanboy? Can you see that I will always be your fanboy?

As I said to Glyph last night while Solaris was installing on my iMac, I have the fondest memories of Sun, SunOS and Solaris. It ranks right up there with the time I spent on the Kaypro II CP/M machine as a kid in the early 80s, and far more special than the considerable time I spent on VAX/VMS machines at APSU and UMASS in the mid 90s. I remember doing physics projects in Mathematica and creating the first site for my web design business in '96 on SunOS 5 machines at UMCP. It felt real and solid... and it was exciting. After using SunOS, I rarely touched a PC. Even my early Linux experiments were just that: hacking ethernet drivers so that Slackware was usable for me (it was a couple years before I was running Mathematica on Linux :-)); it was the Sun machines I used for the real work.

So yesterday, once the install finished and I started up a CDE session, you can imagine my nostalgia :-) In case you can't, I took some pictures to remember it by... I'm gonna miss the CDE when it goes :-( It was great being back in these Motif stomping grounds, iconizing windows to the desktop, tweaking the environment in the same old ways. I look forward to the possibility of more of this in the near future.

As for add-on software, those Sunfreeware guys are both still at it and still insanely awesome. What would I ever do without them? I"ve been using the packages at this site almost as long as I've been using Solaris. This may be a question for the Lazy Web, but can we get an apt-get for sunfreeware.com? Post-OS-install installs are so slow, due to the manual pick-and-choose-and-download-and-unzip-and-pkgadd... honestly. Oh, wow -- after some digging, I just discovered that this already exists. (doesn't seem to work with Sunfreeware; does work with the old SunSite/ibiblio hosts, though).

Thanks to Sunfreeware, I had Python 2.5 and Subversion up and running in no time, co'ed Twisted and the Divmod code base, got Combinator installed, and started running Twisted test suites. Quick Combinator plug: with just four commands (2 per repository), I was on branch-management easy street. If you're using svn, Combinator should be your best friend...

Bad

You already know the big-bad, since you can see the title of this post :-) Twisted unit tests had 3 failures and 66 errors. Given that this represents about 0.07% and 1.59% of the tests, respectively, that's not too bad. The errors were higher at first, but my pyOpenSSL build was pointing at the wrong libs; there were obvious improvements once I pointed to the right libs.

Solaris x86 doesn't seem to perform as well. This matches the stories I heard a couple years ago, so I'm not that surprised. Solaris has a proven track record on Sparc; high performance on x86 may be a ways off. I ran the Twisted test suite on a VM install of Solaris and Ubuntu in Parallels (and thus both using identical hardware). During each test run, I left my machine alone, so there were no resource drains (or differences in utilization) either from the VM or Mac OS X.

Ubuntu 7.10
real 3m25.151s
user 0m54.655s
sys 1m22.189s

Solaris 10
real 5m32.329s
user 1m17.935s
sys 0m37.875s

As you can see, Solaris x86 runs a little more slowly. Once the test errors are addressed, it will be interesting to see if there is any change.

Ugly

It seems that one or more of the partitions started filling up towards the end of the tests, so several of the errors were exceptions.OSError: [Errno 12] Not enough space. During the install, I just accepted the default partitioning that Solaris offered for the 32GB Parallels virtual drive, so this is going to need some tweaking. For starters, /tmp is way too tiny, especially since that is what is used for package installs (unzipped temp files). Likewise / is way over-used; I should probably put /usr on its own partition.

However, the point is not to blow the dust off my old sysadmin skills, but rather that a default install isn't actually a proper environment for developers. If Sun wants devs to be able to jump right in and get going, then they need to either provide a better default or, if there is one, make it more obvious how to get to that default.

What's more (less?), when attempting to run the unit tests on Solaris, I kept getting GUI popup error messages saying that there were too many processes open and I should close some applications. I didn't dig at all to see which resources were limited; the purpose was to just get a complete test suite run, not do log file forensics and discover the source of the issues (which I hope to do in the future). I had to close all windows, save the terminal where I was running the tests.

Summary

So, what have I taken away from all this? Possibly the following:

Sun needs to put some resources into integrating the work that has been done with pkg-get; user communities thrive on easily available software, and developers count on it.
If I'm going to seriously consider using Solaris as a development platform, I need to start hanging out on the forums, get back up to speed; I'm sure there's lots of good stuff out there that simply hasn't gotten any exposure.
Similarly, I'm going to have to do some Solaris brush-up and then uncover some best practices -- using/adapting those that are extant, or creating new ones where there is a need.

The last two would be an investment on my part, but if Sun is serious about the Python community and supporting Python on Solaris, I'm willing to give it a shot.

So what's next? If that mythical entity "free time" truly does exist, I'm going to take a deeper look at Twisted on Solaris and start identifying the problems. I'll probably explore some system setup best practices too, and see what it would take to get pkg-get to support Sunfreeware.com.

Technorati Tags: community, open source, python, solaris, sun, testing, twisted

High Dynamic Range Imaging

Mon, 17 Mar 2008 18:03:39 GMT

So David Reid has a killer desktop background that looks like a game screenshot from Half Life 2. Turns out, it's a real photo and it was done using multiple exposure/composite techniques, something known as "high dynamic range" imaging. There's more info about that here:

http://en.wikipedia.org/wiki/High_dynamic_range_imaging

Here are some nice ones that I have found on flickr and elsewhere:

I still haven't found the one that is on his desktop background, but when I do, I'll post a link :-)

Technorati Tags: photography , pycon

Twisted Announces TSF

Sun, 16 Mar 2008 23:03:56 GMT

Hey everyone, in case you weren't at PyCon (or were, but slept-in on Saturday morning... or aren't subscribed to any of the Twisted feeds), we made an announcement about the Twisted's new membership in the Software Freedom Conservancy. There's a news post about it here where you can get the latest details.

Update: Thanks to feedback from Grig Gheorghiu, we now have two domains that direct to the TSF page:

Technorati Tags: community, tsf, pycon, twisted