Older blog entries for oubiwann (starting at number 281)

13 Oct 2012 (updated 14 Mar 2013 at 04:03 UTC) »

GIMP 2.8 and the Taming of Two Decades' Graphics Habits

At long last, I find myself in a 100% comfort zone with the GIMP. As a user, it's been a long road to get here ... I can't even imagine what the development teams have experienced over the past 16 years. Regardless, I am infinitely grateful for their efforts over that time, their hard work to bring the world a completely free, open source application for incredibly sophisticated image manipulation, drawing, and even painting.

I'll provide some more details on my experiences with GIMP 2.8, but first... a romp through the past!

I've been using graphics programs since my first exposure to the Macintosh Plus and SE machines in the late 1980s, where the head of the science department at Bangor High School often found himself without access to his machine (yeah, thanks to me...). As much as a science geek as I was, it was actually the graphics programs to which I was addicted. In particular, I was obsessed with using MacPaint to draw -- pixel by pixel -- a dithered bitmap of a photograph leaning next to the monitor.

About 6 or 7 years later, I was at it again doing graphics work on a non-profit's Mac (running System 7) with Photoshop 3.0 -- my first exposure to this software. It was absolutely mind-blowingly awesome what Photoshop enabled one to do with digital images. I was consumed. If I was awake, I was at the computer, creating something whacky from scratch or morphing something old into a new form. Though the work that I had done was focused on graphics for HTML and getting into the nascent field of "web design," some of my efforts ended up getting published as illustrations for a book.

I continued doing graphics as part of my work (in some form or another) from then on, usually borrowing a friend's machine in order to use Photoshop (which I could not afford). Despairing of not having my own copy of the software,  I would often try out other drawing, painting, and image manipulations applications. Without fail, none would ever measure up to the power and even more, the usability, of Photoshop.

It wasn't too much after this that I got involved with Linux and open source software. As for many from that time, my life has not been the same since. But those early years were painful. My first distro (Slackware), I had to manually hack the ethernet drivers in C just to get connected to the network using my particular hardware. Regardless, Linux was all I could think about; it was all I wanted to use.

Desperately hungering for a Photoshop-like experience on Linux, I searched HotBot on a regular basis, and finally discovered GIMP. Based on a quick glance at the now-historic splash screens, it looks like my first use of the GIMP started after the 1.0 release, though I didn't start using it more regularly until 1.2 (which had my favorite GIMP splash screen to date).

Despite my hopes and the best efforts of an amazing development team, the GIMP infuriated me. What took one step in Photoshop usually took about 5 steps in the GIMP. The user interface was anti-intuitive, and seemed to be built by folks with an intimate knowledge of the underlying API and reflected this far too well.

With each major release of the GIMP, I would find myself downloading and installing it, but only actually using it a handful of times. Ultimately, I'd run crying back to Photoshop, borrowing a friend's computer or persuading someone to give me an unused copy. The very nature of graphic arts is visual and intuitive... I felt that the GIMP was blocking that natural flow. As a result, I couldn't get anything done in it (and what little I did looked bad).

Year later, while working at Canonical, I had several opportunities to create graphics for various fun projects, gags, or practical jokes. Given the nature of the company's mission, I made sure to do that work using the GIMP. Though the pain of the old days had faded considerably, and GIMP had, by 2008, reached a much higher degree of usability, I still found myself enduring awkward workflow moments on a regular basis and this continued throughout the 4 years of version 2.6's release. However, during that time, an impressive selection of GIMP plugins were created, some of which I came to adore so completely that I would switch from Photoshop to GIMP, just so I could use them on a project. This was a new -- and amazingly welcome -- change for me.

Then, earlier this year, the last few pieces fell into place.

With the release of GIMP 2.8 in May, I have not looked for nor pined over any other sophisticated graphics programs. At long last, the GIMP completely satisfies. The biggest single point of pain for me in 2.6 (given that I was running on Mac OS X most of the time) was the lack of single-window  support. For every UI interaction, I had to click twice -- once to focus on the window, and once to actually perform the given action. It was crazy-making. 2.8 finally released the goodness of single-window mode for GIMP users around the globe. Futhermore, since I have been a layer-using maniac since Photoshop 3.0 and my graphics files tend to reach into the 200 and 300 MB range due to the numbers of layers (and resolution, of course), I have a desperate need to organize the chaos of all those layers. Prior to 2.8, my layer pallet on large projects was almost completely unnavigable. With 2.8, sanity and cleanliness abounds.

And there's more :-) Check out this short list:
  • single-window mode
  • multi-column dock windows
  • increased screen real-estate
  • layer groups
  • Cairo is used for all rendering
  • on-canvas text editing
For more details, you'll want to visit this page.

Oddly, this walk down memory lane and ultimate endorsement of the GIMP relates to MindPool.  I used the GIMP to do all the complicated graphics work for our initial branding, logos, etc. Looking at the results, I'm sure if seems fairly silly and even trivial... but there's a LOT that went into that simplicity :-)

For instance, the shot of the moon is actually taken from a photograph of the moon. It has been passed through multiple filters in order to get just the level of abstract color and shape I was looking for. Similarly for the water, but the settings were actually quite different to achieve the result I was seeking. I think the layer count peaked around 30 in one version of the file.

Another example is the logo that we'll be using for the company's main site. This also took advantage of some plugins (new and old), though mostly it relied upon paths, brush strokes, and mask layers (the latter for the water pool reflection watermarked into the image). The new features in 2.8 made all of that work very easy to do. No longer is the tool interfering with my workflow; rather, it is facilitating it. Working with graphics using open source tools is now an absolute pleasure for me.

For giggles, I'm including a screenshot of GIMP 2.8 in action, with my single-window mode view stretched across two 27" monitors (full version is here). Graphical editing has never been so clean!

Thanks, GIMP team :-) It's nice to finally be home.

Fun link: For the historically inclined, you might enjoy this quick read about the beginnings of Photoshop and the family that made it all happen :-)

Syndicated 2012-10-13 21:00:00 (Updated 2013-03-14 03:44:13) from Duncan McGreggor

15 May 2012 (updated 15 May 2012 at 20:02 UTC) »

CERN, OpenStack Keep Resonance Cascades at Bay

Tim Bell preparing to get his
OpenStack on
As previously mentioned, there's a growing momentum around ops-oriented participation in the OpenStack community. DreamHost is deeply invested in DevOps, seeing how that's where we're going to be living in a few months! As Simon Anderson, CEO of DreamHost, recently said:
"When we're running a complex fabric of apps on over 5,000 servers across three data centers, we need a lean and nimble approach to software development and operational implementation. Without a DevOps approach, we wouldn't be able to push code into production as fast or as efficiently as we do, and our customers would not be happy! Today's developers demand up-to-the-hour security and performance updates to Internet infrastructure, so we aim to deliver just that with DevOps."
Though expressed in the context of our work, the import of DevOps that Simon's comment generally highlights is going to be increasingly important for nearly anyone running cloud services. 

In particular, I've been following the work of the intrepid folks at CERN. As such, this post is not about DreamHost; rather, it's a mad tale of OpenStack, DevOps, and averting alien invasion.

After countless long-distance phone conversations, a flight to Switzerland, and spending several days buying pints for a security guard in the know (referred to from now on as "Barney"), I've uncovered some profound truths -- Mulder-style -- and have confirmed that the impact of OpenStack at CERN is huge. 

Superficial examinations turn up the usual: CERN's planning slides, nice quotes, discussions of features and savings in time and money. For instance, in a recent email conversation with Tim "Gordon Freeman" Bell at CERN, I learned that 
"The CERN Agile Infrastructure project aims to develop CERN's computing resources and processes to support the expanding needs of LHC physicists and the CERN organisation."
I think these guys have been hanging out with Simon! But once you slip behind the scenes, peek at some of the whiteboards in unattended rooms, or rifle through notes lying about, you see that things are not what they appear. I've included a shot of Mr. OpenStack-at-CERN himself; this was my first clue.

Publicly, he's been working with other teams at CERN to:
  • modernise the data centre configuration tools and automating operations procedures
  • exploit wide scale use of virtualisation, improving flexibility and efficiency
  • enhance monitoring such that the usage of the infrastructure can be fully understood and tuned to maximise the resources available
But privately, it seems that he and his team have been doing much, much more. This was alluded to in a statement made by team member Jan van Eldik: "We expect the number of requests to insert non-standard specimens into the scanning beam of the Anti-Mass Spectrometer to significantly decrease, once automation is in place and everyone is using the standard infrastructure we are setting up."

That isn't to say there haven't been incidents...

Innocuously enough, the current toolchains are based around:
  • OpenStack as a single Infrastructure-as-a-Service providing physics experiment services, developer boxes, applications servers as well as the large batch farm
  • Puppet for configuration management
  • Scientific Linux CERN as the dominant operating system with sizeable chunk of Windows installs
But that second bullet caught my eye, and one of Barney's pub mates confirmed a rumor that we'd heard: the Puppet instances are actually trained headcrabs. The primary training tool? You guessed it, a crowbar. Barney said that the folks from Dell took inspiration from this and developed it further for their OpenStack deployment framework after an extended visit to CERN.

Although Barney hadn't seen any evidence of resonance cascades, there have been minor cross-dimensional disturbances as a result of some "cowboy" activity and folks not following DevOps best practices. This has been kept quiet for obvious reasons, but has led to a small pest problem in some of CERN's older tunnel complexes. As rouge elements are discovered, CERN has been educating transgressors aggressively. (Sometimes they go as far as sending employees to Xen training... or was it Xen training?)

One artist's conception of what success will
look like for OpenStack at CERN
Despite the minor hiccoughs along the way, CERN is aiming for success. (Given the lack of Combine and forced relocation programs, they're already doing better than Black Mesa's Anomalous Materials team.) Plans are in place for an initial pre-production service, OpenStack deployment this year. Following that, they will be moving towards 300,000 virtual machines on 15,000 hosts spread across two data centres by 2015.

The OpenStack community is supporting them in their efforts with fantastic new features, high-quality discussions on the mail lists, and real-time interaction on the IRC channels. In an act of reciprocity and community spirit, operators at CERN have volunteered to contribute back to the OpenStack community with regard to operations best practices, reference architecture documentation, and support on the operators' mail list.

To see how other institutions were taking this news, I spent several days waiting on hold. In particular, Aperture Science could not be reached for comment. However, Ops team member Belmiro Rodrigues Moreira did say that there's an audio file being circulated at CERN of Cave Johnson threatening to "burn down OpenStack" ... with lemons. Given Aperture Science's failure record with time machine development, it's generally assumed to be a prank audio reconstruction. CloudStack developers are considered to be the prime suspects, seeing how much time they have on their hands while waiting for ant to finish compiling the latest Java contributions.

When asked what advice he could give to shops deploying OpenStack, Tim said simply: "Remember, the cake is a lie. Don't get distracted and don't stop. Just keep hacking."

Alyx, explaining to her dad why she loves DreamHost
Couldn't have said it better myself.

In closing, and interestingly enough, one of DreamHost's employees has an uncle who works at the Black Mesa Research Facility. Though his teleportation research team was too busy for an extended interview, his daughter did mention that she is a DreamHost customer and can't wait to use OpenStack while interning at CERN next summer. After all, that's what she uses to auto-scale her WordPress blog (she's in our private beta program).

It's a small world.

And, thanks to Tim and the rest at CERN, a safer one, too.

Syndicated 2012-05-15 06:00:00 (Updated 2012-05-15 19:05:32) from Duncan McGreggor

13 May 2012 (updated 15 Apr 2013 at 05:03 UTC) »

Twisted SSH: Rendering a Log-in Banner/MOTD in Conch

A few weeks ago, I pinged my peeps on #twisted asking why the banner for a custom SSH server wasn't rendering properly. After some digging around and some inconsistent results (well, consistently bad results for me), we weren't able to resolve anything, and I had to set the problem aside.

The Symptom
The first thing I had tried was subclassing Manhole from twisted.conch.manhole, overriding (and up-calling) connectionMade, writing the banner to the terminal upon successful connection. This didn't work, so I then tried overriding initializeScreen by subclassing twisted.conch.recvline.RecvLine. Also a no-go. And by "didn't work" here's what I mean:

In both Linux (Ubuntu 12.04 LTS, gnome-terminal) and Mac (OS X 10.6.8, Terminal.app), after a successful login to the Twisted SSH server, the following sequence would occur:
  1. an interactive Python prompt was rendered, e.g., ":>>"
  2. the banner was getting written to the terminal, and
  3. the terminal screen refreshed with the prompt at the top
This all happened so quickly, that I usually never even saw #1 and #2. Just the second ":>>" prompt from #3. Only by scrolling up the terminal buffer would I see that the banner had actually been rendered. Even though I was doing my terminal.write after connectionMade and initializeScreen, it didn't seem to matter.

Some time last week, I put together example Twisted plugins showing what the problem was, and the circumstances under which a banner simply didn't get rendered. The idea was that I would provide some bare-bones test cases that demonstrated where the problem was occurring, post them to IRC or the Twisted mail list, and we could finally get it resolved. 'Cause, ya know, I really want my banners ...

While tweaking the second Twisted plugin example, I finally poked my head into the right method and discovered the issue. Here's what's happening:

  • twisted.conch.recvline.RecvLine.connectionMade calls t.c.recvline.RecvLine.initializeScreen
  • t.c.recvline.RecvLine.initializeScreen does a terminal.reset, writes the prompt, and then switches to insert mode. But this is a red herring. Since something after initializeScreen is causing the problem, we really need to be asking "who's calling connectionMade?"
  • t.c.manhole_ssh.TerminalSession.openShell is what kicks it off when it calls the transportFactory (which is really TerminalSessionTransport)
  • openShell takes one parameter, proto -- this is very important :-)
  • openShell instantiates TerminalSessionTransport
  • TerminalSessionTransport does one more thing after calling the makeConnection method on an insults.ServerProtocol instance (the one I had tried overriding without success), and as such, this is the prime suspect for what was preventing the banner from being properly displayed: it calls  chainedProtocol.terminalProtocol.terminalSize
  • chainedProtocol is an insults.ServerProtocol instance, and its terminalProtocol attribute is set when ServerProtocol.connectionMade is called.
  • A quick check reveals that terminalProtocol is none other than the proto parameter passed to openShell.

But what is proto? Some debugging (and the fact that of the three terminalSize methods in all of twisted, only one is an actual implementation) reveals that proto is a RecvLine instance. Reading that method uncovers the culprit in our whodunnit:  the first thing the method does is call terminal.eraseDisplay.

Bingo! (And this is what I was referring to above when I said "poked my head" ...)

Since this was called after all of my attempts to display a banner using both connectionMade and initializeScreen, there's no way my efforts would have succeeded.

Here's What You Do
How do you get around this? Easy! Subclass :-)

The class  TerminalSessionTransport in t.c.manhole_ssh is the bad boy that calls terminalSize (which calls eraseDisplay). It's the last thing that TerminalSessionTransport does in its __init__, so if we subclass it, and render our banner at the end of our __init__, we should be golden. And we are :-)

You can see an example of this here.

Not sure if this sort of thing is better off in projects that make use of Twisted, or if it would be worth while to add this feature to Twisted itself. Time (and blog comments) will tell.

As is evident from the screenshot above (and the link), this feature is part of the DreamSSH project. There are a handful of other nifty features/shortcuts that I have implemented in DreamSSH (plus some cool ones that are coming) and I'm using them in projects that need a custom SSH server. I released the first version of DreamSSH last night, and there's a pretty clear README on the github project page.

One of the niftier things I did last night in preparation for the release was to dig into Twisted plugins and override some behaviour there. In order to make sure that the conveniences I had provided for devs with the Makefile were available for anyone who had DreamSSH installed, I added subcommands... but if the service was already running, these would fail. How to work around that (and other Twisted plugin tidbits) are probably best saved for another post, though :-)

Syndicated 2012-05-13 00:50:00 (Updated 2013-04-15 04:25:41) from Duncan McGreggor

6 May 2012 (updated 13 May 2012 at 02:02 UTC) »


Do you love architecting new and creative software? Are you a hacker with mad Python skills and a freak for distributed services? Would you like to see your work offered to a huge, Internet audience? Do you want to help build a community around your work? Do you always vote for the underdog?

I've got just the position for you!

DreamHost is hiring for a new, senior engineering opening on the Cloud Team in the Development Group, and if you can not only easily imagine the extraordinary skill sets necessary to do what we're planning, but also have that skill set, we have got to talk.

We're a small company that's pure heart-and-soul with a culture that simply can't be beat. We've been hiring some incredible talent from the Python and open source communities, and need to finish building out this visionary team that will be taking DreamHost into the next 10 years of online services and software. This role is particularly focused on shaping that future -- from a technical as well as strategic perspective.

You can email me or ping me on IRC (oubiwann on freenode.net). We can chat, and if you've got what it takes, we will set up an interview with the rest of the Cloud Team.

I look forward to hearing from you :-)

Update: There's be a lot of interest in this position from folks with a wide range of professional experiences, so I've shared the job description here. This should give you a good sense of what we're looking for from your past, and the sorts of things we'd be expecting in your future :-)

Syndicated 2012-05-06 14:38:00 (Updated 2012-05-13 01:54:33) from Duncan McGreggor

New Life: The OpenStack DevOps Community

Last week's OpenStack Design Summit and Conference were pretty fantastic events. Lots of great technical discussions, some good initial planning on large tasks, incredible numbers of conversations -- all of high quality! Looking back, though, I'd have to say that the high point of the event for me was what turned into the DevOps community brainstorm session (etherpad link) at the Design Summit (all etherpads are here).

The session was approved with the title of "OpenStack and Operations: Getting Real" with a major goal of deciding whether we needed to create a new top-level project for DevOps. So it was really "New DevOps Team?" However, all that changed once we got started, and there was some amazing feedback and enthusiasm in that room. By the end of it, we were all pretty pumped up and ready to jump in with all our hands and feet!

The highest-level summary for me was this: The DevOps folks in the OpenStack community really need a point to rally around. Someplace they can not only talk to each other, share stories, get advice, etc., but also where they can have their voices heard by the predominantly developer-oriented OpenStack community. Jay Pipes suggested that a new section be added to the weekly IRC team leader meetings, and as Nova Ops subteam lead (teams list), I volunteered to collect top issues from the DevOps (sub-)community and give these some air-time in the meetings.

Some other highlights from the session:

  • trystack.org needs volunteer admins -- a great opportunity for improving the dev <-> ops interface
  • CERN is deeply invested in DevOps, and wants to share data and possibly help with defining reference architectures
  • The same goes for the University of Melbourne!
  • There was a good corporate presences in the session too, rallying around a new and improved DevOps community: HP, Yahoo, AT&T, Canonical, Rackspace, DreamHost, and more (sorry if I forgot you!) 

There is a tremendous amount of information that we collected, but in written form (etherpads) as well as conversations during last week's event. As this is collected, we'll be reaching out to folks via the operators mail list, blog posts, tweets, and Google+ messages. Be sure to do the same! If you've got concerns, bring them up on the mail list, we'll get some bugs filed and blueprints put together, and come up with plans for addressing these.

Thanks again, everyone!

Syndicated 2012-04-25 19:47:00 (Updated 2012-04-25 19:47:40) from Duncan McGreggor

7 Apr 2012 (updated 12 Apr 2012 at 05:02 UTC) »

New Domain Name for the Blog

JP's wonderful blog sub-domain (and clever joke sub-domain) got me to thinking about what to do for a blog name again. Especially in the event that I ever leave blogger. For a year I tried using tau.tologo.us, but I think that was far too obscure a play on words. I gave up for a few years after that, and just stuck with the original username.blogspot.com setup.

After revisiting the possibilities again, looking at everything from rephrasing Maxwell's equations as sub-domains to using a play on Hindi words (I almost went with blog.bija.li, doing the electric thing as well as going for the esoteric seed syllable/mantra reference). Also too obscure.

After too many hours on Google Translate (trying something out, getting it translated, back-translating it; that often provides humorous -- if not enlightening -- results), I managed to stumble across one I really liked. Immediately, in fact: cogitat.io.

How is it that this hasn't been taken? I was dumb-founded. And delighted :-) Check out the possible translations of the Latin goodness:
  • thought
  • cogitation
  • thinking
  • considering
  • deliberating
  • reflection
  • meditation
  • reflexion
  • thought
  • design
  • plan
  • reasoning power

Splitting words at the tld, we get the aptly wonderful cogitat i/o: "he thinks I/O." As a programmer who specializes in side effects (don't most of us?), I'm surrounded by I/O. I view it as software's analog to the ever-important physics (and life) concept of "cause and effect." Needless to say, I do love the tld.

With the domain all set up, I was off to Blogspot... only to find that Google doesn't let you use "naked" domains as your blog name.


So then it was back to the drawing board (Google Translate). Fortunately, after just a few minutes of playing around, technicae popped up, and I thought this might be it. technicae cogitat io can almost be translated as "meditating on technology" or "the design of technology."

But there's more: I saw that if io was typed with a capital I, it would render the translation as "John." Which has led to the ultimate, most profound, inner-most secret essence of this blog:
Technicae cogitat Io.
Videre Io currunt.
Currere, Io! Currere!
Probably not something something Cicero would have recognized, nor even something parsable in Vulgar Latin. Most of you have probably already got it, though. Regardless, here it is in naked English:
"John thinks about technology.
See John run.
Run, John! Run!"

Syndicated 2012-04-06 23:11:00 (Updated 2012-04-12 04:45:36) from Duncan McGreggor

5 Apr 2012 (updated 5 Apr 2012 at 16:02 UTC) »

Recent Stackiness


Tomorrow is OpenStack Atlanta's second event (the first being a HackIn).  Ken Pepple is going to be talking about deploying OpenStack, something he should feel very comfortable doing, given his book as well as Internap's latest announcement :-)

There's more info about tomorrow's event at the Meetup page.

OpenStack Design Summit

In a couple weeks, a bunch of us from DreamHost are going to be heading to the OpenStack Summit and Conference in San Francisco. There's a lot of buzz about it both inside the company, in the offices of our fellow OpenStack collaborators, and in the wider open source community. With Citrix's recent announcement, Internap's deployment, Eucalyptus' approval by Amazon, there's plenty of Cloud Drama to go around. Fortunately, the focus of the Summit and Conference is on the important positives: how to improve an extraordinary piece of software and disseminating expertise. Can't wait!

GitHub Love

Last but not least, the Dev team at DreamHost has been using Github in conjunction with Launchpad in a manner similar to how the OpenStack project does it. The increased interest in open source software in our offices is starting to make its way out to our customers, and we've got a new web presence that is the first step in supporting this new direction. We're cooking up a bunch more stuff, so be sure to check in on our repos from time to time :-)

Syndicated 2012-04-05 04:55:00 (Updated 2012-04-05 15:14:10) from Duncan McGreggor

24 Mar 2012 (updated 26 Mar 2012 at 14:02 UTC) »

A Conversation with Guido about Callbacks

In a previous post, I promised to share some of my PyCon conversations from this year -- this is the first in that series :-)

As I'm sure many folks noticed, during Guido van Rossum's keynote address at PyCon 2012, he mentioned that he likes the way that gevent presents asynchronous usage to developers taking advantage of that framework.

What's more, though, is that he said he's not a fan of anything that requires him to write a callback (at which point, I shed a tear). He continued with: "Whenever I see I callback, I know that I'm going to get it wrong. So I like other approaches."

As a great lover of the callback approach, I didn't quite know how to take this, even after pondering it for a while. But it really intrigued me that he didn't have the confidence in being able to get it right. This is Guido we're talking about, so there was definitely more to this than met the eye.

As such, when I saw Guido in the hall at the sprints, I took that opportunity to ask him about this. He was quite generous with his time and experiences, and was very patient as I scribbled some notes. His perspective is a valuable one, and gave me lots of food for thought throughout the sprints and well into this week. I've spent that intervening time reflecting on callbacks, why I like them, how I use them, as well as the in-line style of eventlet and gevent [1].

The Conversation

I only asked a few initial questions, and Guido was off to the races. I wanted to listen more than write, so what I'm sharing is a condensed (and hopefully correct!) version of what he said.

The essence is this: Guido developed an aesthetic for reading a series of if statements that represented async operations, as this helped him see -- at a glance -- what the overall logical flow was for that block of code. When he used the callback style, logic was distributed across a series of callback functions -- not something that one can see at a glance.

However, more than the ability to perceive the intent of what was written with a glance is something even more pragmatic: the ability to avoid bugs, and when they arise, debug them clearly. A common place for bugs is in the edge cases, and for Guido those are harder to detect in callbacks than a series of if statements. His logic is pretty sound, and probably generally true for most programmers out there.

He then proceded to give more details, using a memcache-like database as an example. With such a database, there are some basic operations possible:

  • check the cache for a value
  • get the value if present
  • add a value if not present
  • delete a value
At first approach, this is pretty straight-forward for both approaches, with in-line yielding code being more concise. However, what about the following conditions? What will the code look like in these circumstances?
  • an attempt to connect to the database failed, and we have to implement reconnecting logic
  • an attempt to get a lock, but a key is already locked
  • in the case of a failed lock, do re-trys/backoff, eventually raise an exception
  • storing to multiple database servers, but one or more might not contain updated data
  • this leaves the system in an inconsistent state and requires a all sorts of checking, etc.
I couldn't remember all of Guido's excellent points, so I made some up in that last set of bullets, but the intent should be clear: each of those cases requires code branching (if statements or callbacks). In the case of callbacks, you end up with quite a jungle [2]... a veritable net of interlacing callbacks, and the logic can be hard to follow.

One final point that Guido made was that batching/pooling is much simpler with the in-line style, a point I conceded readily.

A Tangent: Thinking Styles

As mentioned already, this caused me to evaluate closely my use of and preference for callbacks. Should I use them? Do I really like them that much? Okay, it looks like I really do -- but why?

Meditating on that question revealed some interesting insights, yet it might be difficult to convey -- please leave comments if I fail to describe this effectively!

There are many ways to describe how one thinks, stores information in memory, retrieves data and thoughts from memory, and applies these to the solutions of problems. I'm a visual thinker with a keen  spacial sense, so my metaphors tend follow those lines, and when reflecting on this in the context of using and creating callbacks, I saw why I liked them:

The code that I read is just a placeholder for me. It happens to be the same thing that the Python interpreter reads, but that's a happy accident [3]; it references the real code... the constructs that live in my brain. The chains of callbacks that conditionally execute portions of the total-possible-callbacks net are like the interconnected deer paths through a forest, like the reticulating sherpa trails tracing a high mountain side, like the twisty mazes of an underground adventure (though not all alike...). 

As I read the code, my eyes scan the green curves and lines on a black background and these trigger a highly associative memory, which then assembles a landscape before me, and it's there where I walk through the possibilities, explore new pathways, plan new architectures, and attempt to debug unexpected culs-de-sac. 

Even stranger is this: when I attempt to write "clean" in-line async code, I get stuck. My mental processes don't fire correctly. My creative juices don't flow. The "inner eye" that looks into problem spaces can't focus, or can't get binocular vision. 

The first thing I do in such a situation? Figure out how I can I turn silly in-line control structures into callback functions :-)  (see footnote [1]),

Now What?

Is Guido's astute assessment the death of callbacks? Well, of course not. Does it indicate the future of the predominant style for writing async Python code? Most likely, yes.

However, there are lots of frameworks that use callbacks and there are lots of people that still prefer that approach (including myself!). What's more, I'd bet that the callbacks vs. in-line async style comes down to a matter of 1) what one is used to, and possibly, 2) the manner in which one thinks about code and uses that code to solve problems in a concurrent, event-driven world.

But what, as Guido asked, am I going to do with this information?

Share it! And then chat with fellow members of the Twisted community. How can we better educate newcomers to Twisted? What best practices can we establish for creating APIs that use callbacks? What patterns result in the most readable code? What patterns are easiest to debug? What is the best way to debug code comprised of layers of callbacks?

What's more, we're pushing the frontiers of Twisted code right now, exploring reactors implemented on software transaction memory, digging through both early and recent research on concurrency and actor models, exploring coroutines, etc. (but don't use inlineCallbacks! Sorry, radix...). In other words, there's so much more to Twisted than what's been created; there's much more that lies ahead of us.

Regardless, Guido's perspective has highlighted the following needs within the Twisted community around the callback approach to writing asynchronous code: 
  • education
  • establishing clear best practices
  • recording and publicizing definitive design patterns
  • continued research
These provide exciting opportunities for big-picture thinkers for both those new to Twisted, as well as the more jaded old-timers. Twisted has always pushed the edge of the envelope (in more ways than one...), and I see no signs of that stopping anytime soon :-)


[1] In a rather comical twist of fate, I actually have a drafted blog post on how to write gevent code using its support for callbacks :-) The intent of that post will be to give folks who have been soaked in the callback style of Twisted a way of accepting gevent into their lives, in the event that they have such a need (we've started experimenting with gevent at DreamHost, so that need has arisen for me).

[2] There's actually a pretty well-done example of this in txzookeeper by Kapil Thangavelu. Kapil defined a series of callbacks within the scope of a method, organizing his code locally and cleanly. As much as I like this code, it is probably a better argument for Guido's point ;-)

[3] Oh, happy accident, let me count the hours, days, and weeks thy radiant presence has saved me ...

Syndicated 2012-03-24 16:03:00 (Updated 2012-03-26 13:49:35) from Duncan McGreggor

Python for iOS

I do a lot of traveling, and I don't always like to lug my laptop around with me. Even when I do, I'd rather leave it in the bag unless I absolutely need to get it out (or if I'm setting up my mobile workspace). As such, I tend to use my iPhone for just about everything: reading, emails, calendar, etc.

So, imagine my delight, when I found out (just after PyCon this year) that I can now run Python 2.7.2 on my iPhone (and, when I get it, my iPad 3 ;-) ). This is just too cool for words... and given what pictures are worth, I'll use those instead :-)

I've put together a small Flickr set that highlights some of the functionality offered in this app, and each image in the set describes a nifty feature. For the image-challenged, here's a quick list:

  • an interactive Python prompt for entering code directly using the iPhone keyboard
  • a secondary, linear "keyboard" that one can use in conjunction with the main keyboard, extending one's ability to type faster
  • multiple options for working with/preserving one's code (email, saving to a file, viewing command history)
I can't even begin to count the number of times such an awesome Python scratchpad would have come in handy. And now we have it :-) At $2.99, this is a total steal.

Thanks Jonathan Hosmer!

(And thanks to David Mertz for pointing it out to folks on a Python mail list.)

Syndicated 2012-03-17 20:50:00 (Updated 2012-03-17 20:50:44) from Duncan McGreggor

PyCon 2012: To Be Continued

PyCon was just fabulous this year.

It's been a couple years since I was able to go, and I was quite surprised by how much I had been missing it. The Python community is not only one of the most technically astute and interesting ones to which I belong, but also the kindest. That last point is so incredibly important, and it ends up fostering a very strong familiar sense amongst its members.

There were so many good conversations with such great people: Anna, Alex, Guido, David (Mertz), Donovan, JP, Maciej, Allen, Glyph, Paul, Sean... the list goes on and on! Fortunately, I took notes and (and even have some book recommendations to share!) so there are many blog posts to come :-)

But this has brought something into focus quite strongly for me: the interaction at PyCon is one of the most fertile grounds for me all year -- and going without it since Chicago has been a genuine drought! There were some folks at DreamHost that couldn't make it, and we've already started looking around at various local, mini Python conferences that we can attend. This was initially so that those who couldn't make PyCon could receive similar benefits. But now there's something equally important that's contributing to the importance of this search: attending local conferences will mean not as much time has to pass between those fertile interactions and that recharging that we give each other at such events.

Until next time, I hope all Pythonistas everywhere are getting ready for a great weekend :-) Those who have been traveling, I hope you get lots of rest and share with everyone the treasures gathered at this year's PyCon :-)

Syndicated 2012-03-16 18:43:00 (Updated 2012-03-16 18:43:47) from Duncan McGreggor

272 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!