Older blog entries for oubiwann (starting at number 92)

Tired of coding twisted? Need a break to relax?

Play a game...

Now playing:

Muse - Butterflies and hurricanes

One of the first things I did when I got back home on Sunday was check the laptop I left at home... and sort the 1800+ emails, spam a-taunting.

Of the odd hundred or so that were valid communications, one was from a friend with a heads-up/contact about RedHat's new effort to break into the burgeoning monitoring market. Apperantly, they are looking for python talent with coders having a proven background in providing systems management and monitoring solutions.

I find this interesting not only because of my long involvement in the monitoring arena (including the pymon and CoyMon projects), but also because of my recent work with Zenoss and the contact I have had with competitors in the field over the past year.

My interest in architecting efficient and fast monitoring systems led me to develop pymon in twisted. This was, in part, a reaction to my not-so-happy experience with NetSaint/Nagios. On the other hand, CoyMon was positively inspired by Cacti, with an aim to give similar usability to NetFlow data. I have found it instructive to learn the various impetuses that drove current monitoring solutions to being developed. I wonder what is driving RedHat. Business need? Business opportunity? Chronic issues with their current system?

On a side note, I did some more work on pymon this weekend, and I am really happy with the way the internals of it are progressing, primarily 1) how damned fast it runs, and 2) how easy it is to setup and configure. Most of all, though, I find the design increasingly flexible and elegant. There's still some horrible crud tucked in the older crevices, but that stuff's on its way out.

pymon is not explicitly systems management software, but rather a service-checking application. A part of me is curious, though, about what could be built using pymon's design principles. There could be some interesting possibilities to explore there.

Anyway, back to the topic: Systems monitoring and management a huge market and it is getting a lot of development effort and investment attention. When the dust settles in a year or two, it will be most interesting to see where all of this leads and what code bases/companies are left standing.

Now playing:

Muse - Ruled by secrely

Sunday I returned from a two-week road trip through Wyoming, Montana, Idaho, Washington, Utah and then back to Colorado. Though the scenery was amazing, the best part was the actual purpose of the trip: seeing friends, some of whom it has been 15 years since we last hung out.

Of the many very cool things that happened on the trip, one of the things I can share in a public forum is the RPG experience I had in Pullman, WA visiting my dear friend Jim Roach. He and his good friend Jacob DMed/GMed two sessions each and it was just phenomenal. Jacob's was a Ravenloft campaign I got to join with a 54-year old character named Li Po (monk) on a journey to Romania in the late 1800s. The other was a GURPS character -- a kilt-clad goblin samurai. I haven't gamed with sessions like that in over 10 years, and it was so much fun I will probably joining them again via web cam.

Upon my return to Colorado, I had a new Sector 9 bamboo longboard waiting for me. Two days and no spills. I'm totally not into tricks, but damn, I love to cruise and slalom. I've had a delightful time getting reacquainted with my skating muscles, though my body currently disagrees, as I can barely walk up the stairs right now.

Now playing:

Muse â€`` Plug In Baby

This is the BEST VIDEO EVAR.

How can you possible beat it? It's got kung fu, tai chi, spaghetti western, gun fighting, post apocalyptic America, lasers guns, holograms, and rock and roll. Take the best parts of the 70s (pop-culturally and ideologically) and pack as much of that as you can into a 6-minute video, and this is what you get. It's the music video equivalent of this poster.

And it's now responsible for making me a Muse fan.

I've recently needed to profile some very subtle issues that cropped up in a customer's python application. However, when I tried to use hotshot, I consistently got tracebacks. After some digging around on the net, I saw folks saying that profiling is basically busted in python2.4 (and then I remembered Itamar saying basically the same thing at PyCon 2006 when we were looking at web2 slowness).

To get around this, I built python2.5 from svn and copied its cProfile, _lsprof and pstats files to my python2.4 libs. This was a complete desperation move and I totally didn't expect it to work -- but it did (with only a warning about a version mismatch).

Earlier this year, JP and Itamar updated an lsprof patch to work as a standalone. However, I've never done any profiling in python, so it took a few minutes to get up to speed. Looking at the patch source and the python2.5 cProfile docs and then doing the usual dir() and help() on cProfile.Profile in the python interpreter is what helped the most.

To give others new to profiling a jumpstart, I'm including a quick little toy howto below.

Import the junk:

>>> import os
>>> import cProfile
>>> import lsprofcalltree

Define a silly test function:

>>> def myFunc():
...   myPath = os.path.expanduser('~/kcrw_s')
...   print "Hello, world! This is my home:"
...   print myPath
...

Define a profile object and run it:

>>> p = cProfile.Profile()
>>> p.run('myFunc()')
Hello, world! This is my home:
/home/kcrw_s/kcrw_s
<cProfile.Profile object at 0xb7c87304>

Get the stats in a form kcachegrind can use and save it:

>>> k = lsprofcalltree.KCacheGrind(p)
>>> data = open('prof.kgrind', 'w+')
>>> k.output(data)
>>> data.close()

You can now open up the prof.kgrind file in kcachegrind and view the (in this case, very uninteresting) results to your heart's content.

Well, these last two months have been hell. Working on sucky projects just plain sucks. It ate up so much of my time that I've got an unhealthy backlog of blog posts waiting for release into the wild.

The first post I am compelled to write is on my new mail server. In the past, I've used sendmail, qmail and postfix. I have not been happy with any of those (though I did really like qmail, until it got too cumbersome to keep it updated). I didn't have the level of control I wanted -- and this was solely because I couldn't fit my brain into those applications (though, again, qmail came the closest).

I decided to chuck it all, and write my own mail server in twisted, using the pre-built lego code that twisted offers for this sort of thing. I've been running the server for about a month now, and all I can say is "wow". Just WOW.

The level of control I have over the operation of my mail server is insane. I can get this thing to do exactly what I want, when I want. I've got a bazillion domain names for which I (or others) receive email. I was able to write the code that lets me handle that the way that makes sense for ME (and *not* the author(s) of Postfix, etc.).

Today, I needed to add support for aliases that were actually lists of recipients. One "if" statement and an additional implementation of smtp.IMessage later, it was operational. I don't know how I ever ran a mail server any other way.

I've been testing my mail server all month, and it's running beautifully. It has continued to be free of relay issues and spammer attacks. I couldn't be happier with the results.

Now that I am feeling more secure in the custom code, I'm ready to start adding additional features I need:

• white listing: automatically updated with the address of people to whom I send email
• black listing: I am starting to maintain a list of the most heinous offenders in my junk mail box; these will be regularly pushed to the mail server
• greylisting: I have begun planning an implementation of greylisting, but this will take some time to get right
• spammer databases: I am considering using one or more of these. My only problem with them is that I don't trust them. I don't want to block someone inadvertantly just because they were unlucky enough to have one of their boxes raped into becoming an open relay 3 years ago.

Having a mail server that runs on twisted seemed a little daunting at first. I feared maintenance and security burdens, however, I have already begun reaping the benefits and my fears have been shown to be baseless. I spend 1/10th the maintenance time on the twisted version. I have *fun* while updating the server or configurations. I can plug my own code into it instead of using third party applications I don't like or patches I don't understand.

My first exposure to twisted was late 2002 as I was writing my first "real" python script (a networking script, naturally). Since that time, twisted has integrated itself into my life such that I can't imagine living without it. I literally use it for all of my coding activities: my professional life depends upon it nearly 100%, and 50% of my entertainment is derived from programming activities, all of which incorporate some aspect of twisted.

I, for one, welcome our twisted overlords.

Now playing:

Bagpipes â€`` Flow Gently Sweet Afton

It seems I was the last to find out about the "trac spam" phenomenon. I've got a bunch of projects that use trac here, and as a result, I've been cleaning up the huge mess these bastards left on the wiki. Gotta love page history, though...

The pages that are most viewed are the ones with code examples on them, and the spammers were kind enough to delete that code, making the pages useless. The code is now restored and can be referenced again. In particular, the following projects were impacted most heavily:

• AmazonDBDriver
• NetCIDR
• PyRRD

I've now got the wiki set for auth users only. If you want to contribute to the wikis, just send me an email.

Nevow needs a screencast. Because it's so damned hot and people need to see it in action. They need to see how frickin' amazing it is.

Case in point: on a *whim* I decided to migrate a site I built last year in z3 to nevow. I had a spare hour, so what the hell. It took me *minutes*. And I don't mean 6487 minutes -- I mean 30 minutes. Here's what I did:

• Copied HTML output from the other site and used as the basis for an XHTML template
• Copied CSS and images from the other site
• Wrote a backend data store that uses RFC 2822 (python's email package)
• Wrote a ui.UI class, subclassing nevow.rend.Page (including children, data calls, etc)
• Added data and render calls in the XHTML template
• Wrote a .tac file
• Populated 5 RFC 2822-compliant files with data and headers for the content
• Started up the server and watched the new site scream

Let me emphasize a crucial point here: *none* of that was stubbed. No helper-scripts. No installers. There was pre-existsing HTML, CSS and images, and that's it. I did some dicking around and tweaking in the process, so I'm *sure* I could do the whole thing -- from start to finish -- in less than 20 minutes. Probably 15.

I don't understand why more developers aren't using Nevow. It just kicks ass. Yeah, every platform has its issues, and Nevow's no exception. But DAMN. I get record-setting development times with it, it runs fast, and it's insanely flexible.

A few years ago I was working on a sub-project of CoyoteMonitoring where we were writing a database "layer" for the file system, essentially allowing us to query for files using standard SQL commands. We needed to do some exclusionary logic for filtering bad user input. Though mathematically equivalent, I deal with Set Theory and Logic very differently: I'm a set guy, and I really hate using series of if/else statements for problems that lend themselves easily to an approach that is "set theoretic" (we're talking super-mundane set stuff -- nothing really sexy).

On IRC, I was gently preparing a fellow project developer for some code changes I had made that eliminated a series of ugly ifs:

[03-Oct-2004 02:02:33]  <oubiwann> you know, we could do this really 
quickly with sets :-)
[03-Oct-2004 02:02:37]  <oubiwann> check this out:
[03-Oct-2004 02:02:57]  <oubiwann> lets say our table has 4 fields:
[03-Oct-2004 02:03:13]  <oubiwann> ['A','B','C','D']
[03-Oct-2004 02:03:50]  <oubiwann> and we're doing a select for ['D', 
'A', 'C']
[03-Oct-2004 02:05:14]  <oubiwann> and lets say we've got a bad query 
['Z', 'B', 'C']
[03-Oct-2004 02:07:47]  <oubiwann> then, if we perform a set difference 
on select
   tables against defs, the returned set will be zero
[03-Oct-2004 02:08:28]  <oubiwann> and if we do the same with bad 
against defs,
   we'll get a non-empty set, in this case a set with one element, 'Z'

I then shared some python 2.3 code (no built-in set object) that basically illustrated the changes I had made. For me, the results were much more readable. The code was certainly much shorter.

Since then, I've found all sorts of uses for this approach. I've recently made use of this in a Nevow project that has to authenticate off of a Zope instance. There are a series of group and role names that can be qualified under one of three "group-groups" (tiers):

# setup sets for each class of user
admins = set(cfg.user_database.admin_roles)
libs = set(cfg.user_database.tiertwo_roles)
vols = set(cfg.user_database.tierone_roles)
# and then a convenience collection
legal_roles = admins.union(libs).union(vols)

Later in the code, we do checks like this:

if admins.intersection(avatarRoles):
     return IAdministratorResource

I like this much more than explicitly checking for the presence of elements in lists.

Well, the folks at tummy.com have done it again :-) Baz camp is rocking. We're right on the lake, with beaches and 90-degree weather in Loveland, CO... so you can imagine that the view is most agreeable :-) We not only have wireless access donated by the most gracious local provider LP Braoadband, but some of the boys brought a big-screen projector for outdoor movies at night as well as satellite TV. So yeah, we're *really* roughing it.

We are certainly roughing it more than the folks at Foo and Bar Camp!

We've got a camp cook who's making meals for us (including 1:00am chili!) and the menu has been just great. We've got shaded gazebos up, tents. Watched Primer and Hackers last night (Hackers was enjoyed a la MST3K). Sean share Icelandic treats he brought back from the Need for Speed sprint, including fish jerky. Yummy.

I also happen to be getting a lot of work done, too :-) All offices should operate like Hack Camp.

Hey Radix: why aren't you here? Evelyn Mitchell and I were talking about fellow social geeks and agreed that this would be just your style :-)