oubiwann is currently certified at Journeyer level.

Name: Duncan McGreggor
Member since: 2001-11-21 22:13:44
Last Login: 2008-10-23 14:55:14

FOAF RDF Share This

Homepage: You're looking at it.

Notes: If you would like to leave comments on the blog posts, visit the blogger version.


  Electric Duncan
  Earth. It's where I keep my stuff.



[del.icio.us] [project pages] [hacking society] [LinkedIn] [sourceforge] [orkut] [Last.fm] [openid]

blog roll:

[python] [zope] [politics] [friends] [physics] [math] [madness]

I can be reached at the following address: 
echo "wngvtg@twrmnf.nl" | tr a-z h-za-g
(thanks for the tip, fxn !)

Projects

Recent blog entries by oubiwann

Syndication: RSS 2.0

21 Jun 2009 (updated 21 Jun 2009 at 05:05 UTC) »

Mac OS X: Execute Shell Commands via Icon-Clicks



My main development machine is a custom PowerBook running Ubuntu natively. I use it when I'm sitting on the couch, my office comfy chair, the futon, floor, etc. Every once in a while, though, I want to work at a desk from my 24" iMac. Just to mix it up a little. However, that box is my gaming and web-browsing machine: it runs Mac OS X and that's the way I want to keep it. So, if I'm going to do work on the iMac, I need to ssh into the machines that have the environments set up for development.

In the course of an average day of writing code, I'll connect to anywhere from 1 to 5 remote machines open up 5-10 ssh sessions in a terminal to each machine. If I'm at the iMac, this get's tedious. Today, it got tedious enough for me to do somthing about it. Here's what I want: to click on a Terminal icon and have an ssh connection automatically established to the box I need to work on. This it pretty easy on Linux and Windows, but I had no idea how to accomplish this on a Mac until tonight.

I thought I'd share my solution; others may like it... but I'm betting there are some pretty cool ways of doing this that didn't occur to me -- so feel free to share yours!


Profile Hack

From previous messing about with the open command, I knew I could open Terminal.app from the terminal:
open -n "/Applications/Utilities/Terminal.app"
This got me part way there... if only I could dynamically execute a command upon login... so, yeah, I did something nasty:
vi ~/.bash_profile
And then:
if [ ! -z "$REMOTE_CONNECTION" ]; then
    ssh $REMOTE_CONNECTION
    REMOTE_CONNECTION=""
fi

.command Files

I was stumped at that point, until some googling revealed a nifty trick I didn't know about:
  • Create a new file in your favorite editor, using the .command extension
  • Add the commands you want executed
  • Save it and chmod 755
  • Double-click it and enjoy
So here's what I added to rhosgobel.command:
REMOTE_CONNECTION=rhosgobel \
   open -n "/Applications/Utilities/Terminal.app"

The Obligatory Icon Tweak

I then used the standard "Get Info" trick of icon copying: "Get Info" for Terminal.app, copy icon, "Get Info" for all my .command files, paste icon.


Usage

Now, I just click my "Shells" menu, choose the destination, and start working on that machine. A new window or new tab opened with that instance of Terminal.app will give me a new session to that server, without having to manually ssh into it -- this is even more convenient than having an icon to double-click!

One bit of ugly I haven't figured out how to remove: when I open a shell to a remote server, there's another shell opened at the same time with a [Process completed] message.


Syndicated 2009-06-21 03:01:00 (Updated 2009-06-21 04:09:57) from Duncan McGreggor

18 Jun 2009 (updated 18 Jun 2009 at 22:05 UTC) »

A Sinfonia on Messaging with txAMQP, Part III


A Sinfonia on Messaging:
  1. The Voice of Business
  2. The Voice of Architecture
  3. A RabbitMQ and txAMQP Interlude

Before we play our third voice in this three-part invention, we need to do some finger exercises. In particular, let's take a look at the concepts and tools we'll be using to implement and run our kilt store messaging scenario.


Messaging

The RabbitMQ FAQ has this to say about messaging:
Unlike databases which manage data at rest, messaging is used to manage data in motion. Use messaging to communicate between and scale applications, within your enterprise, across the web, or in the cloud.
Paraphasing Wikipedia's entry on AMQP:

The AMQ protocol is for managing the flow of messages across an enterprise's business systems. It is middleware to provide a point of rendezvous between backend systems, such as data stores and services, and front end systems such as end user applications.


AMQP Essentials

AMQP is a protocol for middleware servers -- servers that accept, route, and buffer messages. The AMQP specification defines messaging server LEGO blocks that can be combined in various ways and numbers, achieving any manner of messaging goals, whose final forms are as diverse as the combinations of the components.

For the visually inclined, below is a simple diagram of the AMQ protocol. I've put multiple virtual hosts in the diagram to indicate support for multiple server "segments" (domains in the most general sense). There could just as easily be multiple exchanges and queues in each virtual host, as well. Likewise for publishers and consumers.




















I highly recommend reading the spec: it is exceedingly clear at both intuitive and practical levels. To better understand the diagram above, be sure to read the definition of terms at the beginning as well as the subsections in 2.1 about the messaging queue and the exhange. Don't miss the message life-cycle section either -- you'll be reminded of circuitry diagrams and electronics kits, which is what AMQP really boils down to :-)

The Advanced Messaging Queing Protocol specifies that the the protocol can be used to create exchanges, message queues, chain them together, and do all of this dynamically. Any piece of code that has access to an API for your AMQP server can connect to it and communicate with other code -- using or creating simple messaging patterns or deeply complex ones. And everything in between.


RabbitMQ Quickstart

RabbitMQ is a messaging system written in Erlang, but in particular, it is an implementation of AMQP. The RabbitMQ web site provides documentation on installing and administering the messaging server. I run mine on Ubuntu, but since I've got a custom Erlang install, I didn't install the package (I dumped the source in /usr/lib/erlang/lib). To participate in the code play for this blog series, you'll need to install RabbitMQ.

Once you've got it installed, you'll need to start it up. If you've used something like Ubuntu's apt-get to install RabbitMQ, starting it up is as simple as this:
sudo rabbitmq-server

If you've got a custom setup like mine, you might need to do something like this (changing the defaults as needed):
BASE=/usr/lib/erlang/lib/rabbitmq-server-1.5.5/
BIN=$BASE/scripts/rabbitmq-server

RABBITMQ_MNESIA_BASE=$BASE/mnesia \
  RABBITMQ_LOG_BASE=/var/log/rabbitmq \
  RABBITMQ_NODE_PORT=5672 \
  RABBITMQ_NODENAME=rabbit \
  $BIN &


A txAMQP Example

Now that we've got a messaging server running, but before we try to implement out kilt store scenarios, let's take a quick sneak peek at txAMQP with a simple example having the following components:
  • a RabbitMQ server
  • a txAMQP producer
  • a txAMQP consumer
From reading the spec, we have a general sense of what needs to happen in our producer. It needs to:
  • connect to the RabbitMQ server
  • open a channel
  • send a message down the channel
Similarly, our reading lets us anticipate the needs of the consumer:
  • connect to the RabbitMQ server
  • open a channel
  • create an exchange and message queue on the RabbitMQ server, binding the two
  • check for in-coming messages and consume them
I have refactored some examples that the author of txAMQP created and I've put them up here. Once you download the three Python files (and the spec file, one directory level up), you can run them in two separate terminals. In terminal 1, start up the consumer:
python2.5 consumer amqp0-8.xml
In terminal 2, fire off a message:
python2.5 producer amqp0-8.xml \
 "producer-to-consumer test message 1"
After running the producer with that message, you should see the same text rendered in the consumer terminal window. You can also fire the message off first, then start up the consumer. The message is sitting in a queue on your RabbitMQ instance and will be available to your consumer as soon as it connects.

Now that you see evidence of this working, you're going to be curious about the code :-) Go ahead and take a look. There are lots of comments in the code that give hints as to what's going on and the responsibilities that are being addressed.

If you are familiar with Twisted, you may have noted that the code looks a little strange. If you're not, you may have noticed that the code looks normal, with the exception of extensive yield usage and the inlineCallbacks decorator. Let me give a quick overview:

Ordinarily, Twisted-based libraries and applications use the asynchronous Twisted deferred idiom. However, there's a little-used bit of syntactic sugar in Twisted (for Python 2.5 and greater) that lets you write async code that looks like regular, synchronous code. This was briefly explored in a post on another blog last year. The Twisted API docstring for inlineCallbacks has a concise example.

Briefly, the difference is as follows. In standard Twisted code, we assign a deferred-producing function's or method's return value to a variable and then call that deferred's methods (e.g., addCallback):
def someFunc():
    d1 = someAsyncCall()
    d1.addCallback(_someCallback)
    d2 = anotherAsyncCall()
    d2.addCallback(_anotherCallback)

With inlineCallbacks, you decorate your function (or method) and yield for every deferred-producing call:
@inlineCallbacks
def someFunc():
    result1 = yield someAsyncCall()
    # work with result; no need for a callback
    result2 = yield anotherAsyncCall()
    # work with second result; no need for a callback

Visually, this flows as regular Python code. However, know that the yields prevent the function from blocking and execution resumes as soon as the deferred has a result (which is assigned to the left-hand side). Since this latter idiom is used in txAMQP, I use it in the examples as well.

Next, we finally reach our implementation!


References



Syndicated 2009-06-18 17:40:00 (Updated 2009-06-18 21:28:53) from Duncan McGreggor

12 Jun 2009 (updated 18 Jun 2009 at 19:07 UTC) »

A Sinfonia on Messaging with txAMQP, Part II


A Sinfonia on Messaging:
  1. The Voice of Business
  2. The Voice of Architecture
  3. A RabbitMQ and txAMQP Interlude

After writing the last blog post, I found a fantastic site that focuses on messaging in the enterprise. I have really enjoyed the big-picture overview I get from some of the Martin Fowler signature books in this series, so I ordered a copy of this one too.

On the web site, the authors give a nice example for messaging integration in Chapter 1. They provide a more detailed, supplier-version of the kilt store (we're doing "manufacturing" as opposed to distribution) with "Widget-Gadget Corp", but the basic principles are the same. I highly recommend reading that entire page. I used it as the basis for much of this post.


Business Process Overview

At a top-level, we have the following business process for MacGrudder's kilt store:



These are represented by a sales guy or a web store, a third-party billing service, MacGrudder, and a third-party shipping service.

Up until now, the sale process could be either a user deciding to buy something in the online store or the sales guy engaging with a customer. Both generated orders; neither shared resources. The web app interfaced with the payment gateway operating by billing/shipping guy. The sales guy had to call in his orders to the billing/shipping guy. Once orders were charged/approved, a printout was handed to MacGrudder, who then created the ordered kilt. Once completed, he'd set it aside for shipping guy to come box it up and slap a label on it.


The Voice of Architecture

We're now ready to weave in the second voice of our three-part invention. MacGrudder's original infrastructure consists of silos of applications, functionality, data, and process. We want to interconnect these separated areas in order to reduce long-term overhead incurred by redundant components and data. Practically, we want to see the following changes:


Unified orders: at the end of the sales process, there should be one abstraction of the "order", regardless if the source was the web store, a phone call, or the sales guy. The order abstraction will be a message (or series of messages, for orders with multiple items; we'll be addressing only the simple case of a single item).


Unified validation and billing: at the end of the order creation process, there should be a validated order or an invalid order (e.g., if there were insufficient funds). At the end of the billing process, there should be an approved order that has be paid for via a single means (e.g., a single payment gateway, without bothering billing guy for manual entry). Additionally, once an order has been validated, messages should be sent to other components in the system.


Unified status: at the end of the manufacturing process, both the shipping guy and customers should be aware that the product has been completed and is ready to be sent: the shipping guy can connect to our messaging system (probably via a service) and the customer can be notified by email or by checking the order status in the web kilt store.


In the next installment, we will finally start looking at the code. We'll look at the "unified orders" messaging solution after covering some basics with RabbitMQ and Twisted integration, and then see how far we get with implementation details and descriptions. Unified validation, billing, and status might have to be pushed to additional posts.


References



Syndicated 2009-06-12 03:03:00 (Updated 2009-06-18 17:58:53) from Duncan McGreggor

7 Jun 2009 (updated 18 Jun 2009 at 19:07 UTC) »

A Sinfonia on Messaging with txAMQP, Part I


A Sinfonia on Messaging:
  1. The Voice of Business
  2. The Voice of Architecture
  3. A RabbitMQ and txAMQP Interlude

Prelude

A complete messaging solution is a three-part invention of business need, architecture, and implementation. In its final form, these three voices blend in harmony, with each one taking a dominant role depending upon which part of the solution one examines.

Neither have I the ability nor skill to seamlessly weave three concepts together while clearly explaining their roles. Therefore, I will separate out the voices from each other and leave it as an exercise for the reader to construct an application and practice the principles involved, thus experiencing well-earned contrapuntal pleasures first-hand.

Introduction

As computing and data exchange systems increased in complexity over the past 30 years, so has the need for improvements -- and where possible, simplifications. Some of these efforts have been focused on decentralization of communications (shared, distributed load) and decoupling of messaging from applications (removing redundancy and increasing delivery speed/throughput). The first steps towards this were made in the 1990s with explorations in "middleware" application universe.

Messaging, as we now refer to it in the industry, arouse from those middleware adventures: out of the business drive to refactor old software as new services to wider, more sophisticated audiences. With many new services replacing a single, monolithic application, formal and well-architected solutions were needed for creating, editing, and deleting shared data.

From Wikipedia:
Message-oriented middleware (MOM) is infrastructure focused on message sending that increases the interoperability, portability, and flexibility of an application by allowing the application to be distributed over multiple heterogeneous platforms. It reduces the complexity of developing applications that span multiple operating systems and network protocols by insulating the application developer from the details of the various operating system and network interfaces.
AMQP (Advanced Message Queuing Protocol) is one of these protcols.

In a recent blog post about Ubuntu as a business server, Vaughan-Nichols provides evidence for Ubuntu's and Canonical's commitment to enterprise, saying "... the new [version of] Ubuntu also includes AMQP [...] support. AMQP is an important set of middleware and SOA [...] protocols."

AMQP has demonstrated itself as a compelling protocol for messaging solutions, even to the point of being included in two Linux distributions. The code included in this blog series is txAMQP, an asynchronous Python AMQP library built with Twisted.

Note that last year I had planned to write a blog series on messaging with Twisted and XMPP, but was unable to as a result of time constraints. These days, I'm working with AMQP instead of XMPP, but I still hold some hope that I'll be able to write an anolog for this series from the perspective of XMPP.


The Voice of Business

Dipping into the future, one of the goals for AMQP as developed by a special interest group is the following:
Decentralized, Locally Governed Federated Mesh of AMQP Brokers with standardized Global Addressing. The killer application for AMQP is transacted secure business messages between corporations - e.g. send a banking confirmation message to confirms@bank.com [...]
I find this rather exciting due to my interest in ultra large-scale systems; scenarios like the one described above are the seeds for tomorrow's ULS systems :-)

For now, though, let's look at a more immediate use for AMQP: a messaging protocol for shared services between departments in a small store. In this exercise, the voice of business is the primary melody; everything else (architecture and implementation) is done in support of this theme.


An Example

Fhionnlaidh MacGrudder creates hand-made kilts to order. He's got a sales guy who works with movie costume design shops and the like. He's got a web girl who wrote and maintains a custom store front app. He's got a friend who does shipping and billing for him (as well as some other local Glen Orchy artisans). Until now, these three business "groups" associated with the kilt shop have been maintaining their own records, sometimes copying and updating them manually from each others' various export files.

Fhionnlaidh's niece Fíona is programmer, business student, and is dating shipping guy's son. Horrified by the inefficiencies in her uncle's business processes (and tired of her boyfriend's father's complaints), she has proposed the following:
  • sales guy will maintain customer contact info offer this as a service to the store and billing
  • the web store will dynamically update displayed data when sales guy changes it
  • the CRM will dynamically update displayed data when a web store customer updates their info
  • MacGrudder will have a new web page he goes to where all pending orders are presented with their full details; changes can be updated by a customer in real-time until MacGrudder has started working on the order
  • billing/shipping guy will be notified instantly as soon as MacGrudder marks a kilt order as completed
This setup has the following benefits:
  • Contacts will be maintained in a single data store
  • There is zero latency between customer-driven updates and sales guy-driven updates
  • Customers have increased post-purchase flexibility with their orders
  • Shipping guy can plug into MacGudder's messaging and be notified when packages are ready for pickup
  • Everyone has more time for buttered scones and tea (especially shipping guy, who will no longer be making unneeded trips down the glen)
The following changes will be made to the current software:
  • Contacts will need to be merged into the CRM
  • A read/write data service for the contacts will need to be created
  • The CRM front-end will need to be upgraded to an AJAX-enabled version
  • The web store app will need to be updated to support AJAX
  • A new page will be created which displays the status of all orders and allows MacGrudder to change an order from "pending" to "in-progress" to "completed"
  • The current "new order" email notification code in the web app will need to be changed so that it uses the same messaging as MacGrudder's status page
  • A new service needs to be created for shipping guy so that he can choose to be notified about pending pickups by email or he can check a web page or even make a query directly to the service, thus preventing unnecessary trips to MacGrudder's isolated little shop
  • After all the work is done, somone's going to need to order more scones
This example is not meant to fully justify messaging for businesses, but rather to provide a simple use case for which we can write some simple (and less than robust) code. It is a toy, but a conceptually useful one with a solid, concrete foundation.

In the next installment, we'll review the business process (with diagrams!) and the explore the architecture of the system, before and after. Another post will take that architecture and combine it with MacGrudder's already extant infrastructure, reusing as much as possible. With that in place, we will have the opportunity to look at some RabbitMQ basics and some actual txAMQP code.


References



Syndicated 2009-06-07 21:22:00 (Updated 2009-06-18 17:59:07) from Duncan McGreggor

28 May 2009 »

After the Cloud: Epilogue


After the Cloud:
  1. Prelude
  2. So Far
  3. The New Big
  4. To Atomic Computation and Beyond
  5. Open Heaps
  6. Heaps of Cash
  7. Epilogue

Though wildly exciting to imagine a future of computing where ubiquitous devices are more deeply integrated into the infrastructure we use to power our applications, the real purpose of these posts has been to explore possibilities.

Let's have crazy thoughts. Let's build upon them, imagining ways in which they could become a reality. Let's not only munch on the regular diet of the technical "now"; let's plant seeds in many experiments for the future.

This post is about thinking of small, mobile devices and cloud computing. But it's also a rough template. Let's do the same thing for a the desktop: how might it evolve? Where will users be spending their time? Let's do it for the OS and the kernel: what radical changes can we envision there? The technology behind health care. Education. Our new patterns of behaviour in a constantly changing world. All of these and more deserve our attention.

The more we discuss such topics in a public forum, the more thought will be given to them. Such increased awareness and attention might spark the light of innovation years ahead of time, and do so in the context of an open exchange of ideas. Let's have Moore's law for the improved quality of life with regard to technology; let's take it out of the chip and into our lives.



Syndicated 2009-05-28 10:45:00 (Updated 2009-05-28 10:49:18) from Duncan McGreggor

235 older entries...

 

oubiwann certified others as follows:

  • oubiwann certified fuzzyping as Master
  • oubiwann certified oubiwann as Journeyer
  • oubiwann certified nornCabor as Apprentice
  • oubiwann certified moshez as Master
  • oubiwann certified itamar as Master
  • oubiwann certified amk as Master
  • oubiwann certified effbot as Master
  • oubiwann certified Uche as Master
  • oubiwann certified calderone as Master
  • oubiwann certified spiv as Journeyer
  • oubiwann certified glyph as Master
  • oubiwann certified mesozoic as Journeyer
  • oubiwann certified ChrisMcDonough as Master
  • oubiwann certified fdrake as Master
  • oubiwann certified ianb as Master
  • oubiwann certified tav as Journeyer
  • oubiwann certified jafo as Master
  • oubiwann certified Namosui as Apprentice
  • oubiwann certified rbhalotia as Journeyer
  • oubiwann certified essallen as Apprentice
  • oubiwann certified roozbeh as Journeyer
  • oubiwann certified titus as Master
  • oubiwann certified pvanhoof as Journeyer
  • oubiwann certified cactus as Master
  • oubiwann certified wingo as Journeyer
  • oubiwann certified etrepum as Master
  • oubiwann certified gstein as Master
  • oubiwann certified eikeon as Journeyer
  • oubiwann certified faassen as Master
  • oubiwann certified jml as Master

Others have certified oubiwann as follows:

  • fuzzyping certified oubiwann as Master
  • oubiwann certified oubiwann as Journeyer
  • nornCabor certified oubiwann as Master
  • lerdsuwa certified oubiwann as Journeyer
  • Namosui certified oubiwann as Master
  • rbhalotia certified oubiwann as Master
  • pvanhoof certified oubiwann as Journeyer
  • fxn certified oubiwann as Journeyer
  • mirwin certified oubiwann as Master
  • Omnifarious certified oubiwann as Master

[ Certification disabled because you're not logged in. ]

New Advogato Features

FOAF updates: Trust rankings are now exported, making the data available to other users and websites. An external FOAF URI has been added, allowing users to link to an additional FOAF file.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

X
Share this page