Older blog entries for osullivj (starting at number 56)

Hacking: new release of templ. Minor fixes in the Python version driven by levitt development. C#, Java and Tcl code is unchanged.

Hardware acceleration: my first reaction to hearing about XML hardware acceleration was disbelief. Just another symptom of inappropriate application of XML, and code bloat in general, I thought. After all, XML seems to be the default formatting choice for any messaging job these days, irrespective of performance, interoperability or code ownership requirements. And so much enterprise coding reduces to shuffling data between UI layer, database and messaging format.

But then I thought again. Remember the 8087 ? CPUs didn't always have onboard FPUs. If you needed to do a lot of number crunching, you added the floating point capability. Likewise with graphics. CPUs used to do the bit blitting. As the demand for graphics applications increased, dedicated cards came onto the market that relieved the CPU of the graphics processing. Could we be seeing a similar trend for text ? After all, the string processing workload for the typical PC has been ever rising for the last ten years, with ever increasing volumes of email, HTML, XML etc. Maybe CPUs need specialised Text Processing Units alongside their FPUs and GPUs.

SSL & pymqi: I got an email about SSL and Les Smithson's pymqi this morning. I added some code to pymqi a couple of years ago, and it turns out that my code needs further extension to enable SSL support. I don't have time to do this work myself. But I thought I'd better blog my recommmendations so they're in the Google cache for anyone else who runs into this issue.

First off, you need IBM's SSL MQ sample. You can find this by Googling for ssl mq mc6c. It has some example code: amqscnxc.c, which shows the SSL config info being put into a struct that's passed to MQCONNX. The struct is an MQCNO which pymqi 4c doesn't implement. MQCNO holds a ptr to an MQCD, which I added to an earlier version of pymqi.

So to enable SSL configuration in pymqi, you must implement a Python class for the MQCNO struct, and allow it to be passed into QueueManager.connectWithOptions, and then into MQCONNX. The following changes to pymqi will be needed...

• Add class cno(MQOpts) to pymqi.py. Use my cd class as a template, but follow the structure of MQCNO.
• Change QueueManager.connectWithOptions() to add an extra param: an instance of your new cno class.
• Change pymqe_MQCONNX() in pymqe.c to extract the new param you added in 2, and pass it through to MQCONNX.

Insane programming: there's a myth about investment banking software commonly voiced by coders outside the pressure cooker world of trading systems development. It goes something like this: huge amounts of money are at stake, so the software must be super reliable and robust. Very often, the opposite is true. Time to market is the driving factor. Traders want functionality yesterday. If a bank can execute profitable business, it can afford to throw support staff at a flaky system. Bugginess will be tolerated if it means executing business that may have gone elsewhere.

In an earlier entry I mentioned that I don't blog about my day job, since wholesale banks are secretive and paranoid. Most other techies working on Wall St and the City of London keep quiet too. But some are starting to lift the lid. If you're curious about what it's like to code on a trading floor, you should be reading this blog.

Generators: are doing my head in ! Been coding up the generator based async IO approach I mentioned recently this afternoon. I want real, non blocking, multi threaded concurrency. I use generators to yield after sending down a socket. And then I resume when I get a Twisted callback telling me a response has arrived. That way no thread blocks waiting on a response. Less threads, more busyness.

Figuring out where to yield and when to return, and thinking about when I want resumable stack state, and when I don't, has been a head twisting experience. Which is great, because that means I am acquiring a non trivial design idiom. Last time I had this experience was my first multithreaded code in the late 90s. Before that it was OO in the early 90s. All good stuff.

Plan to throw six away: more levitt hacking last night. Really happy with the way the codebase is coming along at the moment. I've been working on this system since 2001. I've restructured and refactored so many times. For server infrastructure I started off with a plain medusa front end, and ZODB at the back. Then I switched to a Zope product approach, to leverage Zope's UI. Then I switched to a browser plug in implementation. Now I'm doing a standalone impl, that will probably use Twisted in a personal server deployment. And there were minor refactorings within those larger implementation decisions. I know Fred Brooks enjoined us to "plan to throw one away", but maybe I'm going too far. However, the upside is the implementation feels smaller and cleaner and far better factored.

clarkbw: my fave Skynyrd tune !

Generators: Python has them, and C# is proposing to add them. What are they ?

def GeneratorFunc( n):
    for i in range( n):
        yield i

 for i in GeneratorFunc( 3):
    print "i=%d" % i

The call to GeneratorFunc does not execute any of the code inside the GeneratorFunc. Instead it returns an iterator object that we can use in the for loop at the bottom. The for loop implicitly calls the iterator's next() method, which in turn executes the code inside GeneratorFunc. Every time this happens, the yield keyword supplies the value to be returned by the iterator's next() method. The important thing to remember is that all the local variables, and stack state, inside the GeneratorFunc body are preserved between invocations of the iterator's next() method. All this may be a little clearer if we invoke the iterator directly rather than relying on for to do it...

def GeneratorFunc( n):
    for i in range( n):
        yield i

 iter = GeneratorFunc( 3)
while 1:
    print "i=%d" % iter.next()

All well and good, you may say. But where can I apply this idiom ? I can see tremendous leverage in two common coding areas: recursive tree walks and blocking IO handler code. More later...

Financial technology blogging: I ply my trade as a software developer in the City of London, working for wholesale banks. So do many of the developers I know. Since wholesale banks tend to be secretive and paranoid, most of us who develop trading systems don't talk about it much in public forums. I don't talk about my professional work at all on line. But a few people are starting to put their heads above the parapet...

A highly talented colleague Quant dev in SF

Hacking: rewriting the core system for my main project -levitt - for the fourth or fifth time. I had the original idea for this stuff in early 2001, and I've been tinkering ever since. It started off as a Zope product, coded in Python. I'm sticking with Python, naturally, but have dropped Zope, and I'm moving away from object orientation too. Zope has a lot to offer in terms of server architecture and ready made UI. ZODB persistence, transactions, threading model and the management UI give a lot of leverage. But to exploit them you have to buy into the Zope Object Model - including acquisition - and that's too big an imposition. Especially since I'm now moving away from OO.

Why move away from OO ? Because it introduces too much coupling between data and behaviour. Mainstream OO, with single inheritance and strong compile time type checking, as exemplified in Java and C#, binds your method body code very tightly to your data, making it less reusable. Dynamic languages like Python and Smalltalk introduce less coupling between data and behaviour, so code is more generic and reusable. I aim to loosen that coupling even further in levitt.

Seeing an Advogato link to Xerblin got me really excited the other day. There are some similarities to levitt: it's a programming system project, coded in Python. Xerblin's developer has used Forth ideas in his kernel. I have been using a Forth like stack based core in levitt too. I'm now moving to something different, since I've come to think there's too much state on the stack. Oberon ideas are there in Xerblin too. I'm far less familiar with this system than Forth and Python, but it sounds cool. It's good to see some fresh approaches to programming systems, rather than the same tired old ideas from Java and .Net.

Books: war reading continues with Hitler and the Power of Aesthetics. The obvious and pressing question on Nazism is: how could it happen ? Most answers are framed in terms of brutality, coercion, desperation, economic catastrophe. These explanations have seemed incomplete to me for some time now. Last year I read Sebastian Haffner's Defying Hitler, which gives one a tremendous sense of how the Nazis skillfully played on the psychology of loyalty, camaraderie and patriotism to sweep people into complicity. Spotts book explains in compelling fashion how seductive the aesthetics of Nazism were: the torchlight parades, the SS uniforms, the swastika banners, Wagner's music, Speer's monumental neo-classical architecture, and Hitler's theatrical sense of neo-religious ritual. The book reminds me again of Paul Feyerabend commenting on how he nearly joined the SS instead of the German Army, because the uniform was so cool.

Another thing that made Nazism possible was Hitler's charisma. Our image of Hitler now is largely determined by Allied wartime propaganda, which portrays him as a ranting demagogue, foaming at the mouth. That can't have been how the German populace percieved Hitler in the 30s. Spotts has a lot to say on how Hitler didn't engage in traditional political debate, but made an emotional appeal to his audience. By all accounts, his personal presence was hugely charismatic and overpowering. So we should distrust charisma. Personal experience has led me to this conclusion too - when I've made mistaken career choices its been because I've fallen prey to charismatic and persuasive personalities. Charisma can hide a multitude of sins. I must do some more reading on the mechanisms of charisma.

Looks like Jeff Vroom has broken cover. Good luck with the quest Jeff !