Older blog entries for salmoni (starting at number 584)

Well I made it! I'm in New Zealand working for Westpac as an interaction designer for their website. All good fun! The work seems really cool and I have so many ideas to implement.

In other news, I've been exploring neural networks to predict currency markets and found a modicum of success (though nothing that translates into a prediction system that I could make money out of). Been using bpnn in Python. Python slows things down a lot but allows interactive analysis. I tried updating bpnn to use numpy but found my version to be significantly slower (eg, 1.5 seconds against over 5 seconds for the new one) which is odd. Is it worth releasing the code?

captchas

Just an idea for a new captcha system. How's about if you could access a large photographic resource (e.g., Flickr, Picasa), and a thumbnail of the picture is shown to the user. The user then has to guess one of the tags belonging to the picture. If they do, then they pass.

This cannot fight against brute force (I would imagine some tags will be quite common so some research would produce a frequency-based word list that could do this), mistakes would be expected (eg, the photo owner might put in random tags that make no sense to anyone else) and human-based captcha solving will easily get around it, but it's something to consider. Another crack would be to take the thumbnail and compare it against a DB of flickr pictures, but realistically, that's a large job. I wonder if Flickrs API can do that? To defend against this, the thumbnail could be altered somehow (eg, desaturated, change the colour balance etc) so that in machine terms, the images are different but in human terms they mean the same thing.

It's been a while since I posted - work, baby and travel issues have taken all my time and it's only now getting back to normal.

I'm still in the Philippines, still running an interaction design company, and having such a time that my dreams consist of writing algorithms (they are to calm me down).

The big program is still being developed and we're reconsidering UI toolkits. wxPython is wonderful and powerful, but it moves very fast and doesn't offer what I really need (embedded browser for rich interactive experiences, ie, Javascript). I understand that there are bindings to webkit and gecko, but these are not complete or reliable enough for production code. We're also considering XULrunner which does this but offers a less rich basic widget set.

My daughter, Louise Rhiannon Masaredo Salmoni was born at 1.52pm yesterday. Both mother and baby are doing well.

picture of my daughter

I'm totally in love with her: My life has just changed for ever...

Busy, busy, busy.

But not much to show for it. My wife and I are expecting our first child next Friday. This will be a nervous week indeed.

For my statistics project, I wrote a Python module to import SPSS files and was wondering whether anyone would be interested in it if I released it as open source. It's one piece of code that would greatly benefit from community testing. So far, it seems to work on the SPSS files I have without problem but SPSS have added extra things to the format. Cleverly (or rather obviously, but nice to know that they've done it), older versions of the software can still read the new formats, but they just ignore the extra bits that the new formats have. My software does just that: it ignores all the extra bits, though I suspect that there may be some cases where my software misses completely. For example, the architecture: I believe mine only reads one endian.

But it could be useful for some people. There are already FOSS versions in R and PSPP (I think the R version came from the PSPP code) but they are in C and a Python version might be useful. I wonder if SciPy has it? Currently, it can import via COM, but that is Windows only so of limited application. A pure Python module would have no such restrictions. Scientists using Python might appreciate being able to cut one more string to SPSS so I think I will release it. If anyone here is interested, let me know and it could be the spur that motivates me to release it!

I've started a usability consultancy officially instead of doing work ad hoc. This should be fun as I have to learn marketing very quickly indeed. In the Philippines, I think there is one independent consultant who is serious about the work (ie, has advertising) and a few others who seem to do it as a sideline. However, in nearby Hong Kong, there are two that I can find: Apogee and Customer Input. Looking through Google adwords shows that there is only a small market in HK compared to say the UK. However, we will be operating internationally so the location is of less importance. It does create some difficulties in terms of meeting the clients, but for general applications, I can easily get a good sample of users of varying abilities. It's also about time to put my remote testing experience to, erm, the test.

I'm also toying with the idea of joining the UPA whom I gave a talk for some while ago, but I need to check whether I will get my money's worth. It could be good for being noticed by potential customers.

5 Jun 2008 (updated 5 Jun 2008 at 12:17 UTC) »

I have finished a new Python module which is designed to import SPSS data files as a Python object. It seems to work quite well with the data sets I have. Not all functions are enabled yet: some of the type 7 records are not working yet, but for some I have to reverse engineer the solution and for others I need new data sets that use the subtype records.

When it's a bit more solid, I will probably release it under the AGPL.

llkc - I guess I didn't explain well. My idea isn't an interactive debugger, though there are elements of that in it. The best thing is to produce the code.

In other news, I should (hopefully, all being well!) be doing some consultancy soon. I'm not sure of the size of the job, but it sounds like a good one (ie, interesting and a challenge).

I noticed that Google have opened their appengine up completely now. Signups are ongoing here and it doesn't seem as though there is any limit to the number of users this time. Prices are also available here.

Perhaps this has been done, but I have been thinking about my ideal IDE for Python.

I like editors and have tried many. I also like interactive interpreters and have tried many. But my issue is that I often have to have both running at the same time (and yes, I know there are editors with interpreters running in them at the same time, but that's not what I'm thinking of).

But what about a dynamic editor/interpreter?

It sounds fanciful and I'm just beginning to think of the architecture but here is how it would work with Python.

You type some code in. It works interactively, so only executes when a block is entered. Or it may not. Each code block has a flag next to it that when activated causes the code to be marked as executable. When executed, only that code is run.

Ok, still fairly basic.

But what about if the user could also interactively run code separately from the stored blocks. So if I type in a large program, I can still type 'print "hi"' in the middle, click it, and it and only it will run.

But even better: what about if I can execute the code block by block or even line by line?

Again, this is not totally revolutionary. But what if you could change existing code and cause the program (assuming that it's still running and waiting for the user to enter the next code) to step back to a previous state? And then run up to the end with the new code?

And then when saving, the user can save a working version (with the interactive bits in place) and a "parade" version with all the interactive bits taken out.

I'm not sure this has been done (though if anything can, it's probably Eclipse or Emacs). I have probably described this idea poorly, but I think it could be a good thing that unifies the best of editor/IDE operation with the best of interpreters operation.

I'll have to work on a prototype and test it myself to see if it works.

29 May 2008 (updated 29 May 2008 at 09:13 UTC) »
Python Consulting

This is an announcement that I will be doing Python consulting from now. My expertise covers Python, wxPython, NumPy and SQLAlchemy; and the primary area of my work is on numeric analysis / statistics, though of course you get a PhD in human-computer interaction thrown in if you want interfaces made.

If anyone has any Python work they would like help with, I can offer a discount on open source code. I can work internationally as long as requirements can be sent electronically. The best way to contact me as salmoni - at - gmail.com

Apart from that, all is well here in the Philippines! The coding on the new project is going well and I'm considering farming off the database viewer/importer tool as a separate component for database management. I'm not exactly sure what functionality would be necessary for this, but suffice to say that the basics should be easy to implement (and the middling / advanced stuff a nightmare!).

Factorial ANOVA of large sets

I've also solved all the problems concerning factorial analysis of variance for extremely large datasets (ie, those too large to fit into memory). I will crack on with this code now to get it done and to make an industrial quality heavy-weight data analysis tool. This will be open sourced in time, after testing anyway. The real problems that I have are a) getting hold of an environment (ie, a machine with a massive database on it), and b) getting comparison results, though SAS should be able to deliver on this. I understand that SPSS will face problems if the data are too big for memory; but SAS can work around this just like my code can.

Moore's Law makes this of decreasingly utility; but it's nice to have software that you know can handle any task.

Article

I've also enquired about submitting an article to a Python journal about how to use the code module to implement an interactive interpreter and embed it within a Python program. This comes from work on the statistics program where I wrote one for quick debugging and found it so good that I extended it a little to be used as a permanent tool.

One problem we found is that when declaring and using a variable, a user would have to write:

x = newvar()

or

newvar("x")
x.data([3,4,5,6])

It would make more sense (to novices) to write

newvar(x)
x.data(3,4,5,6)

It does this now. What I did was override the code.InteractiveInterpreter.showtraceback method to catch NameErrors (which are risen when x is sent to newvar because x doesn't exist). Then the code works out the command and sends it again to the newvar method but with the x in quotes. It's minor stuff but less annoying to users.

And if a database has awkward variable names that are not valid variable names in Python, they cannot be used: so I added a catcher to showtraceback that catches AttributeErrors and tests to see if a string has been issued with a program method:

"Variable 1 (2000)".variance()

This would never work normally within Python without overriding the string class (which is another possibility). However, the catcher above can catch this attribute error and redirect the 'variance()' bit to the proper variable definition.

All this just means that the application is beginning to work around its users instead of demanding that they work around it.

I also added lots of alternative names for descriptive tests so:

x.samplestandarddeviation()
x.standarddeviation()
x.stddev()
x.stdev()
x.sd()

all call the same function. This helps because when I've used a new statistics program, I have to find out the exact name for the functions. This way, I don't have to remember which one: I just pick a common one, and away I go! :-)

I spent the weekend wrestling with factorial ANOVA code which was nice and fun. All seems to work alright but there is still some finishing and of course testing to do before it's anything like releasable. Plus I need to work on how to work things like post-hoc tests and simple effects for when a significant effect is observed. Lots of fun!

575 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!