He who obeys, obeys Bayes?

Yup, another set of possibly ill-conceived thoughts on spam filtering. There was a good round-up of what's currently going on in yesterday's NTK, incidentally. (Search for Tracking in that page).

What the current crop of Bayesian spam filters are trying to do is detect whether an email is written in 'normal English' or 'spam English'. That language identification requirement is exactly what this this text-based language identifier is trying to work out.

So, can anyone tell me why an n-gram (where 2 <= n <= about 4-5) based system, working on the word level, hasn't been tried on this spam/non-spam classification problem? After all, n-gram statistics are used by virtually every single speech recognition software package out there. That's because n-grams are what let the speech recogniser determine which word has a reasonable chance of following the word it thinks you just said, and this information is used to improve the likelihood of what you said being a 'valid' sentence.

There are problems with this, of course: recognising 'spam English' as opposed to 'proper English' is much more difficult than recognising English as opposed to, say, French. But there are subtleties of vocabulary ('free' can often be followed by 'porn', for instance), that may make this a workable method.

Well, those are my thoughts anyway. Someone's probably tried this already and will tell me why it doesn't work as well as other methods, but even in that case I really would be interested to know why.

[edited for formatting]

For my own benefit:

• Links to Lisp sites
• The current 'official' Portaloo

thomasvs wrote:

...when I look at scripts I wrote five years ago, or code for my thesis, I'm ashamed ;) So the big question is : What will I think five years from now about stuff I did today ?

I could've written that, but, being me, I didn't. It's an incredibly sobering thought though. The mantra of 'Do stuff. Repeat.', and the goal of continuous improvement, is still as pertinent now as it ever was. Maybe programmers, localisers and other free software advocates really are made, and not born.

Festival is great. Not only it is GPLed ('course), but it's also truly multi-lingual, thanks to a Welsh speech researcher (Briony Williams). Unfortunately, the letter-to-sound rules for Welsh are now written in Lisp. Goal, then, is to attempt to gain a passing acquaintance with Lisp. Never thought I'd see myself typing that. Mind, never thought I'd have to use Fortran either; that's another story.

Oops. Created an account as Rhys a while back, but completely forgot the password. Decided that a new account was the better option (sorry to those who've already certified me once...)

This week, I have been mostly looking for Welsh-speaking open source advocates. In my view, and that of a few others, GNU/Linux Must Be Translated. Mail me if you fit that bill. Diolch yn dalpau.