Older blog entries for RhysJones (starting at number 4)

24 Aug 2002 (updated 24 Aug 2002 at 16:30 UTC) »
He who obeys, obeys Bayes?

Yup, another set of possibly ill-conceived thoughts on spam filtering. There was a good round-up of what's currently going on in yesterday's NTK, incidentally. (Search for Tracking in that page).

What the current crop of Bayesian spam filters are trying to do is detect whether an email is written in 'normal English' or 'spam English'. That language identification requirement is exactly what this this text-based language identifier is trying to work out.

So, can anyone tell me why an n-gram (where 2 <= n <= about 4-5) based system, working on the word level, hasn't been tried on this spam/non-spam classification problem? After all, n-gram statistics are used by virtually every single speech recognition software package out there. That's because n-grams are what let the speech recogniser determine which word has a reasonable chance of following the word it thinks you just said, and this information is used to improve the likelihood of what you said being a 'valid' sentence.

There are problems with this, of course: recognising 'spam English' as opposed to 'proper English' is much more difficult than recognising English as opposed to, say, French. But there are subtleties of vocabulary ('free' can often be followed by 'porn', for instance), that may make this a workable method.

Well, those are my thoughts anyway. Someone's probably tried this already and will tell me why it doesn't work as well as other methods, but even in that case I really would be interested to know why.

[edited for formatting]

23 Aug 2002 (updated 23 Aug 2002 at 09:07 UTC) »
thomasvs wrote:

...when I look at scripts I wrote five years ago, or code for my thesis, I'm ashamed ;) So the big question is : What will I think five years from now about stuff I did today ?

I could've written that, but, being me, I didn't. It's an incredibly sobering thought though. The mantra of 'Do stuff. Repeat.', and the goal of continuous improvement, is still as pertinent now as it ever was. Maybe programmers, localisers and other free software advocates really are made, and not born.

Festival is great. Not only it is GPLed ('course), but it's also truly multi-lingual, thanks to a Welsh speech researcher (Briony Williams). Unfortunately, the letter-to-sound rules for Welsh are now written in Lisp. Goal, then, is to attempt to gain a passing acquaintance with Lisp. Never thought I'd see myself typing that. Mind, never thought I'd have to use Fortran either; that's another story.
15 Aug 2002 (updated 15 Aug 2002 at 19:55 UTC) »

Oops. Created an account as Rhys a while back, but completely forgot the password. Decided that a new account was the better option (sorry to those who've already certified me once...)

This week, I have been mostly looking for Welsh-speaking open source advocates. In my view, and that of a few others, GNU/Linux Must Be Translated. Mail me if you fit that bill. Diolch yn dalpau.

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!