Older blog entries for Stevey (starting at number 639)

Random hacking for fun

Recently I've been playing gtetrinet, against the publicly accessible server at tetrinet.debian.net.

If you're unfamiliar with the game it is a multi-player variant of Tetris. You clear many lines and your opponents suffer. Want to make them suffer some more? Use the special blocks you acquire.

Special blocks? How about shuffling your opponents playing field? Adding new semi-formed rows? etc. All good stuff.

There is support for up to six players. To fire a special block at the player in field 1 you press "1". To fire the special block to the player in field 6 press "6". But to fire a block at yourself, to clear your playing field ("nuke") or remove a single line ("clear") you have to know what player-number you are, which will change from day to day, as it is literally a marker for the order you joined the channel in.

It seems obvious that there should be a special-case keybinding "fire to self", and indeed there was bug #291844 filed in 2005 saying as much. I've just submitted a functional patch to resolve this, and already my playing is getting better.

Join me sometime.

Syndicated 2013-09-26 19:31:02 from Steve Kemp's Blog

Some days you just want to do nothing

Today I finally pushed out a new binary release of my slaughter server-automation tool. (Think "CFEngine-lite", with perl. full documetnation is available. Though nobody ever reads it.)

Otherwise the weekend is being quiet; we spent last night mostly drinking vodka, until midnight rolled over, and along with some messing around with a camera ("Wow, your arms are getting bigger!")

Today has consisted of a Turkish breakfast, an Indonesian dinner, and an ice-cream based tea.

I could write more, but I'm hung-over. A rare thing for me.

Syndicated 2013-09-22 14:06:32 from Steve Kemp's Blog

A new wordpress plugin

There is now a new wordpress plugin, for testing against my blogspam site/service.

Now time to talk about something else.

This week my partners sister & niece are visiting from Helsinki, so we've both got a few days off work, and we'll be acting like tourists.

Otherwise the job of this week is to find a local photographer to shoot the pair of us. I've shot her many, many, many times, and we also have many nice pictures of me but we have practically zero photos of the pair of us.

I spent a lot of time talking to local volunteers & models, because I like to shoot them, but I know only a couple of photographers.

Still a big city, we're bound to find somebody suitable :)

Syndicated 2013-09-17 08:47:32 from Steve Kemp's Blog

CIDR-matching, in node.js

I recently mentioned that there wasn't any built-in node.js functionality to perform IP matching against CIDR ranges.

This surprised me, given that lots of other functionality is available by default.

As a learning experience I've hacked a simple cidr-matching module, and published it as an NPM module.

I've written a few other javascript "libraries", but this is the first time I've published a module. Happy times.

The NPM documentation was pretty easy to follow:

  • Write a package.json file.
  • Run "npm publish".
  • Wait for time to pass, and Thorin to sit down and sing about gold.

Now I can take a rest, and stop talking about blog-spam.

Syndicated 2013-09-14 12:04:00 from Steve Kemp's Blog

The blogspam code is live.

Living dangerously I switched DNS to point to the new codebase on my lunch hour.

I found some problems immediately; but nothing terribly severe. Certainly nothing that didn't wait until I'd finished work to attend to.

I've spent an hour or so documenting the new API this evening, and now I'm just going to keep an eye on things over the next few days.

The code is faster, definitely. The load is significantly lower than it would have been under the old codebase - although it isn't a fair comparison:

  • I'm using redis to store IP-blacklists, which expire after 48 hours. Not the filesystem.
  • The plugins are nice and asynchronous now.
  • I've not yet coded a "bayasian filter", but looking at the user-supplied options that's the plugin that everybody seems to want to disable. So I'm in no rush.

The old XML-RPC API is still present, but now it just proxies to the JSON-version, which is a cute hack. How long it stays alive is an open question, but at least a year I guess.

God knows what my wordpress developer details are. I suspect its not worth my updating the wordpress plugin, since nobody ever seemed to love it.

These days the consumers of the API seem to be, in rough order of popularity:

  • Drupal.
  • ikiwiki.
  • Trac

There are few toy-users, like my own blog, and a few other similar small blogs. All told since lunchtime I've had hits from 189 distinct sources, the majority of which don't identify themselves. (Tempted to not process their requests in the future, but I don't think I can make such a change now without pissing off the world. Oops.)

PS. Those ~200 users? rejected 12,000 spam comments since this afternoon. That's cool, huh?

Syndicated 2013-09-12 20:09:24 from Steve Kemp's Blog

I've always relied upon the kindness of strangers

Many thanks to Vincent Meurisse who solved my node.js callback woe.

Some history of the blogspam service:

Back in 2008 I was annoyed by the many spam-comments that were being submitted to my Debian Administration website. I added some simple anti-spam measures, which reduced the flow, but it was a losing battle.

In the end I decided I should test comments, as the users submitted them, via some kind of external service. The intention being that any improvements to that central service would benefit all users. (So I could move to testing comments on my personal blog too, for example).

Ultimately I registered the domain-name "blogspam.net", and set up a simple service on it which would test comments and judge them to be "SPAM" or "OK".

The current statistics show that this service has stopped 20 million spam comments, since then. (We have to pretend I didn't wipe the counters once or twice.)

I've spent a while now re-implementing most of the old plugins in node.js, and I think I'll be ready to deploy the new service over the weekend. The new service will have to handle two different kinds of requests:

New Requests

These will be submitted via HTTP POSTed JSON data, and will be handled by node.js. These should be nice and fast.

Legacy Requests

These will come in via XML-RPC, and be proxied through the new node.js implementation. Hopefully this will mean existing clients won't even notice the transition.

I've not yet deployed the new code, but it is just a matter of time. Hopefully being node.js based and significantly easier to install, update, and tweak, I'll get more contributions too. The dependencies are happily very minimal:

  • A redis-server for maintaining state:
    • The number of SPAM/OK comments for each submitting site.
    • An auto-expiring cache of blacklisted IP adddresses. (I cache the results of various RBL results for 48 hours).
  • node.js

The only significant outstanding issue is that I need to pick a node.js library for performing CIDR lookups - "Does 10.11.12.23 lie within 10.11.12.0/24?" - I'm surprised that functionality isn't available out of the box, but it is the only omission I've missed.

I've been keeping load & RAM graphs, so it will be interesting to see how the node.js service competes. I expect that if clients were using it, in preference to the XML-RPC version, then I'd get a hell of a lot more throughput, but with it hidden behind the XML-RPC proxy I'm less sure what will happen.

I guess I also need to write documentation for the new/preferred JSON-based API...

https://github.com/skx/blogspam.js

Syndicated 2013-09-11 18:26:46 from Steve Kemp's Blog

node.js is kicking me

Today I started hacking on a re-implementation of my BlogSpam service - which tests that incoming comments are SPAM/HAM - in node.js (blogspam.js)

The current API uses XML::RPC and a perl server, along with a list of plugins, to do the work.

Having had some fun and success with the HTTP+JSON mstore toy I figured I'd have a stab at making BlogSpam more modern:

  • Receive a JSON body via HTTP-POST.
  • Deserialize it.
  • Run the body through a series of Javascript plugins.
  • Return the result back to the caller via HTTP status-code + text.

In theory this is easy, I've hacked up a couple of plugins, and a Perl client to make a submission. But sadly the async-stuff is causing me .. pain.

This is my current status:

shelob ~/git/blogspam.js $ node blogspam.js
Loaded plugin: ./plugins/10-example.js
Loaded plugin: ./plugins/20-ip.js
Loaded plugin: ./plugins/80-sfs.js
Loaded plugin: ./plugins/99-last.js
Received submission: {"body":"

This is my body ..

","ip":"109.194.111.184","name":"Steve Kemp"} plugin 10-example.js said next :next plugin 20-ip.js said next :next plugin 99-last.js said spam SPAM: Listed in StopForumSpam.com

So we've loaded plugins, and each has been called. But the end result was "SPAM: Listed .." and yet the caller didn't get that result. Instead the caller go this:

shelob ~/git/blogspam.js $ ./client.pl
200 OK 99-last.js

The specific issue is that I iterate over every loaded-plugin, and wait for them to complete. Because they complete asynchronously the plugin which should be last, and just return "OK" , has executed befure the 80-sfs.js plugin. (Which makes an outgoing HTTP request).

I've looked at async, I've looked at promises, but right now I can't get anything working.

Meh.

Surprise me with a pull request ;)

Syndicated 2013-09-10 17:57:12 from Steve Kemp's Blog

Dynamically discovering settings for a cluster?

Pretend I run a cluster, for hosting a site. Pretend that I have three-six web-nodes, and each one needs to know which database host to contact.

How do I control that?

Right now I have a /etc/settings.conf file, more or less, deployed by Slaughter. That works. Another common pattern is to use a hostname - for example pmaster.example.org.

However failover isn't considered here. If I wanted to update to point to a secondary database I'd need to either:

  • Add code to retry the second host on failure.
    • Worry about divergence if some hosts used DB1, then DB2, then DB1 came back online.
    • Failover is easy. Fail-back is probably best avoided.
  • Worry about DNS caches and TTL.

In short I'm imagining there are several situations where you want to abstract away the configuration in a cluster-wide manner. (A real solution is obviously floating per-service IPs. Via HAProxy, Keepalived, ucarp, etc. People do that quite often for database specifically, but not for redis-servers, etc.)

So I'm pondering what is essentially a multi-cast accessible key-value storage system.

Have a deamon on the VLAN which will respond to multicast questions like "get db", or "get cache", with a hostname/IP/result.

Suddenly your code would read:

  • Send mcast question ("which db?").
  • Get mcast reply ("db1").
  • Connect to db1.

To me that seems like it should be genuinely useful. But I'm unsure if I'm trading one set of problems for another.

I can't find any examples of existing tools/deamons in this area, which either means I'm being novel, innovate, and interesting. Or I'm over thinking...

Syndicated 2013-09-06 08:29:21 from Steve Kemp's Blog

So that forum?

So that forum I mentioned? I've setup a test-installation at:

What does this forum offer? A cross between hacker news and reddit. If the admin of the forums enables it you can create arbitrary tags, and then view them. For example:

  • http://example.com/view/tag-name

It's also very fast, and reasonably easy to customize. Which is good, because the current layout is nasty.

Things I like:

  • Everything is stored in Redis.
  • The code is made of simple primitives which are joined together in a web-application. Which means most of the logic is outside the core.
  • The templates are pretty basic, which means a real designer can do good things.

Not much more to say really; except I've setup a test install and if you wish to login/register and post spam feel free.

Syndicated 2013-09-02 15:11:54 from Steve Kemp's Blog

A shaved head is a sign of a tidy mind?

This week I have mostly:

Knocked up a forum ("gathering")

It is a simple clone of hacker news, storing things in Redis.

None of the aging though, entries are first->last and my tagging support is basic.

I'm in two minds about releasing it. I'm in three minds about deploying it at the location I'd written it for - forums.lumail.org - since there's nothing worse than a forum with no posts, unless it is a forum that used to be popular and is now a wasteland, or circlejerk. (c.f. slashdot. ahem.)

Updated Slaugther

I found a hidden dependency when installing a new slaughter controlled host.

A new release is imminent.

Setup a remote host for backups

Using BigV I configured a system with LUKS-encrypted disks to act as a "remote dropbox" via git-annex, and rsync.

An encrypted volume is manually mounted post-boot. It stays mounted (oops that's bad) but provides security when the guest is offline, or has been retired.

Wrestled with Graphite

Because damn that software is hard to install.

Turned down two weddings

Because I will never shoot a wedding again. And if I did I'd not do it for free for "exposure".

Merged my (digital) music archive with that belonging to my partner

Which took more discussion than a) moving in together, or b) opening a shared bank account.

Secretly decided I do like my kindle

Even though I bought two books from a charity shop tonight I'm almost certainly going to file them away and look for the epub online instead.

He fought with the goblins! He battled the trolls! He riddled with Gollum! The magic ring he stole!

This weekend I'll be mostly offline. Saturating my home broadband while I sync backups.

Syndicated 2013-08-30 19:46:32 from Steve Kemp's Blog

630 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!