Advogato blog for Stevey

Some domains just don't learn

Sun, 5 Feb 2012 13:28:50 GMT

For the past few years the anti-spam system I run has been based on a simplified version of something I previously ran commercially.

Although the code is similar in intent there were both explicit feature removals, and simplifications made.

Last month I re-implimented domain-blacklisting - because a single company keeps ignoring requests to remove me.

So LinkedIn.com if you're reading this:

I've never had an account on your servers.
I find your junk mail annoying.
I suspect I'll join your site/service when hell freezes over.

I've also implemented TLD-blacklisting which has been useful.

TLD-blacklisting in my world is not about blocking mail from foo@bar.ph (whether in the envelope sender, or the from: header), instead it is about matching the reverse DNS of the connecting client.

If I recieve a connection from 1.2.3.4 and the reverse DNS of that IP address matches, say, /\.sa$/i then I default to denying it.

My real list is longer, and handled via files:
steve@steve:~$ ls /srv/_global_/blacklisted/tld/ -C
ar  br  cn  eg  hr  in  kr  lv  mn  np  ph  ro  sg  tg  ua  ve  zw
aw  cc  cy  gm  hu  is  kz  ma  my  nu  pk  rs  sk  th  ug  vn
be  ch  cz  gr  id  it  lk  md  mz  nz  pl  ru  su  tr  uy  ws
bg  cl  ec  hk  il  ke  lt  mk  no  om  pt  sa  sy  tw  uz  za

On average I'm rejecting about 2500 messagse a day at SMTP-time, and 30 messages, or so, hit my SPAM folder after being filtered with CRM114 after being accepted for delivery. (They are largely from @hotmail and @yahoo, along with random compromised machines. The amount of times I see a single mail from a host with RDNS mysql.example.org is staggering.).

(Still looking forward to the development of Haraka, a node.js version of qpsmtpd.)

ObQuote: "Mr. Mystery Guest? Are you still there? " - Die Hard

So mega-upload is gone

Sat, 21 Jan 2012 13:28:39 GMT

So the site http://megaupload.com/ has been taken offline, amidst allegations of knowingly conducting in piracy.

There are probably a lot of legitimate users who have lost access to their uploaded files, even if they were offsite backups you can imagine a user owning a website which now has a million dead-links.

This reminds me of a conversation I overheard on Jon Dowlands blog - the summary is that he'd written a (useful) tool to extract attachments from Maildir folders and was wondering how to store and access those attachments. The upshot seemed to be magical URLs of the form:

https://file.example.com/sha1/509c2fe2eba509e93987c3024a74d74583c274bd

The comments covered an alternative which was hash:///sha1/xxxxxxxxxxxxxxxx, which then becomes close to the magnet:// schema.

I've not yet thought things through, but I can't help thinking that with the redundency already present in the internet we should be looking at non-server-specific links. Yes there are times right now when you might want to address a specific file on a specific server - but otherwise? Wouldn't it be nice if you could just access a file from "anywhere" which happened to have the right contents?

Already my nonporn-but-definitely-adult-site makes its images available as /img/$md5sum.jpg - and similarly the storage at the back-end of my random image upload site uses SHA1 hashes to store the actual files.

To make this more complete what we need is something that crawls the internet to find files by hash; then add support in browsers. Obviously this must be async and could introduce timing issues, but fundamentally it seems like a reasonable approach to the problem of a single host going offline.

(Consider what happens if imgur.com disappears. All those links would die, yet 99% of the images would still be available somewhere.)

I'm tempted to suggest microformat format but I need to consider the matter. Right now I'm going to immediately update my current image hosts to use, at the very least:

 <a href="/foo" rel="sha1:xxxxx md5sum:xxxx">
  <img src="foo.jpg" alt="img name">
 </a>

The unfortunate thing is you cannot have a 'rel="xx"' attribute for an image. So you either have to encode it in the parent link, or add it to the alt attribute which is suboptimal.

ObQuote: "Now, they tell me I paid my debt to society." - Oceans Eleven (2001)

Some misc. updates

Fri, 13 Jan 2012 18:28:23 GMT

Security

Today I made available a 3.2.0 kernel for my KVM guest which has a bastardised version of the PID hiding patch configured:

procfs: add hidepid= and gid= mount options

bugfix - proc: fix null pointer deref in proc_pid_permission()

So now on my guest, as myself, I can only see this:
steve@steve:~$ ls -l /proc/ | egrep ' [0-9]+$'
dr-xr-xr-x  7 steve users          0 Jan 13 17:22 15150
dr-xr-xr-x  7 steve users          0 Jan 13 17:29 15739
dr-xr-xr-x  7 steve users          0 Jan 13 17:29 15740
lrwxrwxrwx  1 root  root          64 Jan 13 17:20 self -> 15739
Running as root I see the full tree:
steve:~#  ls -l /proc/ | egrep ' [0-9]+$'
total 0
dr-xr-xr-x  7 root        root                 0 Jan 13 17:20 1
dr-xr-xr-x  7 root        root                 0 Jan 13 17:20 1052
dr-xr-xr-x  7 root        root                 0 Jan 13 17:20 1086
dr-xr-xr-x  7 root        root                 0 Jan 13 17:20 1101
dr-xr-xr-x  7 root        root                 0 Jan 13 17:20 1104
dr-xr-xr-x  7 root        root                 0 Jan 13 17:21 1331
dr-xr-xr-x  7 pdnsd       proxy                0 Jan 13 17:21 14409
dr-xr-xr-x  7 root        root                 0 Jan 13 17:21 14519
..
This (obviously) affects output from top etc too. It is a neat feature which I think is worth having, but time will tell..

mod_ifier

A long time ago I put together an Apache module which allowed the evaluation of security rules against incoming HTTP requests. mod_ifier was largely ignored by the world. But this week it did receive a little attention.

The recent rash of Hash Collision attacks inspired inspired a fork with parameter filtering. Neat.

Otherwise nothing too much to report - though I guess I didn't actually share the link to the RESTful file store I mentioned previously. Should you care you can find it here:

http://sinatrastore.repository.steve.org.uk/

ObQuote: "I saw a man, he danced with his wife" - Chicago, Frank Sinatra

Review of the Panasonic Lumix FS-16 camera

Sat, 7 Jan 2012 15:28:53 GMT

Recently I've been wanting to replace my old point and shoot camera, a Canon PowerShot A620. I've got a pair of DLSR cameras and I do frequently carry one of them out with me, but there are undoubtedly occasions where I'd rather not bother, or where I find myself wanting to take a picture without having one to hand.

Unfortunately the PowerShot is pretty large itself, although significantly less so than the DSLRS I possess. (I cannot remember the last time I used the PowerShot outside my flat, that is how rarely it goes outdoors).

The PowerShot has been a good camera to me for many years and the three features I liked the most:

A real view-finder.
It runs on 4x AA batteries; easy to find.
Shoots (smallish) movies.
- This last thing is no longer a concern now I have a Kodak Playsport HD video toy.

Picking a replacement camera, even with the help of fine comparison websites like snapsort.com is hard. Cameras have moved on and "improved" a lot over the last few years - to the extent that finding one with a built-in viewfinder is hard. Finding one with a built-in viewfinder and running on easily replaceable batteries was virtually impossible.

Eventually I settled on the Panasonic Lumix FS16, which omits both:

Integrated rechargeable battery.
LCD-only viewfinder.

The way that you use the LCD or viewfinder differs pretty significantly, but the LCD wasn't as bad as I'd feared:

ViewFinder

You hold the camera to your eye, and press the appropriate buttons.

LCD

You typically hold the camera at arms length, which means you're prone to shaking your hands/arms and getting blurry shots.

Because you're holding the camera relatively far away from your eyes if you have the sun at your back you're liable to need to squint.

The LCD on the Lumix FS-16 isn't amazing, but neither is it horrific and it is better than expected in dark locations.

So after a week what do I think? On the whole it is a fine camera, better than the PowerShot in many ways, and while it has draw-backs none are deal-breakers:

Size

The best camera is one you have with you; on that basis this camera is a clear win being smaller, lighter, and more compact than the Canon.

I've taken this camera with me, randomly, to several places and returned with useful and interesting images.

Low Light

Low light performance is pretty poor. With only one manual control you see noise if you're shooting in gloomy pubs, and outdoors. With the flash you can get acceptable pictures if you're careful - but its a tricky thing to get right.

(To update this a little: Outdoors at night? No. In a pub with poor lighting you'll be alright.)

Manual Controls

The camera features precisely two manual controls:

"Flash on" vs. "Flash off".
ISO can be changed from: Auto, 100, 200, 400, 800, 1200 and 1600.

There is no notion of shutter speed, nor is there any ability to change the aperture size. (Though both these values are displayed on the screen as you take a picture I wonder why? As you can't change anything you can't use the information in any useful fashion, and presumably a non-camera-person wouldn't understand what these numbers represent.)

The lack of these two controls is a little galling, but pretty common for the low-end P&S cameras.

Video Recording

There is no external MIC so sounds aren't great, but they're not horrible either.

Video recordings are limited to the smaller of 8 minutes or 2Gb. So no long films, but short ones look fine. Just be aware that once you start recording focus won't change, nor will zooming work.

Compared to the canon the quality is improved; but the Canon allowed you to (optically) zoom whilst recording. Here you can only zoom with your feet.

Recharging Time

When I received the camera it took about an hour to charge. The battery life seems reasonable - the level is 2/3 a week later and I've been shooting, reviewing, and deleting regularly.

(Note: I never use USB to transfer pictures, I always remove the card and plug it into my PC. Whether this makes a difference to battery life I don't know.)

Controls

Physical controls are reasonable. There is a mechanical slide-switch to turn on/off. I like that, as it is less prone to being knocked by keys, change, etc.

There is also a physical slide-switch to change from "shoot" to "review current images/videos". (Same as my Canon) I think this is a mistake, and don't see why it can't be a soft-button.

Full Auto

There are several modes available in the camera (remember the caveat about lack of aperture/shutter speed) I've been using both full-auto and manual modes, and both are good. Full auto would suit most people - it has clever face-tracking.

Focusing Speed

As expected this is not stellar. Walking to the corner shop the other lunchtime I found a cat in the road, I talked to her and she rubbed herself against my ankles. Could I focus fast enough to catch her looking up at me? No.

For static scenes, and candid shots of people it'll suffice. For fast action and moving children probably not a chance.

On balance, the upgrade was worthwhile.

ObQuote: "I don't mean to lecture and I don't mean to preach. And I know I'm not your father..." - Spider-Man

Tonight I've mostly been using Sinatra

Fri, 6 Jan 2012 22:26:57 GMT

This evening I've mostly been using Sinatra to build a little file storage service which uses a REST API.

That means I can upload a file:

skx@birthday:~/hg/sinatra$ curl -X PUT -F file=@/etc/fstab http://localhost:4567/
{"id":"dbd1bdc11b5a1a8e80588a135648b4c2edffb49a","path":"/"}

Download that same file:

skx@birthday:~/hg/sinatra$ curl -X GET -F id=dbd1bdc11b5a1a8e80588a135648b4c2edffb49a  \
   http://localhost:4567/
# /etc/fstab: static file system information.
..
/dev/cdrom        /media/cdrom0   udf,iso9660 user,noauto     0       0

Get an index of files:

skx@birthday:~/hg/sinatra$ curl http://localhost:4567/
[{"id":"dbd1bdc11b5a1a8e80588a135648b4c2edffb49a","type":"file"}]

And finally we can delete a file:

skx@birthday:~/hg/sinatra$ curl -X DELETE -F "id=dbd1bdc11b5a1a8e80588a135648b4c2edffb49a" \
  http://localhost:4567/
Removed

We can also upload to different paths so we can replicate a file-system if we wanted to. (I added in "type" to hold either "file" or "directory", though I guess if we were to code up a FUSE client we'd want to store things like ctime, UID, GID, etc. THe list operation will show both files and sub-directories)

The code was trivial once I got the hang of Sinatra, and I'm pretty pleased with it so far. I don't yet need to use it for anything, but I'm thinking of unifying the way that I store images on a couple of sites - and fetching them via JSON and Javascript might be an option this was an experiment in that direction. (Though I'd probably want to hook in rsync so we replicated the eventual upload location for safety.)

In other news I've been all organized and upgraded the kernel on my guest:

steve@steve:~$ uptime
 22:00:28 up  4:18,  1 user,  load average: 0.14, 0.05, 0.05
steve@steve:~$ uname -r
3.2.0-kvm-hosting.org-i386-20120106

So for once I'm up to date with a cutting edge kernel. Happy times.

ObQuote: "How you expect to run with the wolves come night when you spend all day sparring with the puppies? " - The Wire (Omar)

The final updates of 2011

Sat, 31 Dec 2011 21:27:53 GMT

I've been informed by a couple of people that the Debian Administration site is down. Sadly it is; at the moment the host isn't showing anything on the serial console and remotely power-cycling it isn't showing any signs of life.

At this time of year I don't want to drag anybody in to take care of it, so ETA on recovery/replacement hardware is Monday/Tuesday.

In other news I've made it to year five of the KVM hosting sub-project/thing. Originally started as a Xen host its been running happily for quite some time. I suspect next year, or the year after that the price/specification ratio will end up losing out and we'll cancel the whole thing - but there are no immediate reasons to make any change.

Finally I knocked up a simple tool to validate my TinyDNS records prior to uploading them. It is simplistic, but adequate to catch the kind of mistakes I make:

http://tinydns-lint.repository.steve.org.uk/file

Honestly it probably wants to be rationalised a little more - and check records more carefully. e.g. Ensure that the host a CNAME refers to itself exists, and making sure that the nameservers specified are valid.

I just wanted to make something quick after accidentally uploading a zonefile where I'd managed to fat-finger several important records. le sigh.

Oddly enough asking on serverfault.com showed no real suggestions - other than actually running tinydns locally and doing a zone-xfer to validate records. Overkill and harder than I'd like.

Happy New year if you care about such things..

"I finished growing up, Léon. I just get older. " - Leon

So I'm a peddler of smut nowadays.

Mon, 26 Dec 2011 18:28:05 GMT

Over the Christmas period I've not been doing too much.

December was a month that started out pretty well, in the first 10 days I had five models/friends/random-folk come to pose for me. (I have a few folk lined up for mid-January, but things tailed off quickly this month due to people having little free time).

I spent a while mulling over what I was doing with images, as I've recently been doing a fair number of more NSFW images - In December this and this were my two favourite shots.

I've posted NSFW images in various places over the past year (with permission; some volunteers/victims/friends don't want me to share anything, so I don't. They just hang on my walls.) but I was never consistent.

Anyway I realised that .xxx domains are now available, so I figured I'd snarf one up and use that. That lead to tasteful.xxx - which is mostly full of images I didn't actually take, but that will change. (Sadly "artistic" was gone!)

In more on-topic news I reported #651896 - a trivial security issue in another setgid(games) binary. I've got a couple more of those to tidy up and report in the near future.

ObQuote: "Bright light. Bright light." - Gremlins

So I removed some more software from my host

Sat, 3 Dec 2011 17:28:08 GMT

Today I was idly performing some maintainence upon one of my hosts, and it crossed my mind to look beneath /etc in there I found:

/etc/python
/etc/python2.4
/etc/python2.5
/etc/python2.6

That made me look more closely at the contents of /etc - the following command output was surprising:

steve@steve:~$ ls /etc | wc -l
187

Is that average? Heavy? Light? I have no idea, but I purged a hell of a lot of software today. Now I have only python v2.6 although for some reason I still have:

python
python-apt
python-apt-common
python-central
python-minimal
python-support
python2.6
python2.6-minimal

I suspect I could drop the pything2.6-minimal package, but for the moment I'm done. I have to make pretty people look exceptional with my magical camera.

Anyway as part of this cleanup I ran a quick sanity-check on which processes are running and I think, short of kernel processes, I'm as minimal as I can be. I understand the purpose and reason for every running service:

UID        PID  CMD
root         1  init [2]
pdnsd    14091  /usr/sbin/pdnsd --daemon -p /var/run/pdnsd.pid
root     14199  /usr/sbin/monit -c /etc/monit/monitrc -s /var/lib/monit/monit.state
root     14206  /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid
root     14234  /usr/sbin/cron
102      14595  /usr/sbin/exim4 -bd -q30m
redis    14627  /usr/bin/redis-server /etc/redis/redis.conf
root     14637  /usr/sbin/sshd

These are basic services; I use monit to ensure those essential daemons keep running. The only oddity there is probably the local DNS cache, but it is useful if you run any kind of DNS blacklist-using service, for example.

root     14794  /sbin/getty -L ttyS0 9600 vt100

I need a serial console login for emergencies.

root     14796  runsv node-reverse-proxy
root     14797  /bin/sh ./run
root     14799  /opt/node/bin/node node-reverse-proxy.js --config ./rewrites.js

These three processes combine to run my reverse proxy which routes incoming HTTP requests to a number of local thttpd instances.

qpsmtpd  27309    /usr/bin/perl -Tw /usr/bin/qpsmtpd-prefork --port 25 --user qpsmtpd --pid-file /var/run/qpsmtpd/qpsmtpd.pid --detach
..

The perl SMTP daemon which runs my incoming mail, passing it to exim4 which listens upon 127.0.0.1:2525. You can read about my setup in the out-of-date writeup Chris & I put together.

 /usr/bin/memcached -m 64 -p 11211 -u root -l 127.0.0.1

Memory cache for transient items.

s-blog    thttpd -C /etc/thttpd/sites.enabled/blog.steve.org.uk
1030      thttpd -C /etc/thttpd/sites.enabled/edinburgh-portraits.com
s-hg      thttpd -C /etc/thttpd/sites.enabled/hg.steve.org.uk
s-ipv4    thttpd -C /etc/thttpd/sites.enabled/ipv4.steve.org.uk
s-ipv6    thttpd -C /etc/thttpd/sites.enabled/ipv6.steve.org.uk
s-kvm     thttpd -C /etc/thttpd/sites.enabled/kvm-hosting.org
...

One thttpd instance is launched for each distinct HTTP site my server runs. Each site runs under its own UID, with its own chrooted directory tree. This is important for security.

Each local instance listens upon 127.0.0.1 - and the reverse proxy previously mentioned rewrites connections to the appropriate one.

1016     28812     /usr/bin/perl -I./lib/ -I./ /usr/local/bin/blogspam

My anti-spam filter for blog comments.

Here is my christmas challenge. Can you identify each service upon your host? Do you know why you're running what you're running?

Me? I had no idea I had a dbus deamon running. Now I've purged it. Ha!

ObQuote - "I owe everything to George Bailey. Help him, dear Father." - It's a wonderful life.

Slaughter now tested upon Microsoft Windows hosts.

Sat, 26 Nov 2011 16:06:25 GMT

Recently I said that my perl-based sysadmin tool, Slaughter, was at the cross-roads. I wasn't sure if I should leave it alone, or update it somehow.

As I'm generally lazy and busy (yes it is possible to be both simultaneously!) I didn't do anything.

But happily earlier in the week I received a bunch of updates from Jean Baptiste which implemented support for managing Windows clients, via Strawberry Perl.

So I guess the conclusion is: Do nothing. Change nothing. Just fix any issues which are reported to me, and leave it as-is. (I did a little more than that, refactoring to avoid duplication and improve "neatness".)

As I said at the time I've had some interesting feedback, suggestions and bugfixes from people over the past year or so - so I shouldn't be surprised to learn I'm not the only person using it.

ObQuote: "Oh, yes, a big cat! My salvation depends upon it! " - Dracula (1992)

Goodbye mysql ..

Tue, 22 Nov 2011 20:06:14 GMT

Yesterday evening I updated my server to remove MySQL:

steve:~# dpkg --purge mysql-client-5.1 \
                      mysql-common     \
                      mysql-server-5.1 \
                      mysql-server-core-5.1 \
                      python-mysqldb        \
                      libdbd-mysql-perl     \
                      libdatetime-format-mysql-perl

Until last month I had two database in use, one each for a pair of web-applications. As of now one is using redis - which I'm already using for my image hosting - and the other application is using SQLite.

Until recently I had a high opinion of SQLite, although that has now been downgraded a little, it is still a thoroughly excellent piece of software. I was just surprised at little things it was missing, to the extent I had to rewrite my applications SQL.

Still one less service is a good thing, and the migration wasn't so painful..

In more productive news I recently acquired a nice external flash - the Yongnuo YN-460 II is (very) cheap and cheerful, it can be fired remotely with my triggers so I've had a lot of fun with opportunistically taking pictures and experimenting with lighting.

Most of the results are NSFW, but there are some other examples lurking around including the first time I managed to successfully capture a falling water-drop. (Not the best picture, not the most explicit effect, but fun regardless. I both can and will do better next time!)

Somebody recently asked me to write about "camera stuff under linux" and happily I declined.

Why decline? Because there are so many good tools, applications, and utilities. (I use local tools for organisation and duplicate detection, rawtherapee for RAW conversion and GIMP for touchups). Having many available options is fantastic though, and something hard to appreciate for "newcomers" to Linux.

(Yeah I waited 90 seconds - if I remembered to add -nojava - for Netscape Navigator to start, under X10, with 8Mb of RAM. Happier days are here. Sure DRM is bad, secure boot .. an open question, but damn we have it good compared to almost any previous point in time!)

ObQuote: "Yeah, obviously it is only a tactical party. I'm only having a party to eventually get sex." - Peep Show