So mega-upload is gone

So the site http://megaupload.com/ has been taken offline, amidst allegations of knowingly conducting in piracy.

There are probably a lot of legitimate users who have lost access to their uploaded files, even if they were offsite backups you can imagine a user owning a website which now has a million dead-links.

This reminds me of a conversation I overheard on Jon Dowlands blog - the summary is that he'd written a (useful) tool to extract attachments from Maildir folders and was wondering how to store and access those attachments. The upshot seemed to be magical URLs of the form:

• https://file.example.com/sha1/509c2fe2eba509e93987c3024a74d74583c274bd

The comments covered an alternative which was hash:///sha1/xxxxxxxxxxxxxxxx, which then becomes close to the magnet:// schema.

I've not yet thought things through, but I can't help thinking that with the redundency already present in the internet we should be looking at non-server-specific links. Yes there are times right now when you might want to address a specific file on a specific server - but otherwise? Wouldn't it be nice if you could just access a file from "anywhere" which happened to have the right contents?

Already my nonporn-but-definitely-adult-site makes its images available as /img/$md5sum.jpg - and similarly the storage at the back-end of my random image upload site uses SHA1 hashes to store the actual files.

To make this more complete what we need is something that crawls the internet to find files by hash; then add support in browsers. Obviously this must be async and could introduce timing issues, but fundamentally it seems like a reasonable approach to the problem of a single host going offline.

(Consider what happens if imgur.com disappears. All those links would die, yet 99% of the images would still be available somewhere.)

I'm tempted to suggest microformat format but I need to consider the matter. Right now I'm going to immediately update my current image hosts to use, at the very least:

 <a href="/foo" rel="sha1:xxxxx md5sum:xxxx">
  <img src="foo.jpg" alt="img name">
 </a>

The unfortunate thing is you cannot have a 'rel="xx"' attribute for an image. So you either have to encode it in the parent link, or add it to the alt attribute which is suboptimal.

ObQuote: "Now, they tell me I paid my debt to society." - Oceans Eleven (2001)

Syndicated 2012-01-21 12:42:37 from Steve Kemp's Blog

Some misc. updates

Security

Today I made available a 3.2.0 kernel for my KVM guest which has a bastardised version of the PID hiding patch configured:

• procfs: add hidepid= and gid= mount options
• bugfix - proc: fix null pointer deref in proc_pid_permission()

So now on my guest, as myself, I can only see this:

steve@steve:~$ ls -l /proc/ | egrep ' [0-9]+$'
dr-xr-xr-x  7 steve users          0 Jan 13 17:22 15150
dr-xr-xr-x  7 steve users          0 Jan 13 17:29 15739
dr-xr-xr-x  7 steve users          0 Jan 13 17:29 15740
lrwxrwxrwx  1 root  root          64 Jan 13 17:20 self -> 15739

Running as root I see the full tree:

steve:~#  ls -l /proc/ | egrep ' [0-9]+$'
total 0
dr-xr-xr-x  7 root        root                 0 Jan 13 17:20 1
dr-xr-xr-x  7 root        root                 0 Jan 13 17:20 1052
dr-xr-xr-x  7 root        root                 0 Jan 13 17:20 1086
dr-xr-xr-x  7 root        root                 0 Jan 13 17:20 1101
dr-xr-xr-x  7 root        root                 0 Jan 13 17:20 1104
dr-xr-xr-x  7 root        root                 0 Jan 13 17:21 1331
dr-xr-xr-x  7 pdnsd       proxy                0 Jan 13 17:21 14409
dr-xr-xr-x  7 root        root                 0 Jan 13 17:21 14519
..

This (obviously) affects output from top etc too. It is a neat feature which I think is worth having, but time will tell..

mod_ifier

A long time ago I put together an Apache module which allowed the evaluation of security rules against incoming HTTP requests. mod_ifier was largely ignored by the world. But this week it did receive a little attention.

The recent rash of Hash Collision attacks inspired inspired a fork with parameter filtering. Neat.

Otherwise nothing too much to report - though I guess I didn't actually share the link to the RESTful file store I mentioned previously. Should you care you can find it here:

• http://sinatrastore.repository.steve.org.uk/

ObQuote: "I saw a man, he danced with his wife" - Chicago, Frank Sinatra

Syndicated 2012-01-13 17:33:46 from Steve Kemp's Blog

Review of the Panasonic Lumix FS-16 camera

Recently I've been wanting to replace my old point and shoot camera, a Canon PowerShot A620. I've got a pair of DLSR cameras and I do frequently carry one of them out with me, but there are undoubtedly occasions where I'd rather not bother, or where I find myself wanting to take a picture without having one to hand.

Unfortunately the PowerShot is pretty large itself, although significantly less so than the DSLRS I possess. (I cannot remember the last time I used the PowerShot outside my flat, that is how rarely it goes outdoors).

The PowerShot has been a good camera to me for many years and the three features I liked the most:

• A real view-finder.
• It runs on 4x AA batteries; easy to find.
• Shoots (smallish) movies.
  • This last thing is no longer a concern now I have a Kodak Playsport HD video toy.

Picking a replacement camera, even with the help of fine comparison websites like snapsort.com is hard. Cameras have moved on and "improved" a lot over the last few years - to the extent that finding one with a built-in viewfinder is hard. Finding one with a built-in viewfinder and running on easily replaceable batteries was virtually impossible.

Eventually I settled on the Panasonic Lumix FS16, which omits both:

• Integrated rechargeable battery.
• LCD-only viewfinder.

The way that you use the LCD or viewfinder differs pretty significantly, but the LCD wasn't as bad as I'd feared:

ViewFinder

You hold the camera to your eye, and press the appropriate buttons.

LCD

You typically hold the camera at arms length, which means you're prone to shaking your hands/arms and getting blurry shots.

Because you're holding the camera relatively far away from your eyes if you have the sun at your back you're liable to need to squint.

The LCD on the Lumix FS-16 isn't amazing, but neither is it horrific and it is better than expected in dark locations.

So after a week what do I think? On the whole it is a fine camera, better than the PowerShot in many ways, and while it has draw-backs none are deal-breakers:

Size

The best camera is one you have with you; on that basis this camera is a clear win being smaller, lighter, and more compact than the Canon.

I've taken this camera with me, randomly, to several places and returned with useful and interesting images.

Low Light

Low light performance is pretty poor. With only one manual control you see noise if you're shooting in gloomy pubs, and outdoors. With the flash you can get acceptable pictures if you're careful - but its a tricky thing to get right.

(To update this a little: Outdoors at night? No. In a pub with poor lighting you'll be alright.)

Manual Controls

The camera features precisely two manual controls:

• "Flash on" vs. "Flash off".
• ISO can be changed from: Auto, 100, 200, 400, 800, 1200 and 1600.

There is no notion of shutter speed, nor is there any ability to change the aperture size. (Though both these values are displayed on the screen as you take a picture I wonder why? As you can't change anything you can't use the information in any useful fashion, and presumably a non-camera-person wouldn't understand what these numbers represent.)

The lack of these two controls is a little galling, but pretty common for the low-end P&S cameras.

Video Recording

There is no external MIC so sounds aren't great, but they're not horrible either.

Video recordings are limited to the smaller of 8 minutes or 2Gb. So no long films, but short ones look fine. Just be aware that once you start recording focus won't change, nor will zooming work.

Compared to the canon the quality is improved; but the Canon allowed you to (optically) zoom whilst recording. Here you can only zoom with your feet.

Recharging Time

When I received the camera it took about an hour to charge. The battery life seems reasonable - the level is 2/3 a week later and I've been shooting, reviewing, and deleting regularly.

(Note: I never use USB to transfer pictures, I always remove the card and plug it into my PC. Whether this makes a difference to battery life I don't know.)

Controls

Physical controls are reasonable. There is a mechanical slide-switch to turn on/off. I like that, as it is less prone to being knocked by keys, change, etc.

There is also a physical slide-switch to change from "shoot" to "review current images/videos". (Same as my Canon) I think this is a mistake, and don't see why it can't be a soft-button.

Full Auto

There are several modes available in the camera (remember the caveat about lack of aperture/shutter speed) I've been using both full-auto and manual modes, and both are good. Full auto would suit most people - it has clever face-tracking.

Focusing Speed

As expected this is not stellar. Walking to the corner shop the other lunchtime I found a cat in the road, I talked to her and she rubbed herself against my ankles. Could I focus fast enough to catch her looking up at me? No.

For static scenes, and candid shots of people it'll suffice. For fast action and moving children probably not a chance.

On balance, the upgrade was worthwhile.

ObQuote: "I don't mean to lecture and I don't mean to preach. And I know I'm not your father..." - Spider-Man

Syndicated 2012-01-07 15:08:49 from Steve Kemp's Blog

Tonight I've mostly been using Sinatra

This evening I've mostly been using Sinatra to build a little file storage service which uses a REST API.

That means I can upload a file:

skx@birthday:~/hg/sinatra$ curl -X PUT -F file=@/etc/fstab http://localhost:4567/
{"id":"dbd1bdc11b5a1a8e80588a135648b4c2edffb49a","path":"/"}

Download that same file:

skx@birthday:~/hg/sinatra$ curl -X GET -F id=dbd1bdc11b5a1a8e80588a135648b4c2edffb49a  \
   http://localhost:4567/
# /etc/fstab: static file system information.
..
/dev/cdrom        /media/cdrom0   udf,iso9660 user,noauto     0       0

Get an index of files:

skx@birthday:~/hg/sinatra$ curl http://localhost:4567/
[{"id":"dbd1bdc11b5a1a8e80588a135648b4c2edffb49a","type":"file"}]

And finally we can delete a file:

skx@birthday:~/hg/sinatra$ curl -X DELETE -F "id=dbd1bdc11b5a1a8e80588a135648b4c2edffb49a" \
  http://localhost:4567/
Removed

We can also upload to different paths so we can replicate a file-system if we wanted to. (I added in "type" to hold either "file" or "directory", though I guess if we were to code up a FUSE client we'd want to store things like ctime, UID, GID, etc. THe list operation will show both files and sub-directories)

The code was trivial once I got the hang of Sinatra, and I'm pretty pleased with it so far. I don't yet need to use it for anything, but I'm thinking of unifying the way that I store images on a couple of sites - and fetching them via JSON and Javascript might be an option this was an experiment in that direction. (Though I'd probably want to hook in rsync so we replicated the eventual upload location for safety.)

In other news I've been all organized and upgraded the kernel on my guest:

steve@steve:~$ uptime
 22:00:28 up  4:18,  1 user,  load average: 0.14, 0.05, 0.05
steve@steve:~$ uname -r
3.2.0-kvm-hosting.org-i386-20120106

So for once I'm up to date with a cutting edge kernel. Happy times.

ObQuote: "How you expect to run with the wolves come night when you spend all day sparring with the puppies? " - The Wire (Omar)

Syndicated 2012-01-06 22:04:06 (Updated 2012-01-07 00:27:37) from Steve Kemp's Blog

The final updates of 2011

I've been informed by a couple of people that the Debian Administration site is down. Sadly it is; at the moment the host isn't showing anything on the serial console and remotely power-cycling it isn't showing any signs of life.

At this time of year I don't want to drag anybody in to take care of it, so ETA on recovery/replacement hardware is Monday/Tuesday.

In other news I've made it to year five of the KVM hosting sub-project/thing. Originally started as a Xen host its been running happily for quite some time. I suspect next year, or the year after that the price/specification ratio will end up losing out and we'll cancel the whole thing - but there are no immediate reasons to make any change.

Finally I knocked up a simple tool to validate my TinyDNS records prior to uploading them. It is simplistic, but adequate to catch the kind of mistakes I make:

• http://tinydns-lint.repository.steve.org.uk/file

Honestly it probably wants to be rationalised a little more - and check records more carefully. e.g. Ensure that the host a CNAME refers to itself exists, and making sure that the nameservers specified are valid.

I just wanted to make something quick after accidentally uploading a zonefile where I'd managed to fat-finger several important records. le sigh.

Oddly enough asking on serverfault.com showed no real suggestions - other than actually running tinydns locally and doing a zone-xfer to validate records. Overkill and harder than I'd like.

Happy New year if you care about such things..

"I finished growing up, Léon. I just get older. " - Leon

Syndicated 2011-12-31 20:29:45 from Steve Kemp's Blog