Older blog entries for Stevey (starting at number 542)

Steve, in brief

In brief:

• I made the firefox-wrapper package public. Although it seems that mozilla.debian.net is liable to be preferred by most.
• I wrote a script to archive a reddit inbox as a maildir folder, reddit2maildir. Perfect for archiving those interesting discussions that aren't public.
• I posted some random animal photographs, mostly to remind myself to take more.

Finally having recently bought the Canon 70-200mm f/2.8 lens for a King's ransom I've agreed to buy the 24-105mm f/4.0 lens from a friend - that will be my new portrait lens of choice, and I'll sell my existing 85mm f/1.8.

ObQuote: "I could help you cross your yard." - Up

Syndicated 2011-06-26 11:05:50 from Steve Kemp's Blog

So you want to install the most recent firefox?

If you've been following new releases you'll see there is a new Firefox browser out, version 5.0.

This will almost certainly make its way into Debian's experimental tree soon, but that doesn't help users of the Debian Stable release. The only sane option for those users (such as myself), without a backport, is to install locally.

So I did the obvious thing, I made /opt/firefox then installed the binary release into it. Then I found that it was good, lovely and fast.

Unfortunately the system firefox and the local firefox are not really compatible. Run the local one, then click on a link in the gnome terminal and it wants to open the system one. Ho hum.

The solution:

• Remove your local firefox & iceweasel packages.
• Create the shell scripts /usr/bin/firefox & /usr/bin/iceweasel to exec the one stored beneath /opt.
• Rejoice.

Of course this being Debian we don't want to do that. So instead here is a package that will let you do that:

• http://firefox.repository.steve.org.uk/file

Download. Build. Install. If you install your local package to a location different than /opt/firefox update the configuration file /etc/firefox/firefox.conf to point to it.

Possibly useful?

ObQuote: "I could help you cross your yard." - Up

Syndicated 2011-06-23 22:39:05 from Steve Kemp's Blog

Continuous integration that uses chroots?

I'd like to setup some auto-builders for some projects - and theese projects must be built upon Lenny, Squeeze, Lucid, and multiple other distros. (i386 and amd64 obviously.)

Looking around I figure it should be simple. There are a lot of continuous integration tools out there - but when looking at them in depth it seems like they all work in temporary directories and are a little different to how I'd expect them to be.

Ultimately I want to point a tool at a repository (mercurial), and receive a status report and a bunch of .deb packages for a number of distributions.

The alternative seems to be to write a simple queue submission system, then for each job popped from the queue run:

• Creates a new debootstrap-based chroot.
• Installs build-essential, mercurial, etc.
• Fetches the shource.
• Runs make.
• Copies the files produced in ./binary-out/ to a safe location.
• Cleans up.

Surely this wheel must already exist? I guess its a given that we have to find build-dependencies, and that we cannot just run "pbuilder *.dsc" - as the dsc doesn't exist in advance. We really need to run "make dependencies test build", or similar.

Hudson looked promising, but it builds things into /var/lib/hudson, and doesn't seem to support the use of either chroots or schroots.

ObQuote: "I feel like I should get you another sweater." - "Friends"

Syndicated 2011-06-12 15:01:34 (Updated 2012-03-07 01:08:53) from Steve Kemp's Blog

So I chose fabric and reported a bug..

When soliciting for opinions, recently, I discovered that the python-based fabric tool was not dead, and was in fact perfect for my needs.

During the process of getting acquainted with it I looked over the source code, it was mostly neat but there was a trivial (low-risk) symlink attack present.

I reported that as #629003 & it is now identified more globally as CVE-2011-2185.

I guess this goes to show that getting into the habit of looking over source code when you install a new package is a worthwhile thing to do; and probably easier than organising a distribution-wide security audit </irony>.

In other news I'm struggling to diagnose a perl segfault, when running a search using the swish-a perl modules. Could it be security worthy? Possibly. Right now I just don't want my scripts to die when I attempt to search 20Gb of syslog data. Meh.

ObQuote: "You're scared of mice and spiders, but oh-so-much greater is your fear that one day the two species will cross-breed to form an all-powerful race of mice-spiders who will immobilize human beings in giant webs in order to steal cheese. " - Spaced.

Syndicated 2011-06-06 18:57:55 from Steve Kemp's Blog

How do you deploy applications?

I've got a few projects which are hosted in mercurial repositories. To deploy them I manually checkout the repository, create symlinks by hand, then update apache thttpd to make them work.

When I want to update my applications I manually become the correct user, find the repository and run "hg pull --update".

I think it is about time that I sat down and started doing things neatly. I made a start at this by writing a shell script for each site called .deploy then I drive it like so:

#!/bin/sh
#
# ~/bin/deploy  execute the .deploy file associated with this project.
#
while true; do

    #
    #  If we're at the root directory we're done.
    #
    if [ $PWD = "/" ]; then
        echo "Reached /"
        exit
    fi

    # found our file?
    #
    if [ -x ".deploy" ]; then

       ./.deploy
       exit
    fi

    cd ..
done

It seems the main candidate is capistrano, which was previously very Ruby on Rails centric, but these days seems to be divorced from it.

Alternatively there is the python-based fabric project which has been stalled for two years, vlad the deployer (great name!) which is another Rake-based and thus Ruby-loving system, and finally whiskey disk which is limited to Git-based projects as far as I can tell.

In short each of these projects is very similar, and each relies upon being able to do two things:

• SSH to remote machine(s) and run a command.
• Copy files to the remote command / pull a repository from a known location.

I've automated SSH before, and I've automed SCP/rsync. The hard part is doing both "copy" and "command" over one SSH channel - such that you don't get prompted for passwords multiple times - and handling the case of runnign sudo where appropriate. Still most of the initial stages are trivial.

I wonder what project I should be using:

• I like perl. Perl is good.
• I use mercurial. Mercurial is good.
• Rake is perhaps permissable, but too ruby-centric == not for me.

Anything I've missed? Or pointers to good documentation?

ObQuote: "We need to be a little more constructive here, okay? " - Terminator 2

Syndicated 2011-05-30 13:43:11 from Steve Kemp's Blog

Images transitioned, and mysql solications.

Images Moved

So I've retired my old picture hosting sub-domain, and moved all the files which were hosted by the dynamic system into a large web-root.

This means no more uploads are possible, but each link continues to work. For example:

• http://picshare.steve.org.uk/2OrKrfem9XQWz.jpg

Happily the system generated "random" links, and it was just a matter of moving each uploaded file into that static location, then removing the CGI application.

The code for the new site has now been made public, although I suspect there will need to be some email-pong if anybody wishes to use it. Comments welcome.

MySQL replacements?

Lets pretend I work for a company which has dealings with many MySQL users.

Lets pretend that, even though it is true, such that I don't have to get into specifics.

Let us pretend that we have many many hundreds of users who are very happy with MySQL, but that we have a few users who have "issues". That might be:

• mysqld segfaulting every few months, with no real idea why.
  • Transactions are involved. So are stored proceedures.
  • MySQL paid support might have been perfect, or it might have lead to "yup, its a bug. good luck rebuilding with this patch. let us know how it turns out kthxbai."
  • Alternatively it might not have been re-producable.
• Master-Master and Master-Slave setups being "unreliable" such that data inconsistencies arise despite MySQL regarding them as being in sync.
  • Good luck resolving that when you have two almost-identical "mysqldump" outputs which are 6Gb each and which cause "diff" to exit with "out of memory" even on a 64Gb host.
  • Is it possible to view differences in table-data, via the binary records? That'd be a fun project .. for a masochist.
• Poor usage of resources.
• Heavy concurrancy caused by poorly developed applications in a load-balanced environment, leading to stalling queries. (Wordpress + Poor wordpress plugins I'm looking at you; you're next on my killfile).

To compound this problem some of these installations may or may not be running Etch. let us pretend they are not, just to simplify things. (They mostly arent' these days, but I'm sure I could think of one or two if I tried)

So, in this hypothetical situation what would you recommend?

I know there are new forks aplenty of MySQL. Drizzle et al. I suspect most of the forks will be short-lived - lots of this stuff is hard and non-sexy. I suspect the long-lived forks are probably concentrating on edge-cases we've not hit (yet), or on sexy exciting things like new storage engines and going nosql like all the cool kids.

Realistically going down the postgresql road is liable to lead to wholly different sets of problems, and a significant re-engineering of several sites, applications and tools with no proof of stability.

Without wanting to jump ship entirely, what, if any, are our options?

PS. MySQL I still mostly love you, but my two most recent applications were written to use redis instead. Just a coincidence... I swear. No, put down that axe. Please can't we just talk about it?/p>

ObQoote: "I've calculated your chance of survival, but I don't think you'll like it. " - Hitchhikers Film.

Syndicated 2011-05-16 23:38:01 from Steve Kemp's Blog

So I put together another picture sharing site

Almost a year ago I put together a simple picture sharing site, which was designed to accept uploaded images via OpenID logins and then share them. It was created because none of the existing image sharing sites allowed OpenID logins - and because frankly I rarely trust sites I don't control. (Especially when they're free sites, with no obvious revenue stream. We can reference recent compromises if we wish to be topical, but even without them it is all about trust.)

I opened up the site to allow others to use it, but assumed it would remain Steve-only - After all why would somebody choose to use my site rather than imgur, or similar?

Over the year I've spotted several minor problems with the site. The built-in support for comments was never used, primarily because I didn't have any other users, and some of the tagging aspects were hard to use.

Since I've started using twitter I again found myself wanting a picture-hosting site, one with "short links". I started using twitpic.com, but wasn't happy with it.

So the plan was formed; build another picture sharing site. But this time with short URLS. Sadly my first choice of http://picc.ie was denied to me - I couldn't register it as I don't live in Ireland. But my second choice http://linkti.me was free.

As a challenge to myself this site uses Redis for storing all user-details, all image details, and all session data. It hasn't been difficult to implemenet a multi-user site, with logins, file uploads, and tagging. I guess I cheated as a lot of the code is borrowed from the previous site, but this time it looks prettier and there have been many cleanups due to my previous experience.

This time round I was more focussed upon short links, so each image is allowed an incrementing number, which is then encoded using base 36 math - which is the alphabet you get if you use [0-9A-Z]. I designed three shareable links for each image to keep it simple:

• http://linkti.me/v/S - Show a HTML page with the image embedded.
• http://linkti.me/i/S - Show a raw image.
• http://linkti.me/r/S - Show a "thumbnail" of the image.

Why "/r" for the thumbnail? Well because it is a random thumbnail. To cope with difference sizes, and allow the recent image display to look regular, I wanted each thumbnail to be 150x150 pixels. My thumbnail generation simply picks a random X,Y co-ordinate from the image and then cuts out a 150x150 square block starting at that offset.

The results of a random segment of an uploaded image are sometimes very cool, altough equally often they are misleading. If I upload naked people there is the possability that we'll just get a nipple or a cock - but I'm actually quite smug about the way it turned out. Simpler to code than to resize, scale, or similar and surprisingly random when you see 9x9 random images (as you do on the front page).

Anyway that's all for now. I was tempted to talk about implementation or migration, but I've written too much for an early morning already. Some other time?

ObQuote: "He's just trying to freak us out. " - 30 Days of Night.

Syndicated 2011-05-10 09:07:02 from Steve Kemp's Blog

Goodbye, world.

Today I resigned from the Debian project. The following packages are up for adoption:

• asql
• chronicle
• rinse
  • Thomas Lange is going to take this one by the looks of things.

I'll remove myself from Planet Debian tomorrow, assuming the keyring revokation isn't swift and merciless.

ObQuote: This space is intentionally blank.

Syndicated 2011-04-29 17:10:08 from Steve Kemp's Blog

So I finally succumbed and joined twitter.

So I've succumbed and joined twitter. I'm pretending I'm a photographer, rather than a techie/geeky/random person, whether that persists is anybodys guess.

At the very least I don't feel any real need to write/post/ask about Linux, development, or similar. I can do that here or elsewhere. (Not least because length-restrictions are annoying; yes it is true. Size is important.)

So far I've been posting semi-random pictures, I guess that is one way to pass the time, while I play with my camera.

• http://twitter.com/Stolen_Souls

ObQuote: "Please, please, please drive faster!" - Wanted.

Syndicated 2011-04-20 23:08:32 from Steve Kemp's Blog

IPv6 and thttpd

thttpd is a simple, small, portable, fast, and secure HTTP server which supports both IPv4 & IPv6.

However one noticable omission in the handling of requests for thttpd is support for the X-Forwarded-For header - which is even noted upo nthe thttpd wikipedia entry.

There is a simple patch floating around which claims to fix this; but as I belatedly noticed tonight it only works for IPv4.

If you look at libhttpd.h of the thttpd source you'll see this:

typedef union {
    struct sockaddr sa;
    struct sockaddr_in sa_in;
#ifdef USE_IPV6
    struct sockaddr_in6 sa_in6;
    struct sockaddr_storage sa_stor;
#endif /* USE_IPV6 */
    } httpd_sockaddr;

As a quick hack I updated this structure to add the following member:

    char real_ip[200];

Now I could update that member when a client connects and later update it as a result of any X-Forwarded-For: headers which might be present in incoming requests. Finally I updated the logging to use this field rather than anything else and the job was complete.

Without this work if you're running as a proxy and you receive an IPv6 connection you'll see it reported as 127.0.0.1.

I'm sure my approach isn't as clean as it could be - due to the extra member- but it will suffice for now.

ObQuote: "This gun you're holding belonged to your father; he could conduct a symphony orchestra with it. " - Wanted

Syndicated 2011-04-08 20:18:02 from Steve Kemp's Blog