Older blog entries for Stevey (starting at number 630)

A shaved head is a sign of a tidy mind?

This week I have mostly:

Knocked up a forum ("gathering")

It is a simple clone of hacker news, storing things in Redis.

None of the aging though, entries are first->last and my tagging support is basic.

I'm in two minds about releasing it. I'm in three minds about deploying it at the location I'd written it for - forums.lumail.org - since there's nothing worse than a forum with no posts, unless it is a forum that used to be popular and is now a wasteland, or circlejerk. (c.f. slashdot. ahem.)

Updated Slaugther

I found a hidden dependency when installing a new slaughter controlled host.

A new release is imminent.

Setup a remote host for backups

Using BigV I configured a system with LUKS-encrypted disks to act as a "remote dropbox" via git-annex, and rsync.

An encrypted volume is manually mounted post-boot. It stays mounted (oops that's bad) but provides security when the guest is offline, or has been retired.

Wrestled with Graphite

Because damn that software is hard to install.

Turned down two weddings

Because I will never shoot a wedding again. And if I did I'd not do it for free for "exposure".

Merged my (digital) music archive with that belonging to my partner

Which took more discussion than a) moving in together, or b) opening a shared bank account.

Secretly decided I do like my kindle

Even though I bought two books from a charity shop tonight I'm almost certainly going to file them away and look for the epub online instead.

He fought with the goblins! He battled the trolls! He riddled with Gollum! The magic ring he stole!

This weekend I'll be mostly offline. Saturating my home broadband while I sync backups.

Syndicated 2013-08-30 19:46:32 from Steve Kemp's Blog

Lack of referrers on github is an annoyance

Github is a nice site, and I routinely monitor a couple of projects there.

I've also been using it to host a couple of my own projects, initially as an experiment, but since then because it has been useful to get followers and visibility.

I'm a little disappointed that you don't get to see more data though; today my sysadmin utilities repository received several new "stars". Given that these all occurred "an hour ago" it seems likely that they we referenced in a comment somewhere on LWN, hacker news, or similar.

Unfortunately I've no clue where that happened, or if it was a coincidence.

I expect this is more of a concern for those users who use github-pages, where having access to the access.logs would be more useful still. But ..

Syndicated 2013-08-26 11:40:41 from Steve Kemp's Blog

Soon it will be time for something different

This weekend I'm mostly alternating between reading, writing, and trying to avoid death by the plague.[*]

I've switched Lumail to using a UTF-8 aware string library, which means we can now handle the obvious case:

--
-- Prove keybindings work.
--
keymap['global']['π'] = 'msg("UTF-8 rocks!")'

Similarly we can stuff input into the buffer:

--
-- Pretend the user typed ":msg ...\n"
--
stuff( ":msg('π is pie')\n" );

This transition was annoying to handle, but wasn't too difficult. There is only one more major update required, according to the development roadmap, which is to double check that UTF-8 output is correct.

Otherwise I think I'm almost done. In the sense that I don't see anything obvious missing, barring things that won't ever happen such as mutt-style "tag" support.

I've updated the online examples to include some nice code:

• A nice menu for viewing/opening attachments: view_attachments().
• A nice system for marking messages as read based upon their subjects: mark_read_by_pattern().

I can't claim to have many users, so far the development has been carried out by myself and approximately four other people. But that matters not. I genuinely believe this is a good client and it really suits the way that I handle (large volumes of) email:

• Show folders with unread mail.
• Quickly read it.

Allowing you to open multiple folders at once means you get a great view into your currently-unread mail, regardless of where procmail has placed it.

The overriding feeling having "completed" the client is that Lua rocks. I'm torn between wanting to sleep some more, and wondering what other system/package/tool can be extended by Lua. As epiphanies go my on_idle() update takes some beating.

* - I do not have the black death, but I'm not well.

Syndicated 2013-08-17 13:40:52 from Steve Kemp's Blog

Lumail binaries are wheezy only for the moment

This morning I made a new release of Lumail, which recently completed the transition from using mimetic to GMime for all its MIME needs.

I was happy with mimetic, except it didn't have the facility to decode encoded header-values. I wrote some code, but it was broken, so I made the decision that we should move to something that made this easier, and GMime was chosen.

Jeffrey Stedfast was very helpful in answering my questions with near-perfect code samples.

Beyond that this release features some more Lua primitives and a couple of bug-fixes.

The only annoyance is that the version of GMime I'm using, 2.6.x, isn't available to users of the Squeeze release of Debian GNU/Linux. It is available as a backport, but that means building binaries with sbuild is a pain - due to #700522.

So for the moment I've only built binaries for Wheezy users.

ObQuote: "You know who I am, I've not been off TV for that long!" - Alan Patridge, Alpha Papa

Syndicated 2013-08-14 10:10:47 from Steve Kemp's Blog

Lumail now uses GMime

A hectic day and now we use GMime for all MIME-fu.

This has allowed me to decode headers correctly, setup MIME parts properly in outgoing mails with attachments, and cleanup the code-base.

The next release of Lumail will contain basically just this change, as it is pretty drastic. But first I need to work out how to make binaries for Squeeze compiled against the back-ported version of gmime-2.6.x.

(Previously we used libmimetic. Which was awesome in its way, but caused me some pain with RFC 2047 header-decoding.)

Syndicated 2013-08-11 20:48:36 from Steve Kemp's Blog

TAB completion is hard

I've spent the past few days overhauling the TAB-completion which is included in lumail.

Completing a single token is easy, if there is only one match, and you limit yourself to completing at the start of a line. But doing real completion is hard. Consider the case where you want to complete something like this:

unread_message_colour("re[TAB]

Clearly completing the first part "unread_[TAB]" is simple. But to complete "re" to "red" you need to split up your input line into tokens so that you can recognize a quote as a valid completion point.

Similarly you need to split on "(" to allow:

-- show the path to the editor
msg(edit[TAB]

To allow this to be changed/controlled by the user I defined completion_chars() which contains: SPACE, QUOTE, "(", etc.

I'm pleased with the user-callback for offering completion suggestions, my own is pretty basic and just includes all user-defined functions as well as an address book. The latter allows steve.org.uk[TAB] to complete to: "Steve Kemp" <steve@steve.org.uk> - because we allow matches anywhere in the completion string, rather than just prefix-matching.

I struggled with resolving ambiguities, but now that is handled correctly too. Press "TAB" when there are multiple choices available and you can graphically TAB-through the available choices, or press Esc to cancel.

In conclusion I've spent a few days fighting with user-interface stuff and now the mail-client is better, but I've still to tackle RFC 2047 header decoding because that is really hard!

Syndicated 2013-08-04 09:00:45 from Steve Kemp's Blog

International character sets and encodings are hard.

Today I've made the 0.15 release of lumail, which has several fixups and cleanups.

The previous release included a rewrite of the scrolling code, courtesy of kain88-de. This release fixes a few corner cases in that update which caused empty messages/Maildirs to be highlighted - operating on such ghost-entries would cause a segfault. Oops.

I've received several more great contributions from 7histle, and trou and I'm very happy with the state of the code and the usefulness of the application.

The biggest outstanding issue is RFC 2047 header decoding. Converting subject/to/from fields to readable versions of their encoded form:

Subject: =?utf-8?Q?Blipfoto=20=2D=20Introducing=20the=20all=2Dnew=20Bli

This is annoying because I'm using mimetic for handling all MIME-related code, and this doesn't seem to offer the facilities that I need.

The current plan is to use the RFC-2047 handling from vmime, but I've fought with that library unsucessfully for two days now - and a further complication is that the library is included in Squeeze/Sid, but not the stable release of Debian.

In conclusion I still regard the client as complete, because I'm using it exclusively and I rarely get "foreign" mails. But there is one more push required to fix all the outstanding bugs which generall boil down to:

• Decode headers properly.
• Ensure all our input/output is in UTF-8.

Randomly I'm wondering if I can call out to Lua to do the header decoding. Add "on_header_field()" and display the results. So today I'll be looking at how sensible that is, probably not very.

Syndicated 2013-07-26 09:25:06 from Steve Kemp's Blog

There must be a name for bugs you only find post-release

This week I made two releases of my mail client. Immediately after both releases I found bugs. Despite having been using the github source tree on my box for reading mail for days.

There must be a name for bugs that come up immediately after you've just made a release.

I'm torn between wanting to make a new release right now to fix the thing I spotted, or wait a few more days to fix a few other niggles.

Still I did write some cool code today:

1. If a mail is received on the list debian.security-announce
2. And the package in the Subject: is not installed on the current machine.
3. The mail is marked as read.

Sure this means that a package on my webserver won't be visible to me, but my upgrade tool will see that. It just decreases the odds I read about an update that doesn't apply to me.

ObQuote: "don't pay heed to temptation
for his hands are so cold"

Syndicated 2013-07-18 19:49:25 from Steve Kemp's Blog

lumail is complete

So, writing an email client, how did that turn out? Pretty damn well as it happens.

My views continue to range from "email is easy" to "email is hard". But I'm now using my home-grown mail-client for 100% of my personal mail-handling.

Writing a mail client did seem a little crazy when I started, but the problem breaks down into a few distinct steps and individually they're not so hard:

• Display a list of folders.
• Display a list of messages.
• Display a single message.

Once you've got that you're almost 80% complete. You're just missing things like "delete", "reply", "compose", and those things are pretty simple to implement.

I definitely made the right call in making it scriptable with Lua, because I've been able to write so many functions for working with mail. For example marking messages in all folders matching a regular expression.

The code is pretty well structured, and now I've got TAB-completion support on all primitives and user-additions I'm finding it a lot nicer to use.

The future? I'm not sure. But right now I'm very happy.

Syndicated 2013-07-09 12:31:35 (Updated 2013-07-09 20:13:40) from Steve Kemp's Blog

This weekend I will be mostly upgrading to wheezy

Having migrated my websites away from my ssh/mail box I'm going to upgrade that this weekend.

I've got my new mail client, lumail, working well enough to use exclusively, so the upgrade should be nice and simple.

I spent a few hours last night removing packages from my ssh/mail box to trim it down. Removing a bunch of Perl modules I used in my CGI coding, removing services such as nfs-common, portmapper, etc.

Today I'll have a stab at the upgrade. The only thing I have to be careful of is my backported/tweaked qpsmptd packages. I'll try the native wheezy version now it has caught up.

(I don't use any of the standard qpsmtpd plugins at all. Instead I have a separate tree of my own custom anti-spam and virtual-hosting aware plugins. They all work in a unified fashion. Using these plugins against a new version of qpsmtpd should be just fine. But obviously I need to test that.)

Work on Lumail is probably going to slow down now it is genuinely in use, but I'll keep an eye out for feature requests and missing primitives. Annoyingly I wasted 30 minutes just now implementing a plugin I'd already written: lumail issue #51.

I also need to step-back this weekend and reassess my hosting. When I was tweaking my slaughter setup I recently realized I have more hosts than I thought:

• Cluster running for Debian-Administration.org:
  • 4 x web nodes
  • 2 x database nodes
  • 2 x "planet" nodes
  • 1 x "misc" node.
• Personal stuff
  • 1 x ssh/mail host. (ssh.steve.org.uk/mail.steve.org.uk)
  • 1 x web host. (www.steve.org.uk, www.lumail.org, etc, etc.)
  • 1 x blogspam.net server (www.blogspam.net)
  • 1 x builder node - Runs buildd for producing my packages.

Total: 13 virtual machines. (+ one kvm-host)

Syndicated 2013-07-06 12:07:23 from Steve Kemp's Blog