Older blog entries for Stevey (starting at number 610)

Some good, some bad

Today my main machine was down for about 8 hours. Oops.

That meant when I got home, after a long and dull train journey, I received a bunch of mails from various hosts each saying:

  • Failed to fetch slaughter policies from rsync://www.steve.org.uk/slaughter

Slaughter is my sysadmin utility which pulls policies/recipies from a central location and applies them to the local host.

Slaughter has a bunch of different transports, which are the means by which policies and files are transferred from the remote "central host" to the local machine. Since git is supported I've now switched my policies to be fetched from the master github repository.

This means:

  • All my servers need git installed. Which was already the case.
  • I can run one less service on my main box.
  • We now have a contest: Is my box more reliable than github?

In other news I've fettled with lumail a bit this week, but I'm basically doing nothing until I've pondered my way out of the hole I've dug myself into.

Like mutt lumail has the notion of "limiting" the display of things:

  • Show all maildirs.
  • Show all maildirs with new mail in them.
  • Show all maildirs that match a pattern.
  • Show all messages in the currently selected folder(s)
    • More than one folder may be selected :)
  • Shall all unread messages in the currently selected folder(s).

Unfortunately the latter has caused an annoying, and anticipated, failure case. If you open a folder and cause it to only show unread messages all looks good. Until you read a message. At which point it is no longer allowed to be displayed, so it disappears. Since you were reading a message the next one is opened instead. WHich then becomes marked as read, and no longer should be displayed, because we've said "show me new/unread-only messages please".

The net result is if you show only unread messages and make the mistake of reading one .. you quickly cycle through reading all of them, and are left with an empty display. As each message in turn is opened, read, and marked as non-new.

There are solutions, one of which I documented on the issue. But this has a bad side-effect that message navigation is suddenly complicated in ways that are annoying.

For the moment I'm mulling the problem over and I will only make trivial cleanup changes until I've got my head back in the game and a good solution that won't cause me more pain.

Syndicated 2013-05-14 20:23:08 from Steve Kemp's Blog

11 May 2013 (updated 11 May 2013 at 23:12 UTC) »

The rain in Scotland mainly makes me code

Lumail <http://lumail.org> received two patches today, one to build on Debian Unstable, and one to build on OpenBSD.

The documentation of the lua primitives is almost 100% complete, and the repository has now got a public list of issues which I'm slowly working on.

Even though I can't reply to messages I'm cheerfully running it on my mail box as a mail-viewer. Faster than mutt. Oddly enough. Or maybe I'm just biased.

Syndicated 2013-05-11 14:08:15 (Updated 2013-05-11 23:12:59) from Steve Kemp's Blog

So progress is going well on lumail

A massive marathon has resulted in my lumail mail client working well.

Functionally the application looks little different to the previous C-client, but it is a lot cleaner, neater, and nicer internally.

The configuration file luamail.lua gives a good flavour of the code, and the github repository has brief instructions.

Initially I decied that the navigation/index stuff was easy and the rest of the program would be hard; dealing with GPG-signatures, MIME-parts, etc.

But I'm stubborn enough to keep going.

If I can get as far as reading messages, with MIME handled properly, and replying then I can switch to using it immediately which will spur further development.

I'm really pleased with the keybinding code, and implementing the built-in REPL-like prompt was a real revelation. Worht it for that alone.

The domain name lumail.org was available. So I figured why not?

Syndicated 2013-05-07 18:40:27 from Steve Kemp's Blog

After you've started it seems like a bad idea?

To recap: given the absence of other credible alternatives I had two options:

  • Re-hack mutt to give me a sidebar that will show only folders containing new messages.
  • Look at writing a "simple mail client". Haha. Ha. Hah.

I think there is room for a new console client, because mutt is showing its age and does feel like it should have a real extension language - be it guile, lisp, javascript(!), Lua, or something else.

So I distilled what I thought I wanted into three sections:

  • mode-ful. There would be a "folder-browsing mode", a "message-browsing mode" and a "read-a-single-message" mode.
  • There would be scripting. Real scripting. I chose Lua.
  • You give it ~/Maildir as the configuration. Nothing else. If the damn computer cannot find your mailboxes something is wrong.

So how did I do? I wrote a ncurses-based client which has Lua backed into it. You can fully explore the sidebar-mode - which lets you select multiple folders.

From there you can view the messages in a list.

What you can't do is anything "real":

  • Update a messages flags. new -> read, etc.
  • GPG-validation.
  • MIME-handling.
  • Attachment viewing.

For a two-day hack it is remarkably robust, and allowing scripting shows awesomeness. Consider this:

--
-- show all folders in the Maildir-list.
--
function all()
   -- ensure that the sidebar displays all folders
   sidebar_mode = "all";
   -- we're going to be in "maildir browsing mode"
   cmail_mode = "sidebar";
   reset_sidebar();
   refresh_screen();
end

--
-- Test code, show that the pattern-searching works.
--
-- To use this press ":" to enter the prompt, then enter "livejournal".
--
-- OR press "l" when in the sidebar-mode.
--
function livejournal()
   sidebar_pattern = "/.livejournal.2";
   sidebar_mode = "pattern";
   reset_sidebar();
   refresh_screen();
end

--
-- There is a different table for each mode.
--
keymap = {}
keymap['sidebar'] = {}
keymap['index']   = {}
keymap['message'] = {}

--
-- In the sidebar-mode "b" toggles the sidebar <-> index.
--
-- ":" invokes the evaluator.
-- "q" quits the browser and goes to the index-mode.
-- "Q" quits the program entirely.
--
keymap['sidebar'][':'] = "prompt-eval"
keymap['sidebar']['b'] = "toggle"
keymap['sidebar']['q'] = "toggle"
keymap['sidebar']['Q'] = "exit"

-- show all/unread/livejournal folders
keymap['sidebar']['a'] = "all"
keymap['sidebar']['u'] = "unread"
keymap['sidebar']['l'] = "livejournal"

Neat, huh? See the cmail.lua file on github for more details.

My decision hasn't really progressed any further, though I can see that if this client were complete I'd love to use it. Its just that the remaining parts are the fiddly ones.

I guess I'll re-hack mutt, and keep this on the back-burner.

The code is ropey in places, but should you wish to view:

And damn C is kicking my ass.

Syndicated 2013-04-30 15:03:35 from Steve Kemp's Blog

Modern console mail clients?

I've recently started staging upgrades from Squeeze to Wheezy. One unpleasant surprise was that the mutt-patched package available to Debian doesn't contain the "sidebar-new-only" patch.

This means I need to maintain it myself again, which I'd rather avoid. Over time I've been slowly moving to standard Debian systems, trying to not carry too many local perversions around.

Unfortunately if you've kept all your mail since 1994 you have many mailboxes. having mutt-patched available at all, with the sidebar patch, is a great timesaver. But I don't want to see mailboxes I'm never going to touch; just mailboxes with new mail in them.

Also I find the idea of having to explicitly define mailboxes a pain. Just run inotify on ~/Maildir and discover the damn things yourself. Please computer, compute!

If you divide up "mail client" into distinct steps it doesn't seem so hard:

  • Show a list of folders: all, new-mail-containing only.
  • Viewing a list of mail-messages: all in folder, or folders.
  • Compose a new mail.
  • Reply to a mail.

Obviously there is more to it than that. Sending mail? exec( sendmail ). Filtering mail? procmail/sieve/etc. Editing mail? exec(vim).

I'm sure if I were to start a core of a program, suitable for myself, would be simple enough. Maybe with lua, maybe with javascript, but with a real language at the core.

Anyway I've thought this before, and working with quilt and some ropy patches has always seemed like the way to go. Maybe it still is, but I can dream.

(PS. Sup + Notmuch both crash on my archives. I do not wish to examine them further. Still some interesting ideas. It should be possible to say "maildirs are tags; view "~/Maildir/.livejournal.2003" and ~/Maildir/.livejournal.2007 at the same time. Why just a single directory in the "index-view? So 1994.)

Disjointed posts R Us.

Obquote: "How hard could it be?" -- Patrick.

Syndicated 2013-04-26 19:51:21 from Steve Kemp's Blog

sysadmin tools

This may be useful, may become useful, or may not:

Syndicated 2013-04-16 19:31:05 from Steve Kemp's Blog

A mixed week with minor tweaks

As previously mentioned I was looking to package pwsafe for Wheezy, as this is one of the few tools that I rely upon which isn't present.

There are now packages available, with the source on github.

I've also been doing some minor scripting because I've run into a few common problems recently:

run-parts

run-parts is a simple utility which will run every executable in a directory, more or less.

In Debian-land run-parts is the mechanism for /etc/cron.daily and /etc/cron.hourly - and that is where I've had problems recently.

Imagine you run a backup via cron.daily. Further imagine that you run a post-backup rsync and that this might take many many hours. If your backup takes >=24 hours you're screwed.

To that end I've patched my run-parts tool to alert and exit if a prior invocation is still running.

silent-run

I think everybody has this script - hide all output when running a command, unless the command fails. Looking today I see chronic from Joey's excellent moreutils does this. D'oh.

I think I've done more, but I cannot remember. In conclusion software is both easy and hard - easy because these two trivial changes were within my reach, but hard because years after encountering GNU/Linux we still have to add in the missing pieces.

Still could be worse, I spent four/five hours yesterday evening fighting with MS-SQL server, and that is time I'm never going to get back.

Syndicated 2013-04-13 16:23:47 from Steve Kemp's Blog

So I have a wheezy desktop

I look after a bunch of servers, working for Bytemark that is not a surprise, but I only touch a very small number of desktop systems.

precious - My desktop

This is the machine upon which I develop, check my personal mail, play my music & etc.

steve - My work machine

To keep the working from home separation going I have a machine I only use for work purposes.

travel/travel2 - EEPC box

I have two EEPC machines, a personal 701 and a work-provided 901.

Honestly these rarely get used. One is for when I'm on holiday or traveling, the second for when I'm on-call.

Yesterday I got round to upgrading both the toy EEPC machines to wheezy. The good news? Both of them upgraded/reinstalled easily. Hardware was all detected, sleeping, hibernation, wifi, etc all "just worked".

Unfortunately I am now running GNOME 3.x and the experience is unpleasant. This is a shame, because I've enjoyed GNOME 2.x & bluetile for the past few years.

The only other concern is that pwsafe appears to be scheduled for removal from Debian GNU/Linux - the list of open bugs shows some cause, but there are bugs there that are trivial to fix.

For the moment I've rebuilt the package and if I cannot find a suitable alternative - available for squeeze and wheezy - then I will host the package on my package repository.

In conclusion: Debian, you did good. GNOME, I've loved and appreciated you for years, but you might not be the desktop I want these days. It's not you, it's me.

Syndicated 2013-04-06 04:27:58 from Steve Kemp's Blog

30 Mar 2013 (updated 30 Mar 2013 at 12:12 UTC) »

Time passes, Thorin sits down and starts singing about gold.

This weekend I have mostly been reading Longitude: The True Story of a Lone Genius Who Solved the Greatest Scientific Problem of His Time .

In modern times we divide the earth up into rings of lines, latitude and longitude, as wikipedia will explain.

Finding your latitude is easy, finding your longitude is a difficult process, and it was vitaly important when people started to sail large distances, the book contained lots of stories of sailors being suddenly suprised by the appearance of land - because they'd misjudged their position.

Having four ships, containing garlic, pepper, and other goods of value exceeding the total wealth of the UK, sink all at once was a major blow. Not to mention the large number of sailors who lost their lives.

There were several solutions proposed, involving steady hands and telescopes, etc, but the book mostly discusses John Harrison and his use of watches/clocks.

John Harrison was featured in Only Fools & Horses, as the designer of the watch that made Delboy & Rodney millionaires.

->Time on our hands

The idea of using a clock is that you take one with you, set to the time of your departure location. Using that clock you can compare the time to the local-time, by viewing the sun, etc. Calculating the difference between the two times allows you to see how far away, in degrees, from your port, and thus how far you've traveled.

Until harrison came along clocks weren't accurate enough to keep time. His clocks would lose a second a month, until then clocks might lose 15 minutes a day. (With more variations depending on temperture, location, and pressure. Clearly things like pendulum clocks weren't suitable for rocking ships either.)

All in all this book was a great read, there were mentions of Galilao, Newton, and similar folk who we've all heard of. There was angst, drama, deceit, and some stunning craftmanship.

Harrison was a woodworker, and he made his clocks out of wood (+brass where necessary). Choosing fast/slow-grown wood depending on purpose, and using wood that secreted oils naturally allowed him to avoid lubrication - which improved accuracy, as lubricants tend to thin/thicken when temperature/pressure change.

A lovely read, thank you very much.

In other news I received several patches for my templer static-site generator, and this has resulted in much improvement. I've also started using Test::Exception now, and slowly updating all my perl code to use this.

Syndicated 2013-03-30 10:47:46 (Updated 2013-03-30 12:12:41) from Steve Kemp's Blog

Want to fight about it?

So via hackernews I recently learned about fight code, and my efforts have been fun. Currently my little robot is ranked ~400, but it seems to jump around a fair bit.

Otherwise I've done little coding recently:

I'm pondering libpcap a little, for work purposes. There is a plan to write a deamon which will count incoming SYN packets, per-IP, and drop access to clients that make "too many" requests "too quickly".

This plan is a simple anti-DoS tool which might or might not work in the real world. We do have a couple of clients that seem to be DoS magnets and this is better than using grep + sort against apache access logs.

For cases where a small number of source IPs make many-many requests it will help. For the class of attacks where a huge botnet has members making only a couple of requests each it won't do anything useful.

We'll see how it turns out.

Syndicated 2013-03-24 10:08:31 from Steve Kemp's Blog

601 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!