Older blog entries for dan (starting at number 181)

Debian, runit, chruby, bundler

Pretty much ever since I wrote it the software that powers this blog – a Ruby Sinatra app called “My Way” – has been running on a Bytemark VM inside a tmux session, and every time I’ve rebooted the server I’ve not only had to restart it by hand but first to remember how to restart it by hand.

I’m in the process of migrating the said VM to one of Bytemark’s new BigV VMs (New! Shiny! More RAM! Marginally Cheaper!) and taking the opportunity to clean it up a bit first. After reading Steve Kemp’s article on runit I decided to give that a go. This is notes-to-myself on what I’ve found so far

:; cat /etc/sv/my-way/run 
#!/bin/bash 
exec 2>&1
cd /home/my-way/my-way
. /usr/local/share/chruby/chruby.sh
chruby ruby-2.0.0 
export LANG=en_GB.UTF-8
exec chpst -u my-way -v bundle exec ruby -I lib bin/my-way.rb

:; sudo update-service --add /etc/sv/my-way
Service my-way added.

This is the script that starts the blog server, and the installation procedure thereof

Worthy of note:

1. per convention, the run scripts (and attendant files) live in directories /etc/sv/someservicename, and these directories are are then symlinked into /etc/service by update-service
2. chruby doesn’t run in sh, so we run this script under bash
3. it redirects stderr to stdout so the svlog process (see below) can see it
4. it runs as root up until the chpst invocation, so the ruby that you specify needs to be in /opt/rubies and not in /home/yourusualuser/.rubies. If you ran ruby-install under sudo it will have put it in the right place.
5. runing bundle install with the --deployment flag when installing the ruby project will have sidestepped a whole class of “can’t find your gems” issues. So do that.

Next up is

:; cat /etc/sv/my-way/log/run 
#!/bin/sh
exec svlogd /var/log/my-way

This is the script that makes sure logs go somewhere. Specifically, they go to the file /var/log/my-way/current, which svlog is able (though as far as I know not yet configured) to rotate according to some defined criteria, and without needing to restart the server. The log files are owned by root, but maybe that’s changeable using chpst again.

:; sudo sv  status my-way
down: my-way: 94s, normally up; run: log: (pid 13620) 48806s
:; sudo sv  start my-way
ok: run: my-way: (pid 28343) 0s
:; sudo sv  status my-way
run: my-way: (pid 28343) 8s; run: log: (pid 13620) 48818s
:; pkill ruby
:; sudo sv  status my-way
run: my-way: (pid 28379) 31s; run: log: (pid 13620) 48949s
:; sudo sv  stop my-way
ok: down: my-way: 0s, normally up

And here’s how I start and stop it and stuff. Note that it magically restarted after I ran pkill ruby.

If you can read this, it works.

Syndicated 2014-01-19 11:18:31 from diary at Telent Netowrks

Using the HP IP Console Viewer app on Linux

Another success criterion in my current story to get the Machine That Does Everything out of the living room is being able to do things like kernel upgrades without having to go to where it is and plug in a keyboard and screen, and to that end I bidded for and – somewhat unexpectedly – won an HP 1×1×8 IP KVM switch on Ebay.

1. It appears to be actually made by someone called Avocent, though Avocent seem to change their products in non-trivial ways for different badge engineers
2. Along with the switch itself, you need an “Interface Adaptor” for each connected server. This is a thingy that has an RJ45 at one end and a set of keyboard/video/mouse connectors at the other, and should cost around £7 or £8
3. Although you can plug in a keyboard and mouse – and it works just like a local KVM if you do – you will need to connect to the serial port to configure the network settings, there seems to be no way of doing it from a connected keyboard.
4. Although some variants of these things run web servers on ports 80 and 443 which let you download java applets to connect to the servers plugged into them, mine doesn’t. I know not why.

   Starting Nmap 6.00 ( http://nmap.org ) at 2014-01-08 22:48 GMT
   Nmap scan report for kvm.lan (192.168.0.3)
   Host is up (0.011s latency).
   Not shown: 997 closed ports
   PORT     STATE SERVICE
   2068/tcp open  advocentkvm
   3211/tcp open  avsecuremgmt
   8192/tcp open  sophos
   MAC Address: 00:02:99:03:62:5C (Apex)
   

   None of those responds to HTTP or HTTPS requests
5. So you need to download the software yourself. HP love to rearrange their web site, judging from the number of dead links in the google seearch results, but as of the time I write this you can get it from here and if that link is out of date when you read this you may find the file you need by googling for SP50317.tar
6. Having downloaded it, you must untar it and run the setup.bin shell script. Do this with LOCALE=C or it doesn’t work
7. On a 64 bit platform it may complain about missing libraries that you thought you had. This is because it’s 32 bit. Users of the Universal Operating System (a.k.a Debian) can grab the necessary with

   $ sudo apt-get install  libxext6:i386 libxtst6:i386 
   

8. Once you’re through the setup process, you can start the actual viewer which is called IPViewer. The warning strings: '/lib/libc.so.6': No such file it emits is non-fatal and as far as I can tell entirely ignorable.
9. Its keystroke handling is a bit screwy: I found that it has some kind of ‘double echo’ problem on the console, so each key I press emits a character once when I press it and again when I release.

   loaclhost login: ddaann

   There is an autohiding menu at the middle of the top of the screen - mouse around near the titlebar to see if you can find it. From this menu I selected Tools →Session Options, and then the ‘General’ tab. This pops up a dialog box in wich there is a checkbox ‘Keyboard Pass-through’. Selecting this option fixed the ddoouubbllee kkeeyyss problem for me. It’s not all peachy yet, though, because neither Right Arrow nor DEL seem to do anything in Pass-through mode, and the latter of those is key (sorry) to entering the system BIOS Setup interface.
10. The IPViewer.lax file has some interesting-looking settings, including the path to the JVM it wants (I tried with my system OpenJDK 1.7.0_25 and it kind of worked but the keyboard didn’t work at all) and the jvm max memory size. More as I find it.

Syndicated 2014-01-08 22:40:29 from diary at Telent Netowrks

How to avoid using Outlook for Mac (mostly)

New Year’s Resolution number 2 (number 1 is domestic) is, taking inspiration from Neil Gaiman, to spend less time on Twitter and more time on long-form – or at least, paragraphs-long-forms – of writing. So to ease myself in gently, here’s a plug for DavMail

It might be overstating the case slightly to claim that one of the worst things about working at $JOB is that everyone has to use Outlook, but it’s certainly not one of the better things. So, having a spot of time between Christmas and the New Year to improve my working environment, I started looking for other ways to address the Exchange server.

Step 1 was to install DavMail

DavMail is a POP/IMAP/SMTP/Caldav/Carddav/LDAP exchange gateway allowing users to use any mail/calendar client (e.g. Thunderbird with Lightning or Apple iCal) with an Exchange server, even from the internet or behind a firewall through Outlook Web Access.

This bit went smoothly.

(Step 1.1 was to Paypal the DavMail author a small amount of cash: this is already making my work environment so much nicer)

Step 2: install and configure offlineimap. It would be neat if using offlineimap didn’t require one to learn Python (folder filtering syntax, I am looking at you) but I told it my ‘remote’ server was reachable at localhost:1143 and cargo culted some stuff to drop all the boring folders full of crap, and off it went (very slowly) downloading my mail. This is approximately how my .offlineimaprc looks:

[general]
accounts = Exchange

[Account Exchange]
localrepository = Local
remoterepository = Remote
status_backend = sqlite

autorefresh = 1 
# minutes between refreshes
quick = 30
# partial refreshes between full ones

[Repository Local]
type = Maildir
localfolders = ~/SB-Mail

[Repository Remote]
folderfilter = lambda folder: folder not in ['Trash','Drafts', 'Junk'] and not re.search("INBOX/20\d\d$", folder) and not re.search("^Trash/", folder)
type = IMAP
remotehost = localhost
remoteport = 1143
remoteuser = mydomain\myusername
remotepass = mypassword

mydomain, myusername and mypassword are placeholders: make the obvious substitutions.

Step 3: Gnus. Point it at the local folder that offlineimap is talking to, and tell it to use davmail via smtpmail for sending outgoing messages:

(setq gnus-select-method '(nnmaildir "sb"
                           (get-new-mail nil)
                           (target-prefix "")
                           (directory "~/SB-Mail/")))

(setq user-mail-address "my.email.address@my.place.of.work.co.uk")
(setq smtpmail-smtp-server "localhost")
(setq smtpmail-smtp-service 1025)
(setq smtpmail-auth-credentials "~/.authinfo")

.authinfo looks like this

machine localhost login mydomain\myusername password mypassword port 1025

Again, replace my* with actual values

Step 4: Oh but, dear Lord, this thing could not find an uglier way to render HTML email, what’s up with that? Turns out this is because the Emacs app in Homebrew wasn’t built with libxml support. Turns out this is because the bundled libxml in MacOS Lion (other Bloodthirtsy Yet Cuddly Big Cats are available) is missing the file that pkg-config needs so that any app that might want to build against it can find it. So:

Step 3.9: Building your own Mac Emacs is surprisingly easy – just follow the instructions in nextstep/INSTALL – but unless you take steps to make libxml show up, the resulting app will suffer the same problem as Homebrew’s binary. So:

Step 3.8 Install the homebrew libxml package, and (2) add the .pc file it provides to PKG_CONFIG_PATH, because – as it doesn’t want to clash with the broken builtin libxml – it installs into some obscure out-of-the-way place that nobody will ever find it. “I eventually had to go down to the Cellar”. “Yes, Mr Dent, that’s the display department”.

$ pkg-config --cflags libxml-2.0
Package libxml-2.0 was not found in the pkg-config search path.
Perhaps you should add the directory containing `libxml-2.0.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libxml-2.0' found
$ PKG_CONFIG_PATH=/opt/boxen/homebrew/opt/libxml2/lib/pkgconfig/  pkg-config --cflags libxml-2.0
-I/opt/boxen/homebrew/Cellar/libxml2/2.9.1/include/libxml2

Your Pathnames May Vary. You get the idea. When is Macos Shaved Yak planned for release?

But after all that, it Just Works. Mostly. Sometimes it complains that files go missing, but I think that’s because I’m still checking mail occasionally (so causing them to be marked as read) in Outlook and this makes them jump from new/ to cur/. And I haven’t yet figured out how to deal with calendaring or contacts, but the former is a minor annoyance and the latter is probably just a matter of finding a way to make Gnus talk LDAP to the LDAP proxy in DavMail

Anyway, Happy New Year. Mine will be.

Syndicated 2014-01-06 07:51:17 from diary at Telent Netowrks

finagle the fenestra

So I have a working laptop again, but the touchpad has frankly stupid “touch then drag” behaviour that makes window placement an absolute pain. In my lunch hour yesterday, lacking the internet access I needed to address the Clojure issue that I had originally holed up in the Barbican library to work on (couldn’t get a 3G or a wireless connection) I decided to look again at last year’s ‘throw-window’ sawfish hackery and have turned it into something slightly more brutal: now instead of slinging the focussed window right to the screen edge it only moves it to butt up against the next window on the screen (more or less).

The code is at https://github.com/telent/dotfiles/blob/master/.sawfish/lisp/throw-window.jl (i.e. it hasn’t moved) and there are a couple of other changes

• it’s now a module, which required some experimentation as the documentation is a bit more abstract and less concrete than I’d like. Note that

• The entire file is now inside the body of the define-structure form

• The symbols in the export clause can now be referenced, as written, in (what for want of a keener understanding of how it actually works I am calling) the global environment. By which I mean to say, we call the function from outside the module as throw-window-down not as throw-window.throw-window-down

• the command names didn’t need exporting but are nevertheless globally available anyway. Don’t ask me why.

• do-move has been enhanced with a call to window-frame-offset so that the pointer is warped to where we actually were relative to the window contents instead of being off by the size of the titlebar/left border.

Syndicated 2013-10-24 22:32:20 from diary at Telent Netowrks

Meet the new laptop ...

… same as the old laptop. Seriously. The only time you will see the word ‘fast’ in the same sentence as Sandisk U100 is in articles recycled from the press release they sent out at launch. And maybe now here, in the sentence “I threw away my Sandisk U100 and replaced it with something fast”.

I mean, it has to be said, there may have been something wrong with my particular drive -

  $ dd if=/dev/zero bs=1M count=1000 of=/home/dan/zeros.bin 
  1048576000 bytes (1.0 GB) copied, 230.009 s, 4.6 MB/s 

though I’m not the only one so … maybe not. ANYWAY. Point being, I replaced it with a Plextor M5M and it’s fixed all the odd stalls and hangs and stutters that I used to experience on this machine. Which is lovely, because reformatting it with ext4 a few months ago so completely didn’t.

A summary of the upgrade process

1. remove 11 or so screws from the bottom of the case. Use a decent screwdriver, because if your machine is like mine, the screws are (1) made of cheese; (2) held in with threadlock
2. remove the bottom of the case
3. look for the bit that looks like an msata ssd. Pictures of msata ssds are available on the internet
4. remove the foam pad stuck to it
5. unplug and remove it
6. plug the new one in
7. reassembly is the reverse of disassembly

In summary, not hard. And I can confirm that my puppet manifest worked pretty much perfectly for reinstalling it. With the exception of a couple of dependencies I forgot to declare, but nothing that running puppet twice didn’t fix.

Syndicated 2013-10-21 18:26:19 from diary at Telent Netowrks

Clojure on company time / my first cljs

From: Daniel Barlow <dan@telent.net>
To: london-clojurians <london-clojurians@googlegroups.com>

A few weeks ago I foolishly volunteered to give a "this is what I remember
about functional programming" lunchtime talk for our company-internal
"Level Up" informal training sessions: the material being a combination of
what I learned at university many years ago and since have largely
forgotten[*], with the slightly more practical stuff I've since picked up
working with CL and Clojure.

So the presentation was yesterday, and seemed to be well received inasmuch
as at least nobody walked out: I spent 25 minutes blethering with some
slides, and then we did 4clojure problems for the rest of the hour.

*  I wrote some presentation software for the occasion:
https://github.com/telent/schuko which is also my first use of cljs outside
of a dojo setting.  Shows up that I really need to investigate some more
functional patterns for DOM/event-driven CLJS systems, because what I've
done there really doesn't feel Lispy.  Feedback welcome

* The markdown source for the slides
https://github.com/telent/schuko/blob/master/example/fp-notes.md.
Instructions are in the README for how to convert this into HTML

I make no particular claim to correctness, completeness, or purity of
thought, and the bits about normal-order evaluation probably don't make a
lot of sense because they're written in Clojure syntax yet don't work in
Clojure because it (usually, by default) has strict evaluation - but I cast
this out to the wider world (that's you guys) in case you find it useful
anyway in your own evangelising


-dan

[*] Bless you, Bird & Wadler

Syndicated 2013-10-18 09:18:34 from diary at Telent Netowrks

iwl if you will

I don’t pretend to understand much about wifi, but – historically, at least – I observe that the phrase “wireless net working” is often, like “military intelligence” or “road works ahead”, most charitably described as oxymoronic.

The ingredients:

• a linux box with an “Intel Corporation Centrino Advanced-N 6230” network controller

• the iwlwifi module

• a dual-frequency (2.4GHz and 5GHz) smallish “Enterprise” network based on Ciso Meraki

The symptoms: sometimes it doesn’t work. Sometimes it works for a couple of minutes and then stops.

The hypothesis: iwlwifi or the hardware it’s talking to doesn’t like talking on 5GHz channels

The fix (first attempt):

:; cat /etc/modprobe.d/iwlwifi.conf 
options iwlwifi 5ghz_disable=1

The problem with this fix: it makes no difference. It turns out 5ghz_disable has no effect any longer

The fix (second attempt): add the line

	freq_list=2412,2417,2422,2427,2432,2437,2442,2447,2452,2457,2462,2467,2472,2477,2482

to the relevant network clause in /etc/wpa_supplicant.conf. And be grateful you’re not using dbus

The result: it seems to be working so far.

In other news, Schuko is hacky-but-probably-functional Showoff-style Markdown→HTML presentation software, in Clojure(script). It generates a static HTML file which you can host anywhere, and uses client-side Javascript and CSS for navigation and page flipping transitions.

Syndicated 2013-10-01 08:40:40 from diary at Telent Netowrks

Fully loaded leiningen

Taking the time to figure out how to use ones tools is more often a good thing to do than it is a thing I actually do. In this case, the thing I have now figured out is how to get all of a multi-file Clojure project loaded into Clojure without visiting each file and doing C-c C-l on it

user> (require 'winnow.core)

That’s right. require on a namespace will actually load the file that defines it – and the files defining the namespaces it depends on. ns or in-ns just create it in an empty state.

Syndicated 2013-06-29 22:04:59 from diary at Telent Netowrks

NetworkManager No More

Finally fed up with it randomly stalling and hanging for tens of seconds at a time, I decided yesterday to reinstall my laptop using ext4 instead of btrfs[*]. This time around I’ve done it all with Puppet, so I can tell you about my other recent change – connecting to protected WPA wireless networks without the “assistance” of NetworkManager.

It turns out to be amazingly simple if you’re reasonably acquainted with editing text files: you need a stanza in /etc/wpa_supplicant.conf for each network you want to connect to (and optionally, each network you want to tell it about that it should not connect to)

network={
        ssid="My SSID"
	priority=20
        #psk="not a passphrase"
        psk=8649204ac9cecc467b6c7cb35806627736c3e28f8303d73f2e9afb0a1dff871c
        id_str "default"
}
network={
        ssid="My other SSID"
	priority=20
        #psk="not a passphrase"
        psk=33e4836e7a1325ee4abe6e7ba0f4f9f2f0927aac6d476b001a2469b67e5f2ad7
        id_str "default"
}
network={
        ssid="Some annoying captive portal"
	priority=0
        id_str "ignore"
}

and then you need to put stuff in /etc/network/interfaces to tell it to use wpa-roam

iface wlan0 inet manual
  wpa-driver wext
  wpa-roam /etc/wpa_supplicant.conf

and then some more stuff to correspond to each id_str you’ve used in wpa_supplicant.conf

iface default inet dhcp
iface ignore inet manual

My setup is not exactly what you see in this commit because I’m not quite stupid enough to put real keys into Github – but it’s otherwise pretty close. And for the rest of the repository, see here

[*] it remains to be seen if it makes any difference at all, but it was probably about due for a good decrufting anyway

Syndicated 2013-06-29 21:43:08 from diary at Telent Netowrks

Stranded Strava ride

This morning, due to some unlikely-ever-to-be diagnosed bug perhaps involving poor network connectivity in our office basement, the Android Strava app started denying all knowledge of my ride into work.

It turned out not to be quite as lost as I feared it would be, but the process of getting it uploaded was still kind of complicated. You will need root on your android, plus a real computer (or a Mac will do at a pinch, if that’s all you have available) with the android stuff installed, sqlite3 and gpsbabel.

• copy the strava database onto the computer. Do this in two steps because adb pull doesn’t have the permissions to read the directory where the database lives

  $ adb shell
  shell@android:/ $ su
  shell@android:/ # cp /data/data/com.strava/databases/strava.db /sdcard/tmp-strava
  shell@android:/data/data/com.strava/databases # exit
  shell@android:/ $ exit
  $ adb pull  /sdcard/tmp-strava
  4566 KB/s (1277952 bytes in 0.273s)
  

• use sqlite3 to get the data points out. This involved a bit of digging around

  sqlite> .tables
  CLUB_ACTIVITY           comments                related_activites
  activities              facebook_athletes       segment_leaderboards
  android_metadata        followers               segments
  athletes                followings              streams
  challenge_leaderboards  heartrates              waypoints
  challenge_participants  kudos                   zones
  challenges              notifications
  sqlite> .headers on
  sqlite> select max(timestamp) from waypoints;
  max(timestamp)
  1370851561000
  sqlite> select * from waypoints where timestamp=1370851561000;
  ride_id|pos|timestamp|latiude|longitude|altitude|h_accuracy|v_accuracy|command|speed|bearing|device_time|filtered|elapsed_time|distance
  c558df30-01ae-43ab-99c6-9b4649b8d596|1742|1370851561000|51.5209762891755|-0.0867852196097374|107.0|10.0|||3.56535005569458|13.0|1370851559177|0|1752811|10778.9532725066
  sqlite> .mode csv
  sqlite> .output /tmp/track.csv
  sqlite> select date(timestamp/1000.0,‘unixepoch’) as date,time(timestamp/1000.0,‘unixepoch’) as time,latiude as lat,longitude as lon,altitude as alt from waypoints where ride_id=‘c558df30-01ae-43ab-99c6-9b4649b8d596’;
  sqlite3> ^D
  

  Yes, you read that right, there really is a column called latiude in that table. Note that the ride_id of my missing ride is not necessarily the same as your missing ride: in this case I looked at the timestamp of the most recent waypoint logged and it corresponded to the time I finished my ride, but you might have to dig further.

• now we have a CSV file, we need only turn it into a format that Strava will recognize, and upload it with the Strava upload button. According to the docs, this GPSBabel invocation should transform a list of waypoints in CSV form into a track in GPX form – according to empirical observation, it merely gives a list of waypoints in GPX format. But Strava seems to accept that as a valid upload anyway

  $ gpsbabel  -i unicsv -f /tmp/track.csv -o tcx -x transform,trk=wpt -t   -F /tmp/track.tcx
  

• And here it is – though, you might argue, hardly worth the effort of recovering it. Somewhere in the process it seems to have lost track of my local time conventions too, but UTC is the one true timezone, so no big deal.

Syndicated 2013-06-10 12:50:29 from diary at Telent Netowrks