Recent blog entries for dan

The Invisible AUDIO element

I said this morning that I was going to replace the browser-native audio controls with something which looks (approximately, at least) consistent everywhere. There’s another couple of reasons for wanting to revisit the way we render the audio element

  • on the default Android browser, we don’t get an ended event when the player gets to the end of the track, which means every five minutes I have to pick the phone up and unlock it and press ‘remove’ in the play queue to trigger the next track
  • when the screen is sleeping or the tab is hidden, the requestAnimationFrame handler that triggers Om repaints is called late or not at all. Again, time to pick the phone up and unlock and …
  • I want/need to make it run on Windows, which does not support Ogg in the audio element. Although in principle I could hang multiple source children onto the audio element and let the browser choose which one it likes best, rewriting the DOM after it has been parsed is said to be not a good idea which means using info from the JS canPlayType method to choose the bext format for each track from those available for that track.

The nice thing about Om application state is that it’s also a perfectly ordinary Clojure atom and we can call add-watch on it to have a perfectly ordinary Clojure(Script) function called whenever it changes. So what we’re going to do is

  • a new key in app-state to contain the desired player state
  • an Om component to render a player UI, and update this desired state when buttons are clicked
  • some event handlers to get news from the audio element and figure out what it might be doing (principally, has it reached the end of the track, or is it having connectivity issues) and update the desired state correspondingly
  • a watch on app-state that calls the function currently named sync-player-state, which compares the desired state to what the audio element is actually doing, and updates the audio element appropriately

Syndicated 2015-01-22 19:26:47 from diary at Telent Netowrks

Lucene, you don't do your Daddy's will

I’m only showing you pictures of the new Sledge UI as a stopgap, as it will shortly be replaced with the new new Sledge UI.

I Am Keeping: the way we allow search refining. Each search you request (either by typing something in or by clicking an artist or title name) is turned into one of those little yellow label things and then you can remove it again just by clicking the ‘x’ (I stole this idiom from the amazon ec2 console ui, but it’s also what the gmail composer does when you type addresses into the to/from lines).

I Am Losing: the link to ‘view play queue’. Rather than a ‘tabs’ metaphor, we will have an ‘expand’ button in the player area at the bottom which causes the queue to roll up and overlay the search results.

Also To Go: the browser-native audio controls, which are getting replaced with something that is consistent across browsers.

But the bigger reason for writing: already gone, the Lucene interface for storing the music database. I started working on having Sledge monitor its music folders for additions/deletions, and realised just how hard it is to persuade Lucene to search for an exact match on a particular field with no ambiguity or helpful tokenising – which is necessary when you want to delete the record for a file that’s not in the filesytsem any more. So now we’re using a flat file and EDN and doing our own tokenising into a bunch of maps that are created at load time. It’s actually faster too, but I don’t think that’s anything to do with Lucene, more with it being the second stab at the problem.

This has now taken substantially more than one weekend and it’s still not done. Damn scope creep.

Syndicated 2015-01-22 07:10:54 from diary at Telent Netowrks

core.async with aleph

(More sledge retrospective)

There was a point about three weeks ago when I thought I had a working audio player, then I tried using it on the phone and I got awkward screeches every thirty seconds through my stereo when I told it to play Ziggy Stardust. No, I’m not talking about David Bowie’s voice here, this was genuine “a dog ate my CD” style digital audio corruption. The problem seemed to appear only on Wifi: I could replicate it on my laptop, but it didn’t show up on localhost and it didn’t show up over an ssh tunnel: I suspect it was something related to buffering/backpressure, and facing the prospect of debugging Java code wth locks in it I punted and decided to try switching HTTP server instead.

Documentation on HTTP streaming from core.async channels with Aleph is kind of sparse, at least insofar as it is lacking a simple example of the kind of thing that should work. So here is my simple example of the kind of thing that worked for me: wrap the channel in a call to manifold.stream/->source and make sure that the things received on it are byte-array

(defn transcode-handler [request pathname]
  {:status 200
   :headers {"content-type" "audio/ogg"
             "x-hello" "goodbye"}
   :body (manifold/->source (transcode-chan pathname))})

(from server.clj )

I’m sure there are other things you could put on the channel that would also work, but I don’t know what. java.nio.ByteBuffer doesn’t seem to be one of them, but I’m only going on git commit history and a very fuzzy recollection of what I was doing that day, it might be that I did something else wrong.

Syndicated 2014-12-16 07:34:29 from diary at Telent Netowrks

Using the HTML5 audio element in Om

A quick one: if you want to render the HTML5 audio element with Om and do stuff with the events it raises, you will find that the obvious answer is not the right one. Specifically, this doesn’t work

(dom/audio #js {:controls true
                :autoPlay true
		:ref "player"
                :src bits
		:onEnded #(do-something)
               })

This might be because React has to be taught about each event that each element can trigger and it doesn’t know about this one, or it might be because (it is alleged that) event handling in React is done by placing a single event handler on the top-level component and then expecting events on subelements to bubble up. According to Stack Overflow, audio events don’t bubble

The workaround is to add the event listener explicitly in IDidMount, and to call addEventListener with its third parameter true, meaning that the event is captured by the parent before it even gets gets to the sub-element to be swallowed. Like this

Syndicated 2014-12-15 00:07:51 from diary at Telent Netowrks

clj-webdriver with recent Clojure/Firefox

At the time I write this, the latest release of clj-webdriver is 0.6.1. There are two separate problems with this version, at least as far as I can make out

1) some kind of bug which causes it to fail with the message No such var: clojure.core.cache/through. I haven’t tracked this to its root cause but am guessing that the [org.clojure/core.cache "0.5.0"] in clj-webdriver’s project.clj was too old a version for some other dependency I am pulling in. I added an explicit [org.clojure/core.cache “0.6.4”] in my project and that seems to have fixed it. See clj-webdriver issue 132

2) The version of Selenium it pulls in is 2.39, which is too old to work properly with even the vaguely recent version of firefox I’m using (33.1.1). Fixing this is again just a matter of adding the more recent versions of Selenium stuffz as explicit dependencies in project.clj

With those two changes clj-webdriver now seems pretty happy and I can start adding some basic smoke tests to Sledge so that I don’t have to manually test client-side behaviours whenever I change it

Done: use reference cursors instead of channels for enqueuing/dequeing tracks

Next up: use a channel for xhr search instead of quite so many callbacks

Forthcoming: more work on UI/UX. Add tabs to switch between search view and play queue, unify the different-for-no-good-reason “search” and “filters”.

The branch/commit policy from hereon in is

  • it is a bug if master doesn’t pass regression tests on my machine
  • but there could be any kind of rubbish on branches
  • but I firmly subscribe to the Kanban notion of limiting work-in-progress, so will be striving to keep each of these branches short-lived or to declare them moribund at the earliest opportunity

Note that the tests currently depend on having a music collection containing at least four tracks by Queen. This is not ideal and I will fix it some day but in the meantime you’ll just have to work around it somehow. Maybe try leaving a USB stick in the car for two weeks or something

Syndicated 2014-12-14 23:06:29 from diary at Telent Netowrks

ANN Sledge (We're lost in music)

As a person with a large ripped CD collection at home
I want to find and listen to that music from work/on my phone
So that I don’t have to talk to the people around me

Sledge is a program that you can run on a computer with

  • some music you want to listen to
  • a JVM
  • some means of exposing a TCP server port to the internet
  • libav / ffmpeg

It indexes all the music in the directories you tell it to look in, and then it serves a web page with a search box and some buttons on it, which you can access on a device (computer/phone/tablet/etc) that

  • can access the internet
  • has a web browser that supports the HTML5 AUDIO element and likes Ogg files (most of them, these days)

It’s also the first useful[*] thing I’ve written using Clojure and Clojurescript and Om . Get it at https://github.com/telent/sledge – no jar file download yet, so you’ll need leiningen to build it

[*] defined as: I’m using it.

Standing on the shoulders of github

The heavy lifting was mostly done by others. In addition to the above-mentioned, it uses

  • https://github.com/weavejester/clucy as a Lucene interface
  • https://github.com/ztellman/aleph for streaming the transcoded audio
  • https://github.com/DanPallas/green-tags to wrap JAudioTagger

Future plans

It’s reached MVP, as far as I’m concerned: aside from a couple of bugs it meets my use case. But I do have more planned for it as time permits:

  • UI makeover, make it easier to discover music I’d forgotten I have
  • Make the initial media scan much much faster (currently does about 2000 files a minute on my machine) and/or show the progress as it scans
  • some tools for reporting on duplicate files
  • something to deal with correcting/adding tags to media files that have bad or no metadata
  • transcode to formats other than Ogg Vorbis, maybe, if there are people who wnt to use it with browsers that don’t support Ogg

A long long time ago

Previously: “I started looking at all the UPNP/DLNA stuff once for a “copious spare time” project, but I couldn’t help thinking that for most common uses it was surely way over-engineered”. In the four years (and two days) since my opinions haven’t changed but my tools have.

Syndicated 2014-12-08 07:16:18 from diary at Telent Netowrks

If you can see this, it worked

For values of “this” which you don’t care about and can’t see, but my hacky homebrew blogging engine now watches a bare git repo using inotify and runs a checkout/refreshes its content when it sees changes. I’m not sure it wouldn’t have been simpler just to make it die and then use a, y’know, shell script or something to run a git pull before restarting the server, but I did it this way because ZERO DOWNTIME.

Anyway. You don’t see and can’t care, or possibly vice versa. But the previous Heath Robinson stuff with git hooks wasn’t working now that the bare git repo is owned by someone other than the uid that runs the http daemon, so a different Heath was called for.

Syndicated 2014-02-13 22:50:59 from diary at Telent Netowrks

Keeping secrets in public with puppet

I recently stumbled across dotgpg , which is in essence some scripts to make it easy to securely keep secret files in a public git repo, which are protected by means of having been encrypted (usually for multiple recipients). It comes with capistrano glue for decrypting them again and sending them to your production servers, but it doesn’t quite fit my masterless use case where everything happens on the same box and there isn’t the same notion of a ‘target system’

But it set me to thinking: what if there was some kind of gpg agent that let you type in your key once and used it for multiple decryptions (turns out there is) and what if you then wrote some custom puppet function, let’s call it decrypt, so you could then say

  file {'/etc/wpa_supplicant.conf':
    content=>decrypt("templates/etc/wpa_supplicant.conf.gpg"),
    owner=>root,
    mode=>0600
  }

and everything would Just Work. Well, turns out I did and you can and (as far as I can tell) it does. The custom function is as simple as creating the file puppet/parser/functions/decrypt.rb inside /etc/puppet (or wherever) containing

module Puppet::Parser::Functions
  newfunction(:decrypt, :type=>:rvalue) do |args|
    filename = args[0]
    `/usr/bin/gpg --use-agent --decrypt #{filename}`
  end
end

and now at the expense of a slightly more convoluted puppet invocation

$ sudo  make -C/etc/puppet/ GNUPGHOME=$HOME/.gnupg GPG_AGENT_INFO=$GPG_AGENT_INFO

I can put my wpa network configuration (and my jabber passwords, and smtp client passwords, and some other stuff I can’t right now remember what it is but am sure exists) alongside my all my other configuration instead of either having to do something silly with git submodules or rebuilding it by hand. Am now furiously trying to memorise my passphrase.

Better error checking would be nice, so that it doesn’t overwrite a perfectly good config file with an empty one if the gpg stars aren’t all aligned, but that is left as an exercise for next time.

Syndicated 2014-02-10 18:26:14 from diary at Telent Netowrks

MuDDLe, a faster simpler maildir downloader

One of the services running on my old Bytemark VM was the Dovecot IMAP server. When I started thinking about configuring it on the replacement box I realised that I don’t actually need IMAP these days, so, er, why bother? So I didn’t. But what I do need is a way of getting the Maildir on that machine onto other machines, and everything I looked at to do this job (other than rsync) was fearsomely complicated because it also catered for a zillion other file formats and/or transports. Or because it wanted to sync in both directions, which I don’t really care about that much.

So, muddle

In essence, what it does is this: connect to the remote host, find list of files in cur/ and new/, compare with similar list on local host, create tar stream of differences, transfer it, unpack each transferred file into tmp/ and atomically rename into cur/ when done.

This adds up to one transfer of file names and a second transfer of all the file contents. Each of these is one-way and distinctly non-chatty, so it should have reasonably good network performance characteristics and you can use ssh compression if your network bête noire is bandwidth itself and not just latency.

It might not be as simple as possible, but at the same time it might also be simpler. For example, and as alluded to above, the download is one-way only, so it won’t e.g. update the server to mark messages as read. If you care about that stuff, this is not for you.

(Why not rsync? It can’t detect that a rename from new/foo to cur/foo:2, is a rename, so treats the latter as a new file. Which is a teensy bit suboptimal)

Syndicated 2014-02-02 23:27:24 from diary at Telent Netowrks

Debian, runit, chruby, bundler

Pretty much ever since I wrote it the software that powers this blog – a Ruby Sinatra app called “My Way” – has been running on a Bytemark VM inside a tmux session, and every time I’ve rebooted the server I’ve not only had to restart it by hand but first to remember how to restart it by hand.

I’m in the process of migrating the said VM to one of Bytemark’s new BigV VMs (New! Shiny! More RAM! Marginally Cheaper!) and taking the opportunity to clean it up a bit first. After reading Steve Kemp’s article on runit I decided to give that a go. This is notes-to-myself on what I’ve found so far

:; cat /etc/sv/my-way/run 
#!/bin/bash 
exec 2>&1
cd /home/my-way/my-way
. /usr/local/share/chruby/chruby.sh
chruby ruby-2.0.0 
export LANG=en_GB.UTF-8
exec chpst -u my-way -v bundle exec ruby -I lib bin/my-way.rb

:; sudo update-service --add /etc/sv/my-way
Service my-way added.

This is the script that starts the blog server, and the installation procedure thereof

Worthy of note:

  1. per convention, the run scripts (and attendant files) live in directories /etc/sv/someservicename, and these directories are are then symlinked into /etc/service by update-service
  2. chruby doesn’t run in sh, so we run this script under bash
  3. it redirects stderr to stdout so the svlog process (see below) can see it
  4. it runs as root up until the chpst invocation, so the ruby that you specify needs to be in /opt/rubies and not in /home/yourusualuser/.rubies. If you ran ruby-install under sudo it will have put it in the right place.
  5. runing bundle install with the --deployment flag when installing the ruby project will have sidestepped a whole class of “can’t find your gems” issues. So do that.

Next up is

:; cat /etc/sv/my-way/log/run 
#!/bin/sh
exec svlogd /var/log/my-way

This is the script that makes sure logs go somewhere. Specifically, they go to the file /var/log/my-way/current, which svlog is able (though as far as I know not yet configured) to rotate according to some defined criteria, and without needing to restart the server. The log files are owned by root, but maybe that’s changeable using chpst again.

:; sudo sv  status my-way
down: my-way: 94s, normally up; run: log: (pid 13620) 48806s
:; sudo sv  start my-way
ok: run: my-way: (pid 28343) 0s
:; sudo sv  status my-way
run: my-way: (pid 28343) 8s; run: log: (pid 13620) 48818s
:; pkill ruby
:; sudo sv  status my-way
run: my-way: (pid 28379) 31s; run: log: (pid 13620) 48949s
:; sudo sv  stop my-way
ok: down: my-way: 0s, normally up

And here’s how I start and stop it and stuff. Note that it magically restarted after I ran pkill ruby.

If you can read this, it works.

Syndicated 2014-01-19 11:18:31 from diary at Telent Netowrks

181 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!