Older blog entries for joey (starting at number 543)

completely linux distribution-independent packaging

Sometimes it makes sense to ship a program to linux users in ready-to-run form that will work no matter what distribution they are using. This is hard.

Often a commerical linux game will bundle up a few of the more problimatic libraries, and ship a dynamic executable that still depends on other system libaries. These days they're building and shipping entire Debian derivatives instead, to avoid needing to deal with that.

There have been a few efforts to provide so-called one click install package systems that AFAIK, have not been widely used. I don't know if they generally solved the problem.

More modern appoaches seem to be things like docker, which move the application bundle into a containerized environment. I have not looked at these, but so far it does not seem to have spread widely enough to be a practical choice if you're wanting to provide something that will work for a majority of linux users.

So, I'm surprised that I seem to have managed to solve this problem using nothing more than some ugly shell scripts.

My standalone tarballs of git-annex now seem fairly good at running on a very wide variety of systems.

For example, I unpacked the tarball into the Debian-Installer initramfs and git-annex could run there. I can delete all of /usr and it keeps working! All it needs is a basic sh, which even busybox provides.

Looks likely that the new armel standalone tarball of git-annex will soon be working on embedded systems as odd as the Synology NAS, and it's already been verified to work on Raspbian. (I'm curious if it would work on Android, but that might be a stretch.)

Currently these tarballs are built for a specific architecture, but there's no particular reason a single one couldn't combine binaries built for each supported architecture.

technical details

The main trick is to ship a copy of ld-linux.so, as well as all the glibc libraries and associated files, and of course every other library and file the application needs.

Shipping ld-linux.so lets a shell script wrapper be made around each binary, that runs ld-linux.so and passes it the library directories to search. This way the binary can be run, bypassing the system's own dynamic linker (which might not like it) and using the included glibc.

For example a shell script that runs the git binary from the bundle:

  exec "$GIT_ANNEX_LINKER" --library-path "$GIT_ANNEX_LD_LIBRARY_PATH" "$GIT_ANNEX_SHIMMED/git/git" "$@"

I have to set quite a lot of environment variables, to avoid using any files from the system and instead use ones from my tarball. One important one is GCONV_PATH. Note that LD_LIBRARY_PATH does not have to be set, and this is nice because it allows running a few programs from the host system, such as its web browser.

worse is better

Of course I'll take a proper distribution package anytime over this.

Still, it seems to work quite well, in all the horrible cases that require it.

Syndicated 2013-12-17 01:38:16 from see shy jo

2013 git-annex user survey

Similar to the yearly git user survey, I am doing a 2013 git-annex user survey.

If you use git-annex, please take a few minutes to answer my questions!


Since this blog post is too short, let me also announce a minor spinoff project from git-annex that I have recently released.

git-repair is a complement to git fsck that can fix up arbitrarily damaged git repositories. As well as avoiding the need to rm -rf a damaged repository and re-clone, using git-repair can help rescue commits you've made to the damaged repository and not yet pushed out.

I've been testing git-repair with evil code that damages git repositories in random ways. It has now successfully repaired tens of thousands of damaged repositories. In the process, I have found some bugs in git itself.

Syndicated 2013-11-22 17:16:57 from see shy jo

new and old laptops

I've had a new laptop for a couple months. It's a lot larger and faster than my old Dell Mini 9 netbook, which I somewhat famously used exclusively for 5 years.

The new laptop is a Lenovo Yoga 11, and its screen can bend all the way back and around to under the keyboard, converting it to a tablet. (Held together with some quite powerful magnets, it turns out.. when did those become ok to have around computer equiment?) This is a feature I've always felt laptops should have, especially after the Mini 9, which couldn't even open out to flat. Although since the accelerometer is not yet working under Linux, and Linux GUIs are not very well suited for tablets, I have so far mostly used the feature for a) reading and b) cleaning that usually hard to reach area around the hinge.

Anyway, this is a laptop that wants to be a tablet, including all the bad parts, like a hard to remove battery. (20-some torx screws, according to the service manual.) Yesterday I made the mistake of running down the battery all day, and when I plugged it in, after a gloomy grey day, there was not enough oomph in the house's battery bank to recharge a hungry L-ion battery.

So I was stuck using my old laptop for several hours, with its battery removed, and its lack of fan making it a nice warm lump. It felt kind of like driving around in an old VW bus, everything is awkward and slow, and yet somehow also charming, and at the end you wonder how you put up with it for so long.

These machines and the ways they influence us..

Syndicated 2013-11-18 16:14:58 from see shy jo

license monads

This could have been a blog post about toothbrushes for monkeys, which I seem to have dreamed about this morning. But then I was vaguely listening to the FAIFCast in the car, and I came up with something even more esoteric to write about!

License monads would allow separating parts of code that are under different licenses, in a rigorous fashion. For example, we might have two functions in different license monads:

foo :: String -> GPL String

bar :: Char -> BSD Char

Perhaps foo needs to use bar to calculate its value. It can only do so if there's a way to lift code in the BSD monad into the GPL monad. Which we can legally write, since the BSD license is upwards-compatable with the GPL:

liftGPL :: BSD a -> GPL a

On the other hand, there should be no way provided to lift the GPL monad into the BSD monad. So bar cannot be written using code from foo, which would violate foo's GPL license.

Perhaps the reason I am thinking about this is that the other day I found myself refactoring some code out of the git-annex webapp (which is AGPL) licensed and into the git-annex assistant (which is GPL licensed). Which meant I had to relicense that code.

Luckily that was easy to do legally speaking, since I am the only author of the git-annex webapp so far, and own the whole license of it. (Actually, it's only around 3 thousand lines of code, and another thousand of html.)

It also turned out to be easy to do the refactoring, technically speaking because looking at the code, I realized I had accidentially written it in the wrong monad; all the functions were in the webapp's Handler monad, but all of them used liftAnnex to actually do their work in the Annex monad. If that had not been the case, I would not have been able to refactor the code, at least not without entirely rewriting it.

It's as if I had accidentially written:

foo :: String -> GPL String
foo = mapM (liftGPL . bar)

Which can be generalized to:

foo :: String -> BSD String
foo = mapM bar

I don't think that license monads can be realistically used in the current world, because lawyers and math often don't mix well. Lawyers have, after all, in the past written laws re-defining pi to be 3. Still, license monads are an interesting way to think about things for myself. They capture how my code and licenses are structured and allow me to reason about it on a more granular level than the licences of individual files.

(I have a vague feeling someone else may have written about this idea before? Perhaps I should have been blogging about monkey toothbrushes after all...)

Syndicated 2013-10-30 15:17:57 from see shy jo

insured

Here in the US, the Affordable Care Act is finally going into effect, with accompanying drama. I managed to get signed up today at healthcare.gov. After not having health insurance since 2000, I will finally be covered starting January 1 2014.

Since my income is mosty publically known anyway, I thought it might be helpful to blog about some details.

I was uninsured for 14 years due to a combination of three factors:

  1. Initially, youthful stupidity and/or a perfectly resonable cost/benefit analysis. (Take your pick.)
  2. Due to the US health insurance system being obviously broken, and my preference to avoid things that are broken. Especially when the breakage involved insurers refusing to cover me at any sane level due to a minor and easily controlled pre-existing condition.
  3. Since I'm not much motivated by income levels, and am very motivated to have time to work on things that are important to me, my income has been on average pretty low, and perhaps more importantly, I have intentionally avoided being a full-time employee of anyone at any point in the past 14 years (have rejected job offers), and so was not eligible for any employee plans which were the only reasonable way to be covered in the US. (See point #2.)

So, if you're stuck waiting in line on healthcare.gov (is this an entirely new online experience brought to us by the US government?), or have seen any of the dozen or so failure modes that I saw just trying to register for a login to the site, yeah, it's massively overloaded right now, and it's also quite broken on a number of technical levels. But you can eventually get though it.

Based on some of the bugs I saw, it may help to have an large number of email addresses and use a different one for each application attempt. It also wouldn't hurt to write some programs to automate the attempts, because otherwise you may have to fill out the same form dozens of times. And no, you can't use "you+foo@bar.com" for your email; despite funding the development of RFC-822 in the 80's, the US government is clueless about what consititutes a valid email address.

But my "favorite" misfeature of the site is that it refuses to let you enter any accented characters, or characters not in the latin alphabet when signing up. Even if they're, you know, part of your name. (Welcome back to Ellis Island..) I want to check the git repository to see if I can find the backstory for these and other interesting technical decisions, but they have forgotten to push anything to it for over 3 months.

The good news is that once you get past the initial signup process, and assuming you get the confirmation mail before the really short expiration period of apparently less than 1 hour (another interesting technical choice, given things like greylisting), the actual exchange is not badly overloaded, and nor is it very buggy (comparatively). I was able to complete an application in about an hour.

The irony is that after all that, I was only able to choose from one health insurer covering my area on the so-called "exchange". (Blue Cross/Blue Shield) I also signed up for dental insurance (it was a welcome surprise that the site offers this at all) and had a choice of two insurers for that.

The application process was made more uncertian for me since I have no idea what I'll end up doing for money once my current crowdsourced year of work is done. The site wants you to know how much income you'll have in 2014, and my guess is anywhere between $6000 (from a rental property) and about what I made this year (approx $25000 before taxes). Or up, if I say, answered the Google pings. The best choice seemed to be to answer what I made this year, which is also close to what I made last year.

So, I'll be paying around $200/month for a combination of not very good health insurance, and not very good dental insurance. There is around $750/year of financial aid to people at my guesstimated 2014 income level, which would drop that to $140/month, but I will let them refund me whatever that turns out to be in a lump sum later instead.

For comparison, I am lucky to spend rather less renting a three bedroom house situated in 25 acres of woods..

It's strange to think that all of this is an improvement to the system here in the US, especially given all the better options that could have been passed instead, but it seems that it probably is. Especially when I consider the many people around me who are less fortunate than myself. If you'd like a broader perspective on this, see Tobias Buckell's "American healthcare was already socialized by Reagan, we’re just fighting about how to pay for it".

Syndicated 2013-10-02 22:33:10 from see shy jo

a special announcement from RMS

A little grep has told me that RMS is going to announce something special on olduse.net in two weeks time, on 27 September 1983. Best not to speculate about the details, but this announcement will be something people are still talking about thirty years later. So it's an event you and your news reader won't want to miss.

I have set up an IRC channel #olduse.net on FreeNode, and we'll see who first spots the post!

Syndicated 2013-09-13 23:21:41 from see shy jo

a rope ladder to the dgit treehouse

Ian has said something along the lines of dgit being a private club; if you're not in the club the strange git-centric views of its members don't affect you. With that said, dgit is now a quite usable and nice tool and we would like more people in the club.

To that end, here's one way to get into our little world where the Debian source repository is really a collection of git repositories. I recently made a new package and added it to Debian, and here is every command I typed to do it (other than vim, lintian, and dpkg -i). As you read through, notice that every command has to do with git in one way or another.

git clone https://github.com/blake2-ppc/git-remote-gcrypt

This software makes git be able to push to encrypted git remotes. Very neat stuff.

git archive --format=tar origin/master -o | gzip > ../git-remote-gcrypt_0.20130908.orig.tar.gz

Like most software on github, this one has no upstream tarball; its git repository is its native source format. Needing to manually generate this tarball appendix is the only unpleasant part of the process.

git add debian
git commit -m debianization

Had to write 48 lines of control file and stuff to make the package build.

dgit build -tc

This just runs dpkg-buildpackage. But you need to commit all files before running it.

dgit push --new

The --new is needed because this is a new package. This created the git repository for the package on alioth and pushed my changes and the upstream history to it, as well as uploadng the package to incoming.

Syndicated 2013-09-09 00:33:24 from see shy jo

best DebConf ever

For me, DebConf13 in Switzerland was a perfect DebConf, and generally an great developer meeting and conference. I've attended several past DebConfs in Canada, Norway, Brazil, Finland, Mexico, Scotland, Spain, NYC, Bosnia, and Nicaragua. I have special memories of each. DebConf13 was ideal in every way.

Let's start with the camping. Only a minority of us did it, and my campsite was located in a small field behind a bar that was often noisy late into the night, when I was trying to rest and preserve my European jetlag remedy of inverted sleeping schedule. Still, I love sleeping cocooned in a tent, and going through the routines of camping and spending time alone there was a good counterpoint to DebConf. I generally find myself retreating for a day or two in the middle of a week-long conference, burnt out with social interactions. That didn't happen this time. I need to find way to camp at future DebConfs.

(Also special shout out to wonderful people who lent me a warmer sleeping bag, and bought me a pillow while on a trip to town! Upgraded my first backpacking in Europe experience's comfort level a lot!)

The next perfect thing for me were the talks I gave at DebConf. I always enjoy presenting at DebConf, and barely remember when I'd get nervous doing it. But I really felt comfortable giving these talks in a way I've not before. I'm pleased with how the Debian Cosmology talk turned out; I'd worried it would be a downer talk, or too light, or not well received, and none of those problems surfaced.

I was especially happy with my dh_busfactor session, which I prepared for all of 10 minutes and was structured entirely to benefit me by talking over my thoughts and getting feedback, and in which I probably drove the camerapeople crazy running my own mic into the audience. Despite all that, I think it succeeded being interesting for the participants as well as for me. Hope to follow up on it soon..

But beyond the two talks I submitted, there were a series of ad-hoc scheduled talks. It seemed that every day I spent at least an hour discussing something I'm deeply involved in. This started with the Ikiwiki bof, and then there was an epic series of (unrecorded) BoFs about git stuff, which basically turned into an unofficial git track at DebConf. Much, much discussion of git-annex and other things I'm involved in.

BTW, I really like that DebConf, as well as being a formal conference on top, is really an un-conference underneath, with new BoFs and things popping up on the schedule all week long. You can even have completely unscheduled meetings that somehow get just the right mix of people to make something awesome happen.

But it wasn't all about talks. Indeed, I only attended 2 or 3 sessions a day. The DebConf13 venue was controversial, but I found it excellent for getting things done. Rather than one or two massive rooms that get too noisy to think in, this venue had many little nooks and crannies suited for different types of work. There were so many I only discovered some while walking down the hill as I was leaving the venue! Anchoring it all was the excellent view, and I particularly enjoyed the "scenic hacklab".

I was particularly impressed with the flexability of the venue's management. Someone asked if we could have cheese for breakfast, and every breakfast thereafter had not one but often three excellent cheeses. One day we'd be complaining that there were not enough vegetables in the food, and the next we'd be eating an excellent vegetarian Swiss lunch. They even apparently relaxed stringent Swiss rules on not allowing tents in sightlines visible from people further up the hill. They were clearly happy to welcome our crazy gathering to Le Camp. They let a DD borrow and use a chain saw and build a bonfire!

The venue also encouraged getting out and exploring the surrounding countryside. Too often I end up spending the whole DebConf without seeing much of the area around. This time I at least got down the hill to see the castle in Vaumarcus a few times, and enjoyed many evening walks amoung the grape fields. The conference dinner (on a boat!) tied into this well too. And the daytrip to CERN was a great addition. We got to poke around the control center for a massive particle accellerator! My only regret is we didn't get a chance to actually go inside the LHC tunnels.

Back to DebConf, another thing I really liked about this one was that its location allowed several upstream authors of software in Debian (other than the usual suspects who are both upstreams and Debian Developers) to attend. I don't know if Lennart Poettering changed any minds about systemd with his talk, but I was happy for the opportunity to buy him a beer. I didn't get to Monty's talk, but it was excellent to see him attend as well.

Finally, I got to meet old friends I'd not seen in years, and new ones I've not met in real life before. We're still not sure where this whole DebConf thing is going -- will it be all about the cheese and wine party and greybeards in 20 years time? But one thing we know is we enjoy our little slice of the year where kilts and coding are equally unremarkable, where problems that seem intractable online somehow solve themselves in person, where we come away with ideas for an entire year and memories for a lifetime.

Syndicated 2013-08-30 21:53:10 from see shy jo

southern fried science fiction with plum sauce

Had an afternoon of steak and science fiction. Elysium is only so-so, but look what we found in a bookstore that was half religious materials and half SF, local books, and carefully hidden romance novels:

Star Wars, Star Trek and the 21st Century Christians

Best part was at the end, when I finally found one of the local asian markets Tomoko tells us about when she casually pulls out the good stuff at family gatherings. I will be back for whole ducks, fresh fish, squid, 50 lb bags of rice, tamarind paste, fresh ginger that has not sat on the shelf for 2 months because only I buy it, etc. Only an hour from home in the woods! Between the garlic sprouts, bean sprouts, enoki mushrooms, etc that I got for $10 today and this week's CSA surprise of 18 inch snake beans and smoked pork knuckles, I have the epic stir fry potential..

Spock and R2D2

Syndicated 2013-08-25 22:25:16 from see shy jo

idea: git push requests

This is an idea that Keith Packard told me. It's a brilliant way to reduce GitHub's growing lockin, but I don't know how to implement it. And I almost forgot about it, until I had another annoying "how do I send you a patch with this amazing git technology?" experience and woke up with my memory refreshed.

The idea is to allow anyone to git push to any anonymous git:// repository. But the objects pushed are not stored in a public part of the repository (which could be abused). Instead the receiving repository emails them off to the repository owner, in a git-am-able format.

So this is like a github pull request except it can be made on any git repository, and you don't have to go look up the obfuscated contact email address and jump through git-format-patch hoops to make it. You just commit changes to your local repository, and git push to wherever you cloned from in the first place. If the push succeeds, you know your patch is on its way for review.

Keith may have also wanted to store the objects in the repository in some way that a simple git command run there could apply them without the git-am bother on the receiving end. I forget. I think git-am would be good enough -- and including the actual diffs in the email would actually make this far superior to github pull request emails, which are maximally annoying by not doing so.


Hmm, I said I didn't know how to implement this, but I do know one way. Make the git-daemon run an arbitrary script when receiving a push request. A daemon.pushscript config setting could enable this.

The script could be something like this:

#!/bin/sh
set -e
tmprepo="$(mktemp -d)"
# this shared clone is *really* fast even for huge repositories, and uses
# only a few 100 kb of disk space!
git clone --shared --bare "$GIT_DIR" "$tmprepo"
git-receive-pack "$tmprepo"
# XXX add email sending code here.
rm -rf "$tmprepo"

Of course, this functionality could be built into the git-daemon too. I suspect a script hook and an example script in contrib/ might be an easier patch to get accepted into git though.

That may be as far as I take this idea, at least for now..

Syndicated 2013-08-25 13:07:32 from see shy jo

534 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!