Older blog entries for joey (starting at number 552)

the real Brazil

(click to enlarge)

"This is the real Brazil" -- Fernao

After 3 days of travel, including 22 hours driving from Brasilia to the coast of Bahia, I can't say much more. A few things not pictured above..

The two hours of car-swallowing potholes in an arrow-straight highway in a flat plain, playing chicken against oncoming trucks.

Night swim in the river in Correntina, followed by a whole grilled fish and ice cold guarana. Bliss.

The first sight of ocean waves and the drum circle last night.

Syndicated 2014-04-29 18:03:28 from see shy jo

propellor-driven DNS and backups

Took a while to get here, but Propellor 0.4.0 can deploy DNS servers and I just had it deploy mine. Including generating DNS zone files.

Configuration is dead simple, as far as DNS goes:

     & Dns.secondary hosts "joeyh.name"
                & Dns.primary hosts "example.com"
                        ( Dns.mkSOA "ns1.example.com" 100
                                [ NS (AbsDomain "ns1.example.com")
                                , NS (AbsDomain "ns2.example.com")
                        ) []

The awesome thing is that propellor fills in all the other information in the zone file by looking at the properties of the hosts it knows about.

 , host "blue.example.com"
        & ipv4 ""
        & ipv6 "fe80::26fd:52ff:feea:2294"

        & alias "example.com"
        & alias "www.example.com"
        & alias "example.museum"
        & Docker.docked hosts "webserver"
            `requres` backedup "/var/www"

When it sees this host, Propellor adds its IP addresses to the example.com DNS zone file, for both its main hostname ("blue.example.com"), and also its relevant aliases. (The .museum alias would go into a different zone file.)

Multiple hosts can define the same alias, and then you automaticlly get round-robin DNS.

The web server part of of the blue.example.com config can be cut and pasted to another host in order to move its web server to the other host, including updating the DNS. That's really all there is to is, just cut, paste, and commit!

I'm quite happy with how that worked out. And curious if Puppet etc have anything similar.

One tricky part of this was how to ensure that the serial number automtically updates when changes are made. The way this is handled is Propellor starts with a base serial number (100 in the example above), and then it adds to it the number of commits in its git repository. The zone file is only updated when something in it besides the serial number needs to change.

The result is nice small serial numbers that don't risk overflowing the (so 90's) 32 bit limit, and will be consistent even if the configuration had Propellor setting up multiple independent master DNS servers for the same domain.

Another recent feature in Propellor is that it can use Obnam to back up a directory. With the awesome feature that if the backed up directory is empty/missing, Propellor will automcatically restore it from the backup.

Here's how the backedup property used in the example above might be implemented:

backedup :: FilePath -> Property
backedup dir = Obnam.backup dir daily
    [ "--repository=sftp://rsync.example.com/~/webserver.obnam"
    ] Obnam.OnlyClient
    `requires` Ssh.keyImported SshRsa "root"
    `requires` Ssh.knownHost hosts "rsync.example.com" "root"
    `requires` Gpg.keyImported "1B169BE1" "root"

Notice that the Ssh.knownHost makes root trust the ssh host key belonging to rsync.example.com. So Propellor needs to be told what that host key is, like so:

 , host "rsync.example.com"
        & ipv4 ""
        & sshPubKey "ssh-rsa blahblahblah"

Which of course ties back into the DNS and gets this hostname set in it. But also, the ssh public key is available for this host and visible to the DNS zone file generator, and that could also be set in the DNS, in a SSHFP record. I haven't gotten around to implementing that, but hope at some point to make Propellor support DNSSEC, and then this will all combine even more nicely.

By the way, Propellor is now up to 3 thousand lines of code (not including Utility library). In 20 days, as a 10% time side project.

Syndicated 2014-04-19 07:08:45 from see shy jo

propellor introspection for DNS

In just released Propellor 0.3.0, I've improved improved Propellor's config file DSL significantly. Now properties can set attributes of a host, that can be looked up by its other properties, using a Reader monad.

This saves needing to repeat yourself:

hosts = [ host "orca.kitenet.net"
        & stdSourcesList Unstable
        & Hostname.sane -- uses hostname from above

And it simplifies docker setup, with no longer a need to differentiate between properties that configure docker vs properties of the container:

 -- A generic webserver in a Docker container.
    , Docker.container "webserver" "joeyh/debian-unstable"
        & Docker.publish "80:80"
        & Docker.volume "/var/www:/var/www"
        & Apt.serviceInstalledRunning "apache2"

But the really useful thing is, it allows automating DNS zone file creation, using attributes of hosts that are set and used alongside their other properties:

hosts =
    [ host "clam.kitenet.net"
        & ipv4 ""

        & cname "openid.kitenet.net"
        & Docker.docked hosts "openid-provider"

        & cname "ancient.kitenet.net"
        & Docker.docked hosts "ancient-kitenet"
    , host "diatom.kitenet.net"
        & Dns.primary "kitenet.net" hosts

Notice that hosts is passed into Dns.primary, inside the definition of hosts! Tying the knot like this is a fun haskell laziness trick. :)

Now I just need to write a little function to look over the hosts and generate a zone file from their hostname, cname, and address attributes:

extractZoneFile :: Domain -> [Host] -> ZoneFile
extractZoneFile = gen . map hostAttr
  where gen = -- TODO

The eventual plan is that the cname property won't be defined as a property of the host, but of the container running inside it. Then I'll be able to cut-n-paste move docker containers between hosts, or duplicate the same container onto several hosts to deal with load, and propellor will provision them, and update the zone file appropriately.

Also, Chris Webber had suggested that Propellor be able to separate values from properties, so that eg, a web wizard could configure the values easily. I think this gets it much of the way there. All that's left to do is two easy functions:

overrideAttrsFromJSON :: Host -> JSON -> Host

exportJSONAttrs :: Host -> JSON

With these, propellor's configuration could be adjusted at run time using JSON from a file or other source. For example, here's a containerized webserver that publishes a directory from the external host, as configured by JSON that it exports:

demo :: Host
demo = Docker.container "webserver" "joeyh/debian-unstable"
    & Docker.publish "80:80"
    & dir_to_publish "/home/mywebsite" -- dummy default
    & Docker.volume (getAttr dir_to_publish ++":/var/www")
    & Apt.serviceInstalledRunning "apache2"

main = do
    json <- readJSON "my.json"
    let demo' = overrideAttrsFromJSON demo
    writeJSON "my.json" (exportJSONAttrs demo')
    defaultMain [demo']

Syndicated 2014-04-11 05:05:54 from see shy jo

Kite: a server's tale

My server, Kite, is finishing its 20th year online.

It started as kite.resnet.cornell.edu, a 486 under the desk in my dorm room. Early on, it bounced around the DNS -- kite.ithaca.ny.us, kite.ml.org, kite.preferred.com -- before landing on kite.kitenet.net. The hardware has changed too, from a succession of desktop machines, it eventually turned into a 2u rack-mount server in the CCCP co-op. And then it went virtual, and international, spending a brief time in Amsterdam, before relocating to England and the kvm-hosting co-op.

Through all this change, and no few reinstalls from scratch, it's had a single distinct personality. This is a multi-user unix system, of the old school, carefully (and not-so-carefully) configured and administered to perform a grab-bag of functions. Whatever the users need.

I read the olduse.net hacknews newsgroup, and I see, in their descriptions of their server in 1984, the prototype of Kite and all its ilk.

It's consistently had a small group of users, a small subset of my family and friends. Not quite big enough to really turn into a community, and we wall and talk less than we once did.

Exhibit: Kite as it appeared in the 90's

[Intentionally partially broken, being able to read the cgi source code is half the fun.]

Kite was an early server on the WWW, and garnered mention in books and print articles. Not because it did anything important, but because there were few enough interesting web sites that it slightly stood out.

Many times over these 20 years I've wondered what will be the end of Kite's story. It seemed like I would either keep running it indefinitely, or perhaps lose interest. (Or funding -- it's eaten a lot of cash over the years, especially before the current days of $5/month VPS hosting.) But I failed to anticipate what seems to really be happening to it. Just as I didn't fathom, when kite was perched under my desk, that it would one day be some virtual abstract machine in a unknown computer in anther country.

Now it seems that what will happen to Kite is that most of the important parts of it will split off into a constellation of specialized servers. The website, including the user sites, has mostly moved to branchable.com. The DNS server, git server and other crucial stuff is moving to various VPS instances and containers. (The exhibit above is just one more automatically deployed, soulless container..) A large part of Kite has always been about me playing with bleeding-edge stuff and installing random new toys; that has moved to a throwaway personal server at cloudatcost.com which might be gone tomorrow (or might keep running for free for years).

What it seems will be left is a shell box, with IMAP access to a mail server, and a web server for legacy /~user/ sites, and a few tools that my users need (including that pine program some of them are still stuck on.)

Will it be worth calling that Kite?

[ Kite users: This transition needs to be done by December when the current host is scheduled to be retired. ]

Syndicated 2014-04-10 15:17:38 from see shy jo

propellor type-safe reversions

Propellor ensures that a list of properties about a system are satisfied. But requirements change, and so you might want to revert a property that had been set up before.

For example, I had a system with a webserver container:

  Docker.docked container hostname "webserver"

I don't want a web server there any more. Rather than having a separate property to stop it, wouldn't it be nice to be able to say:

  revert (Docker.docked container hostname "webserver")

I've now gotten this working. The really fun part is, some properies support reversion, but other properties certianly do not. Maybe the code to revert them is not worth writing, or maybe the property does something that cannot be reverted.

For example, Docker.garbageCollected is a property that makes sure there are no unused docker images wasting disk space. It can't be reverted. Nor can my personal standardSystem Unstable property, which amoung other things upgrades the system to unstable and sets up my home directory..

I found a way to make Propellor statically check if a property can be reverted at compile time. So revert Docker.garbageCollected will fail to type check!

The tricky part about implementing this is that the user configures Propellor with a list of properties. But now there are two distinct types of properties, revertable ones and non-revertable ones. And Haskell does not support heterogeneous lists..

My solution to this is a typeclass and some syntactic sugar operators. To build a list of properties, with individual elements that might be revertable, and others not:

        & standardSystem Unstable
        & revert (Docker.docked container hostname "webserver")
        & Docker.docked container hostname "amd64-git-annex-builder"
        & Docker.garbageCollected

Syndicated 2014-04-02 17:09:02 from see shy jo

adding docker support to propellor

Propellor development is churning away! (And leaving no few puns in its wake..)

Now it supports secure handling of private data like passwords (only the host that owns it can see it), and fully end-to-end secured deployment via gpg signed and verified commits.

And, I've just gotten support for Docker to build. Probably not quite work, but it should only be a few bugs away at this point.

Here's how to deploy a dockerized webserver with propellor:

host hostname@"clam.kitenet.net" = Just
    [ Docker.configured
    , File.dirExists "/var/www"
    , Docker.hasContainer hostname "webserver" container

container _ "webserver" = Just $ Docker.containerFromImage "joeyh/debian-unstable"
        [ Docker.publish "80:80"
        , Docker.volume "/var/www:/var/www"
        , Docker.inside
            [ serviceRunning "apache2"
                `requires` Apt.installed ["apache2"]

Docker containers are set up using Properties too, just like regular hosts, but their Properties are run inside the container.

That means that, if I change the web server port above, Propellor will notice the container config is out of date, and stop the container, commit an image based on it, and quickly use that to bring up a new container with the new configuration.

If I change the web server to say, lighttpd, Propellor will run inside the container, and notice that it needs to install lighttpd to satisfy the new property, and so will update the container without needing to take it down.

Adding all this behavior took only 253 lines of code, and none of it impacts the core of Propellor at all; it's all in Propellor.Property.Docker. (Well, I did need another hundred lines to write a daemon that runs inside the container and reads commands to run over a named pipe... Docker makes running ad-hoc commands inside a container a PITA.)

So, I think that this vindicates the approach of making the configuration of Propellor be a list of Properties, which can be constructed by abitrarily interesting Haskell code. I didn't design Propellor to support containers, but it was easy to find a way to express them as shown above.

Compare that with how Puppet supports Docker: http://docs.docker.io/en/latest/use/puppet/

docker::run { 'helloworld':
  image        => 'ubuntu',
  command      => '/bin/sh -c "while true; do echo hello world; sleep 1; done"',
  ports        => ['4444', '4555'],

All puppet manages is running the image and a simple static command inside it. All the complexities that puppet provides for configuring servers cannot easily be brought to bear inside the container, and a large reason for that is, I think, that its configuration file is just not expressive enough.

Syndicated 2014-04-01 08:22:41 from see shy jo


Whups, I seem to have built a configuration management system this evening!

Propellor has similar goals to chef or puppet or ansible, but with an approach much more like slaughter. Except it's configured by writing Haskell code.

The name is because propellor ensures that a system is configured with the desired PROPerties, and also because it kind of pulls system configuration along after it. And you may not want to stand too close.

Disclaimer: I'm not really a sysadmin, except for on the scale of "diffuse administration of every Debian machine on planet earth or nearby", and so I don't really understand configuration management. (Well, I did write debconf, which claims to be the "Debian Configuration Management system".. But I didn't understand configuration management back then either.)

So, propellor makes some perhaps wacky choices. The least of these is that it's built from a git repository that any (theoretical) other users will fork and modify; a cron job can re-make it from time to time and pull down configuration changes, or something can be run to push changes.

A really simple configuration for a Tor bridge server using propellor looks something like this:

main = ensureProperties
    [ Apt.stdSourcesList Apt.Stable `onChange` Apt.upgrade
    , Apt.removed ["exim4"] `onChange` Apt.autoRemove
    , Hostname.set "bridget"
    , Ssh.uniqueHostKeys
    , Tor.isBridge

Since it's just haskell code, it's "easy" to refactor out common configurations for classes of servers, etc. Or perhaps integrate reclass? I don't know. I'm happy with just pure functions and type-safe refactorings of my configs, I think.

Properties are also written in Haskell of course. This one ensures that all the packages in a list are installed.

installed :: [Package] -> Property
installed ps = check (isInstallable ps) go
        go = runApt $ [Param "-y", Param "install"] ++ map Param ps

Here's one that ensures the hostname is set to the desired value, which shows how to specify content for a file, and also how to run another action if a change needed to be made to satisfy a property.

set :: HostName -> Property
set hostname = fileHasContent "/etc/hostname" [hostname]
        `onChange` cmdProperty "hostname" [Param hostname]

Here's part of a custom one that I use to check out a user's home directory from git. Shows how to make a property require that some other property is satisfied first, and how to test if a property has already been satisfied.

installedFor :: UserName -> Property
installedFor user = check (not <$> hasGitDir user) $
        IOProperty ("githome " ++ user) (go =<< homedir user)
                    `requires` Apt.installed ["git", "myrepos"]
    go ... -- 12 lines elided

I'm about 37% happy with the overall approach to listing properties and combining properties into larger properties etc. I think that some unifying insight is missing -- perhaps there should be a Property monad? But as long as it yields a list of properties, any smarter thing should be able to be built on top of this.

Propellor is 564 lines of code, including 25 or so built-in properties like the examples above. It took around 4 hours to build.

I'm pretty sure it was easier to write it than it would have been to look into ansible and salt and slaughter (and also liw's human-readable configuration language whose name I've forgotten) in enough detail to pick one, and learn how its configuration worked, and warp it into something close to how I wanted this to work.

I think that's interesting.. It's partly about NIH and I-want-everything-in-Haskell, but it's also about a complicated system that is a lot of things to a lot of people -- of the kind I see when I look at ansible -- vs the tools and experience to build just the thing you want without the cruft. Nice to have the latter!

Syndicated 2014-03-30 07:51:59 from see shy jo

subverting hierarchy with git

Did you ever want to be able to tag your files, and use the tags to query and select the files you want? For many sorts of files we use, this is clearly better than being locked into a single hierarchial view of nested directories.

Semantic file systems are a way to do things like this. But that's an entire separate filesystem, often implemented as a FUSE layer. I have never wanted tagging enough to take on that layer of complications.

A week ago I realized that there was a way to do this without using an entirely separate filesystem. When we use git, we're used to having different views of our files, called branches, that we switch between.

What then, if a git branch were generated from tags and other metadata that meet a query, providing a custom view that meet the user's current needs and could be incrementally refined.

Here's a short screencast demoing these views.
And here's a walkthrough for setting it up.

To build this, I needed a nice way to store metadata in git, and since git-annex happens to have a very nice way of using git as a database, I naturally built it on top of git-annex.

This means that the tags and other metadata are automatically synchronized between different clones of the repository. Multiple users can be tagging and setting other metadata at the same time, and their changes will merge in a consistent way.

I have a long list of things to do to fully integrate views into git-annex. However, they're already basically usable, and I'm very pleased with how these dymanic views of the contents of a repository are working out.

Today's release of git-annex includes views support, so give them a try!

Syndicated 2014-02-21 18:04:45 from see shy jo

what charlie's missing about bitcoin

(Posted as a comment to Charles Stross's blog post about bitcoin)

Bitcoin is a piece of software which tries to implement a particular SFnal future. One in which the world currency is de-centralized, deflationary, all early bitcoin adopters own their own planetoids, and all visitors are automatically charged for the air they breath.

Thing is, the real world is more complicated than that. Assuming Bitcoin did manage to become an important currency, countries would naturally try to regulate it. In 30 years, by the time bitcoin mining has slowed right down, the legal system will be fully caught up to the internet.

Bitcoin tries to make its code the law (as Lessig used to say), but the law can certainly affect its code.

The law could, for example, require that bitcoin be changed to stop increasing the difficulty of mining new blocks. Then bitcoin is suddenly an inflationary currency. This would be a hard fork in the block chain, but one enforced by financial regulators. Miners would be tracked down and forced to comply. Some would perhaps go underground and run the deflationary bitcoin network on TOR hidden services. Lots of possible ways it could play out.

That's only one scenario, covering one of the many problems with Bitcoin that make Charlie hate it. So it seems to me that Bitcoin should be a gold mine for Science Fiction authors, if nothing else..

Syndicated 2013-12-18 20:40:23 from see shy jo

completely linux distribution-independent packaging

Sometimes it makes sense to ship a program to linux users in ready-to-run form that will work no matter what distribution they are using. This is hard.

Often a commerical linux game will bundle up a few of the more problimatic libraries, and ship a dynamic executable that still depends on other system libaries. These days they're building and shipping entire Debian derivatives instead, to avoid needing to deal with that.

There have been a few efforts to provide so-called one click install package systems that AFAIK, have not been widely used. I don't know if they generally solved the problem.

More modern appoaches seem to be things like docker, which move the application bundle into a containerized environment. I have not looked at these, but so far it does not seem to have spread widely enough to be a practical choice if you're wanting to provide something that will work for a majority of linux users.

So, I'm surprised that I seem to have managed to solve this problem using nothing more than some ugly shell scripts.

My standalone tarballs of git-annex now seem fairly good at running on a very wide variety of systems.

For example, I unpacked the tarball into the Debian-Installer initramfs and git-annex could run there. I can delete all of /usr and it keeps working! All it needs is a basic sh, which even busybox provides.

Looks likely that the new armel standalone tarball of git-annex will soon be working on embedded systems as odd as the Synology NAS, and it's already been verified to work on Raspbian. (I'm curious if it would work on Android, but that might be a stretch.)

Currently these tarballs are built for a specific architecture, but there's no particular reason a single one couldn't combine binaries built for each supported architecture.

technical details

The main trick is to ship a copy of ld-linux.so, as well as all the glibc libraries and associated files, and of course every other library and file the application needs.

Shipping ld-linux.so lets a shell script wrapper be made around each binary, that runs ld-linux.so and passes it the library directories to search. This way the binary can be run, bypassing the system's own dynamic linker (which might not like it) and using the included glibc.

For example a shell script that runs the git binary from the bundle:

  exec "$GIT_ANNEX_LINKER" --library-path "$GIT_ANNEX_LD_LIBRARY_PATH" "$GIT_ANNEX_SHIMMED/git/git" "$@"

I have to set quite a lot of environment variables, to avoid using any files from the system and instead use ones from my tarball. One important one is GCONV_PATH. Note that LD_LIBRARY_PATH does not have to be set, and this is nice because it allows running a few programs from the host system, such as its web browser.

worse is better

Of course I'll take a proper distribution package anytime over this.

Still, it seems to work quite well, in all the horrible cases that require it.

Syndicated 2013-12-17 01:38:16 from see shy jo

543 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!