joey is currently certified at Master level.

Name: Joey Hess
Member since: 2000-03-06 23:42:41
Last Login: 2011-12-31 20:04:52

FOAF RDF Share This

Homepage: http://kitenet.net/~joey

Projects

Recent blog entries by joey

Syndication: RSS 2.0

letsencrypt support in propellor

I've integrated letsencrypt into propellor today.

I'm using the reference letsencrypt client. While I've seen complaints that it has a lot of dependencies and is too complicated, it seemed to only need to pull in a few packages, and use only a few megabytes of disk space, and it has fewer options than ls does. So seems fine. (Although it would be nice to have some alternatives packaged in Debian.)

I ended up implementing this:

  letsEncrypt :: AgreeTOS -> Domain -> WebRoot -> CertInstaller -> Property NoInfo

The interesting part of that is the CertInstaller, which is passed the certificate files that letsencrypt generates, and is responsible for making the web server (or whatever) use them.

This avoids relying on the letsencrypt client's apache config munging, which is probably useful for many people, but not those of us using configuration management systems. And so avoids most of the complicated magic that the letsencrypt client has a reputation for.

And, this API lets other propellor properties integrate with letsencrypt by providing a CertInstaller of their own. Like this property, which sets up apache to serve a https website, using letsencrypt to get the certificate:

  Apache.httpsVirtualHost "example.com" "/var/www"
    (LetsEncrypt.AgreeTos (Just "me@my.domain"))

That's about as simple a configuration as I can imagine for such a website!


The two parts of letsencrypt that are complicated are not the fault of the client really. Those are renewal and rate limiting.

I'm currently rate limited for the next week because I asked letsencrypt for several certificates for a domain, as I was learning how to use it and integrating it into propellor. So I've not quite managed to fully test everything. That's annoying. I also worry that rate limiting could hit at an inopportune time once I'm relying on letsencrypt. It's especially problimatic that it only allows 5 certs for subdomains of a given domain per week. What if I use a lot of subdomains?

Renewal is complicated mostly because there's no good way to test it. You set up your cron job, or whatever, and wait three months, and hopefully it worked. Just as likely, you got something wrong, and your website breaks. Maybe letsencrypt could offer certificates that will only last an hour, or a day, for use when testing renewal.

Also, what if something goes wrong with renewal? Perhaps letsencrypt.org is not available when your certificate needs to be renewed.

What I've done in propellor to handle renewal is, it runs letsencrypt every time, with the --keep-until-expiring option. If this fails, propellor will report a failure. As long as propellor is run periodically by a cron job, this should result in multiple failure reports being sent (for 30 days I think) before a cert expires without getting renewed. But, I have not been able to test this.

Syndicated 2016-02-07 22:10:20 from see shy jo

git-annex v6

Version 6 of git-annex, released last week, adds a major new feature; support for unlocked large files that can be edited as usual and committed using regular git commands.

For example:

  git init
git annex init --version=6
mv ~/foo.iso .
git add foo.iso
git commit -m "added hundreds of megabytes to git annex (not git)"
git remote add origin ssh://sever/dir
git annex sync origin --content # uploads foo.iso

Compare that with how git-annex has worked from the beginning, where git annex add is used to add a file, and then the file is locked, preventing further modifications of it. That is still a very useful way to use git-annex for many kinds of files, and is still supported of course. Indeed, you can easily switch files back and forth between being locked and unlocked.

This new unlocked file mode uses git's smudge/clean filters, and I was busy developing it all through December. It started out playing catch-up with git-lfs somewhat, but has significantly surpassed it now in several ways.

So, if you had tried git-annex before, but found it didn't meet your needs, you may want to give it another look now.


Now a few thoughts on git-annex vs git-lfs, and different tradeoffs made by them.

After trying it out, my feeling is that git-lfs brings an admirable simplicity to using git with large files. File contents are automatically uploaded to the server when a git branch is pushed, and downloaded when a branch is merged, and after setting it up, the user may not need to change their git workflow at all to use git-lfs.

But there are some serious costs to that simplicity. git-lfs is a centralized system. This is especially problimatic when dealing with large files. Being a decentralized system, git-annex has a lot more flexability, like transferring large file contents peer-to-peer over a LAN, and being able to choose where large quantities of data are stored (maybe in S3, maybe on a local archive disk, etc).

The price git-annex pays for this flexability is you have to configure it, and run some additional commands. And, it has to keep track of what content is located where, since it can't assume the answer is "in the central server".

The simplicity of git-lfs also means that the user doesn't have much control over what files are present in their checkout of a repository. git-lfs downloads all the files in the work tree. It doesn't have facilities for dropping the content of some files to free up space, or for configuring a repository to only want to get a subset of files in the first place. On the other hand, git-annex has excellent support for alll those things, and this comes largely for free from its decentralized design.

If git has showed us anything, it's perhaps that a little added complexity to support a fully distributed system won't prevent people using it. Even if many of them end up using it in a mostly centralized way. And that being decentralized can have benefits beyond the obvious ones.


Oh yeah, one other advantage of git-annex over git-lfs. It can use half as much disk space!

A clone of a git-lfs repository contains one copy of each file in the work tree. Since the user can edit that file at any time, or checking out a different branch can delete the file, it also stashes a copy inside .git/lfs/objects/.

One of the main reasons git-annex used locked files, from the very beginning, was to avoid that second copy. A second local copy of a large file can be too expensive to put up with. When I added unlocked files in git-annex v6, I found it needed a second copy of them, same as git-lfs does. That's the default behavior. But, I decided to complicate git-annex with a config setting:

  git config annex.thin true
git annex fix

Run those two commands, and now only one copy is needed for unlocked files! How's it work? Well, it comes down to hard links. But there is a tradeoff here, which is why this is not the default: When you edit a file, no local backup is preserved of its old content. So you have to make sure to let git-annex upload files to another repository before editing them or the old version could get lost. So it's a tradeoff, and maybe it could be improved. (Only thin out a file after a copy has been uploaded?)

This adds a small amount of complexity to git-annex, but I feel it's well worth it to let unlocked files use half the disk space. If the git-lfs developers are reading this, that would probably be my first suggestion for a feature to consider adding to git-lfs. I hope for more opportunities to catch-up to git-lfs in turn.

Syndicated 2016-01-19 17:28:50 from see shy jo

STM Region contents

concurrent-output released yesterday got a lot of fun features. It now does full curses-style minimization of the output, to redraw updated lines with optimal efficiency. And supports multiline regions/wrapping too long lines. And allows the user to embed ANSI colors in a region. 3 features that are in some tension and were fun to implement all together.

But I have a more interesting feature to blog about... I've added the ability for the content of a Region to be determined by a (STM transaction).

Here, for example, is a region that's a clock:

timeDisplay :: TVar UTCTime -> STM Text
timeDisplay tv = T.pack . show <$> readTVar tv

clockRegion :: IO ConsoleRegionHandle
clockRegion = do
    tv <- atomically . newTVar =<< getCurrentTime
    r <- openConsoleRegion Linear
    setConsoleRegion r (timeDisplay tv)
    async $ forever $ do
        threadDelay 1000000 -- 1 sec
        atomically . (writeTVar tv) =<< getCurrentTime
    return r

There's something magical about this. Whenever a new value is written into the TVar, concurrent-output automatically knows that this region needs to be updated. How does it know how to do that?

Magic of STM. Basically, concurrent-output composes all the STM transactions of Regions, and asks STM to wait until there's something new to display. STM keeps track of whatever TVars might be looked at, and so can put the display thread to sleep until there's a change to display.

Using STM I've gotten extensability for free, due to the nice ways that STM transactions compose.

A few other obvious things to do with this: Compose 2 regions with padding so they display on the same line, left and right aligned. Trim a region's content to the display width. (Handily exported by concurrent-output in a TVar for this kind of thing.)


I'm tempted to write a console spreadsheet using this. Each visible cell of the spreadsheet would have its own region, that uses a STM transaction to display. Plain data Cells would just display their current value. Cells that contain a function would read the current values of other Cells, and use that to calculate what to display. Which means that a Cell containing a function would automatically update whenever any of the Cells that it depends on were updated!

Do you think that a simple interactive spreadsheet built this way would be more than 100 lines of code?

Syndicated 2015-11-03 20:03:53 from see shy jo

a tiling region manager for the console

Building on top of concurrent-output, and some related work Joachim Breitner did earlier, I now have a kind of equivilant to a tiling window manager, except it's managing regions of the console for different parts of a single program.

Here's a really silly demo, in an animated gif:

demo2.gif

Not bad for 23 lines of code, is that? Seems much less tedious to do things this way than using ncurses. Even with its panels, ncurses requires you to think about layout of various things on the screen, and many low-level details. This, by contrast, is compositional, just add another region and a thread to update it, and away it goes.

So, here's an apt-like download progress display, in 30 lines of code.

aptdemo.gif

Not only does it have regions which are individual lines of the screen, but those can have sub-regions within them as seen here (and so on).

And, log-type messages automatically scroll up above the regions. External programs run by createProcessConcurrent will automatically get their output/errors displayed there, too.

What I'm working on now is support for multiline regions, which automatically grow/shrink to fit what's placed in them. The hard part, which I'm putting the finishing touches on, is to accurately work out how large a region is before displaying it, in order to lay it out. Requires parsing ANSI codes amoung other things.

STM rules

There's so much concurrency, with complicated interrelated data being updated by different threads, that I couldn't have possibly built this without Software Transactional Memory.

Rather than a nightmare of locks behind locks behind locks, the result is so well behaved that I'm confident that anyone who needs more control over the region layout, or wants to do funky things can dive into to the STM interface and update the data structures, and nothing will ever deadlock or be inconsistent, and as soon as an update completes, it'll display on-screen.

An example of how powerful and beuatiful STM is, here's how the main display thread determines when it needs to refresh the display:

data DisplayChange
        = BufferChange [(StdHandle, OutputBuffer)]
        | RegionChange RegionSnapshot
        | TerminalResize (Maybe Width)
        | EndSignal ()

    ...
                change <- atomically $
                        (RegionChange <$> regionWaiter origsnapshot)
                                `orElse`
                        (RegionChange <$> regionListWaiter origsnapshot)
                                `orElse`
                        (BufferChange <$> outputBufferWaiterSTM waitCompleteLines)
                                `orElse`
                        (TerminalResize <$> waitwidthchange)
                                `orElse`
                        (EndSignal <$> waitTSem endsignal)
                case change of
                        RegionChange snapshot -> do
                ...
                        BufferChange buffers -> do
                ...
                        TerminalResize width -> do
                ...

So, it composes all these STM actions that can wait on various kinds of changes, to get one big action, that waits for all of the above, and builds up a nice sum type to represent what's changed.

Another example is that the whole support for sub-regions only involved adding 30 lines of code, all of it using STM, and it worked 100% the first time.


Available in concurrent-output 1.1.0.

Syndicated 2015-10-31 01:44:47 from see shy jo

concurrent output library

concurrent-output is a Haskell library I've developed this week, to make it easier to write console programs that do a lot of different things concurrently, and want to serialize concurrent outputs sanely.

It's increasingly easy to write concurrent programs, but all their status reporting has to feed back through the good old console, which is still obstinately serial.

Haskell illustrates problem this well with this "Linus's first kernel" equivilant interleaving the output of 2 threads:

  > import System.IO
> import Control.Concurrent.Async
> putStrLn (repeat 'A') `concurrently` putStrLn (repeat 'B')
BABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABA
BABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABA
...

That's fun, but also horrible if you wanted to display some messages to the user:

  > putStrLn "washed the car" `concurrently` putStrLn "walked the dog"
walwkaesdh etdh et hdeo gc
ar

To add to the problem, we often want to run separate programs concurrently, which have output of their own to display. And, just to keep things interesting, sometimes a unix program will behave differently when stdout is not connected to a terminal (eg, ls | cat).

To tame simple concurrent programs like these so they generate readable output involves a lot of plumbing. Something like, run the actions concurrently, taking care to capture the output of any commands, and then feed the output that the user should see though some sort of serializing channel to the display. Dealing with that when you just wanted a simple concurrent program risks ending up with a not-so-simple program.

So, I wanted an library with basically 2 functions:

outputConcurrent :: String -> IO ()
    
createProcessConcurrent :: CreateProcess -> IO whatever

The idea is, you make your program use outputConcurrent to display all its output, and each String you pass to that will be displayed serially, without getting mixed up with any other concurrent output.

And, you make your program use createProcessConcurrent everywhere it starts a process that might output to stdout or stderr, and it'll likewise make sure its output is displayed serially.

Oh, and createProcessConcurrent should avoid redirecting stdout and stderr away from the console, when no other concurrent output is happening. So, if programs are mostly run sequentially, they behave as they normally would at the console; any behavior changes should only occur when there is concurrency. (It might also be nice for it to allocate ttys and run programs there to avoid any behavior changes at all, although I have not tried to do that.)

And that should be pretty much the whole API, although it's ok if it needs some function called by main to set it up:

import Control.Concurrent.Async
import Control.Concurrent.Output

main = withConcurrentOutput $
    outputConcurrent "washed the car\n"
        `concurrently`
    createProcessConcurrent (proc "ls" [])
        `concurrently`
    outputConcurrent "walked the dog\n"
  $ ./demo
washed the car
walked the dog
Maildir/  bin/  doc/  html/  lib/  mail/  mnt/  src/  tmp/

I think that's a pretty good API to deal with this concurrent output problem. Anyone know of any other attempts at this I could learn from?

I implemented this over the past 3 days and 320 lines of code. It got rather hairy:

  • It has to do buffering of the output.
  • There can be any quantity of output, but program memory use should be reasonably small. Solved by buffering up to 1 mb of output in RAM, and writing excess buffer to temp files.
  • Falling off the end of the program is complicated; there can be buffered output to flush and it may have to wait for some processes to finish running etc.
  • The locking was tough to get right! I could not have managed to write it correctly without STM.

It seems to work pretty great though. I got Propellor using it, and Propellor can now run actions concurrently!

Syndicated 2015-10-29 02:07:34 from see shy jo

587 older entries...

 

joey certified others as follows:

  • joey certified joey as Journeyer
  • joey certified davidw as Journeyer
  • joey certified bombadil as Journeyer
  • joey certified dhd as Journeyer
  • joey certified ajt as Journeyer
  • joey certified chrisd as Journeyer
  • joey certified scandal as Journeyer
  • joey certified lewing as Journeyer
  • joey certified jwz as Master
  • joey certified graydon as Journeyer
  • joey certified cas as Journeyer
  • joey certified garrett as Journeyer
  • joey certified lupus as Journeyer
  • joey certified octobrx as Journeyer
  • joey certified pudge as Journeyer
  • joey certified marcel as Journeyer
  • joey certified ljlane as Journeyer
  • joey certified uzi as Journeyer
  • joey certified quinlan as Journeyer
  • joey certified bribass as Journeyer
  • joey certified jonas as Journeyer
  • joey certified dsifry as Journeyer
  • joey certified plundis as Journeyer
  • joey certified deirdre as Journeyer
  • joey certified crackmonkey as Journeyer
  • joey certified jim as Journeyer
  • joey certified vincent as Journeyer
  • joey certified apenwarr as Journeyer
  • joey certified schoen as Journeyer
  • joey certified CentralScrutinizer as Apprentice
  • joey certified wichert as Master
  • joey certified doogie as Journeyer
  • joey certified espy as Journeyer
  • joey certified omnic as Journeyer
  • joey certified hands as Journeyer
  • joey certified stig as Journeyer
  • joey certified nick as Journeyer
  • joey certified tausq as Journeyer
  • joey certified broonie as Journeyer
  • joey certified dunham as Journeyer
  • joey certified austin as Journeyer
  • joey certified lordsutch as Journeyer
  • joey certified Gimptek as Apprentice
  • joey certified jimd as Journeyer
  • joey certified chip as Master
  • joey certified jgg as Master
  • joey certified branden as Journeyer
  • joey certified z as Journeyer
  • joey certified srivasta as Journeyer
  • joey certified danpat as Journeyer
  • joey certified lilo as Journeyer
  • joey certified seeS as Journeyer
  • joey certified netgod as Journeyer
  • joey certified dres as Journeyer
  • joey certified cech as Journeyer
  • joey certified knghtbrd as Journeyer
  • joey certified calc as Journeyer
  • joey certified ruud as Journeyer
  • joey certified edlang as Journeyer
  • joey certified gorgo as Journeyer
  • joey certified jwalther as Journeyer
  • joey certified bma as Journeyer
  • joey certified claw as Apprentice
  • joey certified hp as Journeyer
  • joey certified esr as Master
  • joey certified tobi as Journeyer
  • joey certified ajk as Journeyer
  • joey certified Joy as Journeyer
  • joey certified ejb as Journeyer
  • joey certified corbet as Journeyer
  • joey certified rcw as Journeyer
  • joey certified woot as Journeyer
  • joey certified bcollins as Journeyer
  • joey certified neuro as Journeyer
  • joey certified biffhero as Journeyer
  • joey certified Trakker as Journeyer
  • joey certified bdale as Journeyer
  • joey certified foka as Journeyer
  • joey certified davem as Master
  • joey certified logic as Journeyer
  • joey certified mstone as Journeyer
  • joey certified drow as Journeyer
  • joey certified clameter as Journeyer
  • joey certified mdorman as Journeyer
  • joey certified bwoodard as Journeyer
  • joey certified JHM as Journeyer
  • joey certified lalo as Journeyer
  • joey certified edb as Journeyer
  • joey certified shaleh as Journeyer
  • joey certified x as Apprentice
  • joey certified stephenc as Journeyer
  • joey certified bodo as Journeyer
  • joey certified jpick as Journeyer
  • joey certified ncm as Journeyer
  • joey certified gord as Journeyer
  • joey certified mpav as Journeyer
  • joey certified lazarus as Apprentice
  • joey certified starshine as Journeyer
  • joey certified che as Journeyer
  • joey certified brother as Journeyer
  • joey certified joeysmith as Journeyer
  • joey certified bod as Journeyer
  • joey certified decklin as Journeyer
  • joey certified gibreel as Journeyer
  • joey certified torsten as Journeyer
  • joey certified alfie as Apprentice
  • joey certified aclark as Journeyer
  • joey certified kju as Journeyer
  • joey certified psg as Journeyer
  • joey certified zed as Journeyer
  • joey certified evo as Journeyer
  • joey certified mbaker as Journeyer
  • joey certified cmr as Journeyer
  • joey certified Tv as Journeyer
  • joey certified xtifr as Journeyer
  • joey certified sstrickl as Journeyer
  • joey certified etbe as Journeyer

Others have certified joey as follows:

  • joey certified joey as Journeyer
  • dhd certified joey as Journeyer
  • ajt certified joey as Master
  • davidw certified joey as Journeyer
  • alan certified joey as Journeyer
  • uzi certified joey as Journeyer
  • caolan certified joey as Journeyer
  • tron certified joey as Master
  • bombadil certified joey as Journeyer
  • cas certified joey as Journeyer
  • garrett certified joey as Master
  • lupus certified joey as Journeyer
  • graydon certified joey as Journeyer
  • marcel certified joey as Journeyer
  • mblevin certified joey as Journeyer
  • bribass certified joey as Master
  • plundis certified joey as Journeyer
  • matias certified joey as Journeyer
  • ajv certified joey as Journeyer
  • crackmonkey certified joey as Master
  • jim certified joey as Master
  • CentralScrutinizer certified joey as Master
  • schoen certified joey as Master
  • pedro certified joey as Master
  • omnic certified joey as Master
  • hands certified joey as Master
  • tausq certified joey as Journeyer
  • suzi certified joey as Master
  • broonie certified joey as Master
  • nick certified joey as Journeyer
  • lordsutch certified joey as Master
  • jimd certified joey as Master
  • chip certified joey as Master
  • jgg certified joey as Master
  • branden certified joey as Master
  • srivasta certified joey as Master
  • danpat certified joey as Master
  • darkewolf certified joey as Master
  • z certified joey as Journeyer
  • cech certified joey as Master
  • dres certified joey as Master
  • gorgo certified joey as Master
  • ruud certified joey as Master
  • kaig certified joey as Master
  • wichert certified joey as Master
  • ajk certified joey as Master
  • ljlane certified joey as Master
  • Joy certified joey as Journeyer
  • andrei certified joey as Master
  • rcw certified joey as Master
  • Trakker certified joey as Master
  • neuro certified joey as Master
  • starshine certified joey as Master
  • seeS certified joey as Master
  • foka certified joey as Master
  • pretzelgod certified joey as Master
  • mstone certified joey as Master
  • bcollins certified joey as Master
  • doviende certified joey as Master
  • dmarti certified joey as Master
  • splork certified joey as Master
  • bdale certified joey as Master
  • drow certified joey as Master
  • edward certified joey as Master
  • ljb certified joey as Journeyer
  • claw certified joey as Master
  • edb certified joey as Master
  • shaleh certified joey as Master
  • jpick certified joey as Master
  • zacs certified joey as Journeyer
  • jae certified joey as Master
  • benson certified joey as Journeyer
  • wardv certified joey as Master
  • jeroen certified joey as Master
  • lazarus certified joey as Journeyer
  • mpav certified joey as Master
  • walken certified joey as Master
  • ncm certified joey as Master
  • Barbwired certified joey as Master
  • kraai certified joey as Master
  • che certified joey as Master
  • lstep certified joey as Master
  • brother certified joey as Master
  • nas certified joey as Journeyer
  • acme certified joey as Master
  • moshez certified joey as Master
  • tca certified joey as Journeyer
  • cord certified joey as Master
  • sethcohn certified joey as Master
  • bod certified joey as Journeyer
  • tripix certified joey as Journeyer
  • jLoki certified joey as Master
  • sh certified joey as Master
  • lerdsuwa certified joey as Master
  • torsten certified joey as Master
  • alfie certified joey as Master
  • mhatta certified joey as Master
  • aclark certified joey as Master
  • kju certified joey as Master
  • psg certified joey as Master
  • zed certified joey as Master
  • karlheg certified joey as Master
  • evo certified joey as Master
  • ole certified joey as Master
  • jfs certified joey as Master
  • bma certified joey as Master
  • jtc certified joey as Master
  • gibreel certified joey as Master
  • Jordi certified joey as Master
  • jhasler certified joey as Master
  • cpbs certified joey as Journeyer
  • ths certified joey as Master
  • decklin certified joey as Master
  • Tv certified joey as Master
  • xtifr certified joey as Master
  • joeysmith certified joey as Master
  • mishan certified joey as Master
  • keverets certified joey as Master
  • pa certified joey as Master
  • Slimer certified joey as Master
  • weasel certified joey as Master
  • technik certified joey as Master
  • baretta certified joey as Master
  • robster certified joey as Master
  • juhtolv certified joey as Master
  • rcyeske certified joey as Master
  • kmself certified joey as Master
  • andersee certified joey as Master
  • asuffield certified joey as Master
  • charon certified joey as Master
  • claviola certified joey as Master
  • chrisd certified joey as Master
  • mdz certified joey as Master
  • buckley certified joey as Master
  • moray certified joey as Master
  • jtjm certified joey as Master
  • mwk certified joey as Master
  • proski certified joey as Master
  • cmiller certified joey as Master
  • pau certified joey as Master
  • rkrishnan certified joey as Master
  • dieman certified joey as Master
  • eckes certified joey as Master
  • fxn certified joey as Master
  • etbe certified joey as Master
  • Sam certified joey as Master
  • fallenlord certified joey as Master
  • hanna certified joey as Master
  • maxx certified joey as Master
  • dopey certified joey as Master
  • tfheen certified joey as Master
  • ttroxell certified joey as Master
  • Netsnipe certified joey as Master
  • quarl certified joey as Journeyer
  • amck certified joey as Master
  • riverwind certified joey as Master
  • pere certified joey as Journeyer
  • NoWhereMan certified joey as Master
  • jochen certified joey as Master
  • faw certified joey as Master
  • mako certified joey as Master
  • Pizza certified joey as Master
  • sysdebug certified joey as Master
  • vern certified joey as Master
  • ctrlsoft certified joey as Master
  • lkcl certified joey as Master
  • hasienda certified joey as Master
  • gesslein certified joey as Master
  • ean certified joey as Master
  • dangermaus certified joey as Master

[ Certification disabled because you're not logged in. ]

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

X
Share this page