joey is currently certified at Master level.

Name: Joey Hess
Member since: 2000-03-06 23:42:41
Last Login: 2011-12-31 20:04:52

FOAF RDF Share This

Homepage: http://kitenet.net/~joey

Projects

Recent blog entries by joey

Syndication: RSS 2.0

propellor is d-i 2.0

I think I've been writing the second system to replace d-i with in my spare time for a couple months, and never noticed.

I'm as suprised as you are, but consider this design:

  • Installation system consists of debian live + haskell + propellor + web browser.

  • Entire installation UI consists of a web-based (and entirely pictographic and prompt based, so does not need to be translated) selection of the installation target.

  • Installation target can be local disk, remote system via ssh (wiping out crufty hacked-up pre-installed debian), local VM, live ISO, etc.

  • Really, no other questions. Not even user name/password! The installed system will only allow login via the same method that was used to install it. So a locally installed system will accept console/X login with no password and then a forced password change. Or a system installed via ssh will only allow login using the same ssh key that was used to install it.

  • The entire installation process consists of a disk format, followed by debootstrap, followed by running propellor in the target system. This also means that the installed system includes a propellor config file which now describes the properties of the system as installed (so can be edited to tweak the installation, or reused as starting point for next installation).

  • Users who want to configure installation in any way write down properties of system using a simple propellor config file. I suppose some people still use more than one partiton or gnome or some such customization, so they'd use:

main :: IO
main = Installer.main
    & Installer.partition First "/boot" Ext3 (MiB 256)
    & Installer.partition Next "/" Ext4 (GiB 5)
    & Installer.partition Next "/home" Ext4 FreeSpace
    & Installer.grubBoots "hd0"
    & os (System (Debian Stable) "amd64")
    & Apt.stdSourcesList
    & Apt.installed ["task-gnome-desktop"]
  • The installation system is itself built using propellor. A free feature given the above design, so basically all it will take to build an installation iso is this code:
main :: IO
main = Installer.main
    & Installer.target CdImage "installer.iso"
    & os (System (Debian Stable) "amd64")
    & Apt.stdSourcesList
    & Apt.installed ["task-xfce-desktop", "ghc", "propellor"]
    & User.autoLogin "root"
    & User.loginStarts "propellor --installer"
  • Propellor has a nice display of what it's doing so there is no freaking progress bar.

Well, now I know where propellor might end up if I felt like spending a month and adding a few thousand lines of code to it.

Syndicated 2014-09-08 09:32:25 from see shy jo

using a debian package as the remote for a local config repo

Today I did something interesting with the Debian packaging for propellor, which seems like it could be a useful technique for other Debian packages as well.

Propellor is configured by a directory, which is maintained as a local git repository. In propellor's case, it's ~/.propellor/. This contains a lot of haskell files, in fact the entire source code of propellor! That's really unusual, but I think this can be generalized to any package whose configuration is maintained in its own git repository on the user's system. For now on, I'll refer to this as the config repo.

The config repo is set up the first time a user runs propellor. But, until now, I didn't provide an easy way to update the config repo when the propellor package was updated. Nothing would break, but the old version would be used until the user updated it themselves somehow (probably by pulling from a git repository over the network, bypassing apt's signature validation).

So, what I wanted was a way to update the config repo, merging in any changes from the new version of the Debian package, while preserving the user's local modifications. Ideally, the user could just run git merge upstream/master, where the upstream repo was included in the Debian package.

But, that can't work! The Debian package can't reasonably include the full git repository of propellor with all its history. So, any git repository included in the Debian binary package would need to be a synthetic one, that only contains probably one commit that is not connected to anything else. Which means that if the config repo was cloned from that repo in version 1.0, then when version 1.1 came around, git would see no common parent when merging 1.1 into the config repo, and the merge would fail horribly.

To solve this, let's assume that the config repo's master branch has a parent commit that can be identified, somehow, as coming from a past version of the Debian package. It doesn't matter which version, although the last one merged with will be best. (The easy way to do this is to set refs/heads/upstream/master to point to it when creating the config repo.)

Once we have that parent commit, we have three things:

  1. The current content of the config repo.
  2. The content from some old version of the Debian package.
  3. The new content of the Debian package.

Now git can be used to merge #3 onto #2, with -Xtheirs, so the result is a git commit with parents of #3 and #2, and content of #3. (This can be done using a temporary clone of the config repo to avoid touching its contents.)

Such a git commit can be merged into the config repo, without any conflicts other than those the user might have caused with their own edits.

So, propellor will tell the user when updates are available, and they can simply run git merge upstream/master to get them. The resulting history looks like this:

* Merge remote-tracking branch 'upstream/master'
|\  
| * merging upstream version
| |\  
| | * upstream version
* | user change
|/  
* upstream version

So, generalizing this, if a package has a lot of config files, and creates a git repository containing them when the user uses it (or automatically when it's installed), this method can be used to provide an easily mergable branch that tracks the files as distributed with the package.

It would perhaps not be hard to get from here to a full git-backed version of ucf. Note that the Debian binary package doesn't have to ship a git repisitory, it can just as easily ship the current version of the config files somewhere in /usr, and check them into a new empty repository as part of the generation of the upstream/master branch.

Syndicated 2014-08-19 22:55:20 from see shy jo

abram's 2014

pics from trip to Abram's Falls

The trail to Abram's Falls seems more trecherous as we get older, but the sights and magic of the place are unchanged in our first visit in years.

Syndicated 2014-08-05 19:37:37 from see shy jo

laptop death

So I was at Ocracoke island, camping with family, and I brought my laptop along as I've done probably half a dozen times before. An enormous thuderstorm came up. It rained for 8 hours and thundered for 3 of those. Some lightning cracks quite close by as we crouched in the food tent, our feet up off the increasingly wet ground "just in case". The campground flooded. Luckily we were camped in the dunes and tents mostly avoided being flooded with 2-3 inches of water. (That was just the warmup; a hurricane hit a week after we left.)

My laptop was in my tent when this started, and I got soaked to the skin just running over there and throwing it up on the thermarest to keep it out of any flooding and away from any drips. It seemed ok, so best not to try to move it to the car in that downpour.

Next time I checked, it turned out the top vent of the tent was slightly open and dripping. The laptop bag was damp. But inside it seemed ok. Rain had slackened to just heavy, so I ran it down to the car. Laptop appeared barely damp, but it was hard to tell as I had quite forgotten what "dry" was. Turned it on for 10 seconds to check the time. It was 7:30 and we still had to cook dinner in this mess. Transferred it to a dry bag.

(By the way, in some situations, discovering you have a single dry towel you didn't know you had is the best gift in the world!)

Next morning, the laptop was dead. When powered on, the fan came on full, the screen stayed black, and after a few seconds it turned itself back off.

I need this for work, so it was a crash priority to get it fixed or a replacement. Before I even got home, I had logged onto Lenovo's website to check warantee status and found 2 things:

  1. They needed some number from a sticker on the bottom of my laptop. Which was no longer there.
  2. The process required some stange login on an entirely different IBM website.

At this point, I had a premonition of how the beuracracy would go. Reading Sesse's Blehnovo, I see I was right. I didn't even try. I ordered a replacement with priority shipping.

When I got home, I pulled the laptop apart to try to debug it. I still don't know what's wrong with it. The SSD may be damaged; it seems to cause anything I put it into to fail to work.

New laptop arrived in 2 days. Since this model is now a year old, it was a few hundred dollars cheaper this time around. And now I have an extra power supply, and a replacment keyboard, and a replacement fan etc. And I've escaped the dead USB port and broken rocker switch of the old laptop too.

The only weird thing is that, while my old laptop had no problem with my Toshiba passport USB drive, this new one refuses to recognize it unless I plug it into a USB 1.0 hub. Oh well..

Syndicated 2014-07-08 00:00:41 from see shy jo

what does docker.io run -it debian sh run?

When you type docker.io run -it debian sh, it goes off and gets "debian" and runs it. But what is in this "debian" image? How was it built?

The docker hub does not really say. All it tells us is this is a "(Semi) Official Debian base image" and that its sources.list uses http.debian.net for geolocation.

There's a link to https://github.com/dotcloud/stackbrew/blob/master/library/debian which in turn uses a very strange git repository, owned by Debian maintainer Tianon Gravi, that contains compressed tarballs of Debian: http://github.com/tianon/docker-brew-debian "Git is not a fan of what we're doing here."

The "source", such as it is, that is used to build this image consists of:

FROM scratch
ADD rootfs.tar.xz /
CMD ["/bin/bash"]

and

mkimage.sh -t tianon/debian:wheezy -d . debootstrap --variant=minbase --components=main --include=inetutils-ping,iproute wheezy http://http.debian.net/debian

I don't know where mkimage.sh is. And anyway, I have no reason to trust that this image is built the way it claims to be built. So, the question remains: What is in this image?

To find out, I did a debootstap --variant=minbase stable and diffed the entire docker debian image against it. The diff was 6738 lines, from which I found the following interesting differences.

added packages

The image has iputils-ping and netbase and iproute added. These are not in a minbase debootstrap, but are in a regular debootstrap. It's rather weird that the docker image is based on a minbase debootstrap, since this means they have to add back important stuff like this on an ad-hoc basis.

If the expectation is that an experienced Unix person who found it missing would say "What on earth is going on, where is 'foo'?", it must be an 'important' package. -- Debian Policy

apt hooks

DPkg::Post-Invoke { "rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true"; };
APT::Update::Post-Invoke { "rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true"; };

Dir::Cache::pkgcache "";
Dir::Cache::srcpkgcache "";

Acquire::Languages "none";

These are some strange modifications to apt's config. The intent is clearly to avoid wasting disk space, even at the expense of making apt slower (by disabling caches) and losing translations.

I am curious if apt might ever invoke the DPkg::Post-Invoke twice in an upgrade in which it runs dpkg twice. I'm also curious whether deleting /var/cache/apt/archives/lock could cause a problem.

unsafe-io

dpkg is configured to use unsafe-io.

motd

Linux viper 3.12.20-gentoo #1 SMP Sun May 18 12:36:24 MDT 2014 x86_64

Yes, that's "gentoo". Presumably this tells us something about the build host.

policy-rc.d

/usr/sbin/policy-rc.d contains "exit 101", which prevents daemons from being automatically started after they are installed. This may or may not be desirable, depending on what you're doing with docker.

It notably also prevents restarting running daemons in this container if they're upgraded for eg, a security fix. It would almost certianly be better if this script allowed restarting running daemons.

diversions

/sbin/initctl is diverted and replaced with /bin/true. This is a workaround for a bug in sysvinit; when upgraded inside a docker container it hangs while trying to run initctl.

missing devices

Several devices are missing, including /dev/full, /dev/null, /dev/zero, /dev/[u]random, /dev/shm, and /dev/loopN. Docker probably fixes some of these up when running the image, but it certianly works ok with an image with all these missing devices present, so I don't know the rationalle for omitting them.

(See also this bug)

some gpg thing is different

Binary files pure-debootstrap/etc/apt/trustdb.gpg and from-docker/etc/apt/trustdb.gpg differ

Oh well, that can't be important.. Or can it? I did not check.

conclusions

I would hardly consider this to be an "(Semi) Official Debian image". Some of the changes are quite dubious. The build environment is not Debian. There is no guarantee you'll get the same image I examined. Diffing thousands of lines of filesystem changes is not particularly fun or reliable way to spot accidental or malicious changes.

I'd recommend only trusting docker images you build yourself. I have some docker images published somewhere that are built with 100% straight debootstrap with no modifications (and even an armel image that can be used on an x86 system thanks to qemu). But I'm not going to link to them, because again, you should only trust docker images you built yourself. To help increase your mistrust of me, I present this IRC snippet:

I'll bet I could publish an image that just did a killall5 as root on startup and get plenty of people to nuke their container hosts

Here are some ideas for things Debian could do to improve this:

  • Make a package that can build docker images of Debian, in a fully reproducible fashion. Ie, same versions of debs in, same byte stream out.
  • If it makes sense for the docker image to not contain all the packages in a standard debootstrap, or to contain other packages, write down the rationalle for this, and make a --variant=docker.
  • Make a package that provides appropriate tweaks for Debian in a container. This might include a policy-rc.d that allows restarting daemons on upgrade if they're already running in the container, and otherwise prevents running daemons.
  • Make a low-disk-space package that eg, prevents apt from caching debs.
  • Provide some way to verify, through gpg signatures, that docker has pulled an actual trusted image and not some https-MITMed thing.

PS, if this wasn't enough fun, just consider the tweaks made to the "Debian" images on all the VPS hosts out there.

Syndicated 2014-06-19 15:58:26 from see shy jo

556 older entries...

 

joey certified others as follows:

  • joey certified joey as Journeyer
  • joey certified davidw as Journeyer
  • joey certified bombadil as Journeyer
  • joey certified dhd as Journeyer
  • joey certified ajt as Journeyer
  • joey certified chrisd as Journeyer
  • joey certified scandal as Journeyer
  • joey certified lewing as Journeyer
  • joey certified jwz as Master
  • joey certified graydon as Journeyer
  • joey certified cas as Journeyer
  • joey certified garrett as Journeyer
  • joey certified lupus as Journeyer
  • joey certified octobrx as Journeyer
  • joey certified pudge as Journeyer
  • joey certified marcel as Journeyer
  • joey certified ljlane as Journeyer
  • joey certified uzi as Journeyer
  • joey certified quinlan as Journeyer
  • joey certified bribass as Journeyer
  • joey certified jonas as Journeyer
  • joey certified dsifry as Journeyer
  • joey certified plundis as Journeyer
  • joey certified deirdre as Journeyer
  • joey certified crackmonkey as Journeyer
  • joey certified jim as Journeyer
  • joey certified vincent as Journeyer
  • joey certified apenwarr as Journeyer
  • joey certified schoen as Journeyer
  • joey certified CentralScrutinizer as Apprentice
  • joey certified wichert as Master
  • joey certified doogie as Journeyer
  • joey certified espy as Journeyer
  • joey certified omnic as Journeyer
  • joey certified hands as Journeyer
  • joey certified stig as Journeyer
  • joey certified nick as Journeyer
  • joey certified tausq as Journeyer
  • joey certified broonie as Journeyer
  • joey certified dunham as Journeyer
  • joey certified austin as Journeyer
  • joey certified lordsutch as Journeyer
  • joey certified Gimptek as Apprentice
  • joey certified jimd as Journeyer
  • joey certified chip as Master
  • joey certified jgg as Master
  • joey certified branden as Journeyer
  • joey certified z as Journeyer
  • joey certified srivasta as Journeyer
  • joey certified danpat as Journeyer
  • joey certified lilo as Journeyer
  • joey certified seeS as Journeyer
  • joey certified netgod as Journeyer
  • joey certified dres as Journeyer
  • joey certified cech as Journeyer
  • joey certified knghtbrd as Journeyer
  • joey certified calc as Journeyer
  • joey certified ruud as Journeyer
  • joey certified edlang as Journeyer
  • joey certified gorgo as Journeyer
  • joey certified jwalther as Journeyer
  • joey certified bma as Journeyer
  • joey certified claw as Apprentice
  • joey certified hp as Journeyer
  • joey certified esr as Master
  • joey certified tobi as Journeyer
  • joey certified ajk as Journeyer
  • joey certified Joy as Journeyer
  • joey certified ejb as Journeyer
  • joey certified corbet as Journeyer
  • joey certified rcw as Journeyer
  • joey certified woot as Journeyer
  • joey certified bcollins as Journeyer
  • joey certified neuro as Journeyer
  • joey certified biffhero as Journeyer
  • joey certified Trakker as Journeyer
  • joey certified bdale as Journeyer
  • joey certified foka as Journeyer
  • joey certified davem as Master
  • joey certified logic as Journeyer
  • joey certified mstone as Journeyer
  • joey certified drow as Journeyer
  • joey certified clameter as Journeyer
  • joey certified mdorman as Journeyer
  • joey certified bwoodard as Journeyer
  • joey certified JHM as Journeyer
  • joey certified lalo as Journeyer
  • joey certified edb as Journeyer
  • joey certified shaleh as Journeyer
  • joey certified x as Apprentice
  • joey certified stephenc as Journeyer
  • joey certified bodo as Journeyer
  • joey certified jpick as Journeyer
  • joey certified ncm as Journeyer
  • joey certified gord as Journeyer
  • joey certified mpav as Journeyer
  • joey certified lazarus as Apprentice
  • joey certified starshine as Journeyer
  • joey certified che as Journeyer
  • joey certified brother as Journeyer
  • joey certified joeysmith as Journeyer
  • joey certified bod as Journeyer
  • joey certified decklin as Journeyer
  • joey certified gibreel as Journeyer
  • joey certified torsten as Journeyer
  • joey certified alfie as Apprentice
  • joey certified aclark as Journeyer
  • joey certified kju as Journeyer
  • joey certified psg as Journeyer
  • joey certified zed as Journeyer
  • joey certified evo as Journeyer
  • joey certified mbaker as Journeyer
  • joey certified cmr as Journeyer
  • joey certified Tv as Journeyer
  • joey certified xtifr as Journeyer
  • joey certified sstrickl as Journeyer
  • joey certified etbe as Journeyer

Others have certified joey as follows:

  • joey certified joey as Journeyer
  • dhd certified joey as Journeyer
  • ajt certified joey as Master
  • davidw certified joey as Journeyer
  • alan certified joey as Journeyer
  • uzi certified joey as Journeyer
  • caolan certified joey as Journeyer
  • tron certified joey as Master
  • bombadil certified joey as Journeyer
  • cas certified joey as Journeyer
  • garrett certified joey as Master
  • lupus certified joey as Journeyer
  • graydon certified joey as Journeyer
  • marcel certified joey as Journeyer
  • mblevin certified joey as Journeyer
  • bribass certified joey as Master
  • plundis certified joey as Journeyer
  • matias certified joey as Journeyer
  • ajv certified joey as Journeyer
  • crackmonkey certified joey as Master
  • jim certified joey as Master
  • CentralScrutinizer certified joey as Master
  • schoen certified joey as Master
  • pedro certified joey as Master
  • omnic certified joey as Master
  • hands certified joey as Master
  • tausq certified joey as Journeyer
  • suzi certified joey as Master
  • broonie certified joey as Master
  • nick certified joey as Journeyer
  • lordsutch certified joey as Master
  • jimd certified joey as Master
  • chip certified joey as Master
  • jgg certified joey as Master
  • branden certified joey as Master
  • srivasta certified joey as Master
  • danpat certified joey as Master
  • darkewolf certified joey as Master
  • z certified joey as Journeyer
  • cech certified joey as Master
  • dres certified joey as Master
  • gorgo certified joey as Master
  • ruud certified joey as Master
  • kaig certified joey as Master
  • wichert certified joey as Master
  • ajk certified joey as Master
  • ljlane certified joey as Master
  • Joy certified joey as Journeyer
  • andrei certified joey as Master
  • rcw certified joey as Master
  • Trakker certified joey as Master
  • neuro certified joey as Master
  • starshine certified joey as Master
  • seeS certified joey as Master
  • foka certified joey as Master
  • pretzelgod certified joey as Master
  • mstone certified joey as Master
  • bcollins certified joey as Master
  • doviende certified joey as Master
  • dmarti certified joey as Master
  • splork certified joey as Master
  • bdale certified joey as Master
  • drow certified joey as Master
  • edward certified joey as Master
  • ljb certified joey as Journeyer
  • claw certified joey as Master
  • edb certified joey as Master
  • shaleh certified joey as Master
  • jpick certified joey as Master
  • zacs certified joey as Journeyer
  • jae certified joey as Master
  • benson certified joey as Journeyer
  • wardv certified joey as Master
  • jeroen certified joey as Master
  • lazarus certified joey as Journeyer
  • mpav certified joey as Master
  • walken certified joey as Master
  • ncm certified joey as Master
  • Barbwired certified joey as Master
  • kraai certified joey as Master
  • che certified joey as Master
  • lstep certified joey as Master
  • brother certified joey as Master
  • nas certified joey as Journeyer
  • acme certified joey as Master
  • moshez certified joey as Master
  • tca certified joey as Journeyer
  • cord certified joey as Master
  • sethcohn certified joey as Master
  • bod certified joey as Journeyer
  • tripix certified joey as Journeyer
  • jLoki certified joey as Master
  • sh certified joey as Master
  • lerdsuwa certified joey as Master
  • torsten certified joey as Master
  • alfie certified joey as Master
  • mhatta certified joey as Master
  • aclark certified joey as Master
  • kju certified joey as Master
  • psg certified joey as Master
  • zed certified joey as Master
  • karlheg certified joey as Master
  • evo certified joey as Master
  • ole certified joey as Master
  • jfs certified joey as Master
  • bma certified joey as Master
  • jtc certified joey as Master
  • gibreel certified joey as Master
  • Jordi certified joey as Master
  • jhasler certified joey as Master
  • cpbs certified joey as Journeyer
  • ths certified joey as Master
  • decklin certified joey as Master
  • Tv certified joey as Master
  • xtifr certified joey as Master
  • joeysmith certified joey as Master
  • mishan certified joey as Master
  • keverets certified joey as Master
  • pa certified joey as Master
  • Slimer certified joey as Master
  • weasel certified joey as Master
  • technik certified joey as Master
  • baretta certified joey as Master
  • robster certified joey as Master
  • juhtolv certified joey as Master
  • rcyeske certified joey as Master
  • kmself certified joey as Master
  • andersee certified joey as Master
  • asuffield certified joey as Master
  • charon certified joey as Master
  • claviola certified joey as Master
  • chrisd certified joey as Master
  • mdz certified joey as Master
  • buckley certified joey as Master
  • moray certified joey as Master
  • jtjm certified joey as Master
  • mwk certified joey as Master
  • proski certified joey as Master
  • cmiller certified joey as Master
  • pau certified joey as Master
  • rkrishnan certified joey as Master
  • dieman certified joey as Master
  • eckes certified joey as Master
  • fxn certified joey as Master
  • etbe certified joey as Master
  • Sam certified joey as Master
  • fallenlord certified joey as Master
  • hanna certified joey as Master
  • maxx certified joey as Master
  • dopey certified joey as Master
  • tfheen certified joey as Master
  • ttroxell certified joey as Master
  • Netsnipe certified joey as Master
  • quarl certified joey as Journeyer
  • amck certified joey as Master
  • riverwind certified joey as Master
  • pere certified joey as Journeyer
  • NoWhereMan certified joey as Master
  • jochen certified joey as Master
  • faw certified joey as Master
  • mako certified joey as Master
  • Pizza certified joey as Master
  • sysdebug certified joey as Master
  • vern certified joey as Master
  • ctrlsoft certified joey as Master
  • lkcl certified joey as Master
  • hasienda certified joey as Master
  • gesslein certified joey as Master
  • ean certified joey as Master
  • dangermaus certified joey as Master

[ Certification disabled because you're not logged in. ]

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

X
Share this page