Older blog entries for mdz (starting at number 25)

DebConf 10: Day 1

This week, I am attending DebConf 10 at Columbia University in New York.

The first day of DebConf is known as Debian Day. While most of DebConf is for the benefit of people involved in Debian itself, Debian Day is aimed at a wider audience, and invites the public to learn about, and interact with, the Debian project.

These are the talks I attended.

Debian Day Opening Plenary (Gabriella Coleman, Hans-Christoph Steiner)

Hans-Christoph discussed Debian and free software from a big picture perspective: why software freedom matters, challenging the producer/consumer dichotomy, how the Debian ecosystem hangs together, and so on.

Steps to adopting F/OSS in government (Andy Oram)

Andy discussed FLOSS adoption in governments, drawing on examples from Peru, the city of Munich, the state of Massachusetts. He covered the reasons why this is valuable, the relationship between government transparency and software freedom, and practical advice for successful adoption and deployment.”

Pedagogical Freedom (panel, Jonah Bossewitch et al)

The panelists discussed the use of technology in education, especially free software, some of the parallels between free software and education, and what these communities could learn from each other. This is a promising topic, though the perspectives seemed to be mostly from the education realm. There is much to be learned on both sides.

Google Summer of Code 2010 at Debian (Obey Arthur Liu)

This talk covered the student projects for this year’s Summer of Code. Most of the students were in attendance, and presented their own work. They ranged from more specialized projects like the Hurd installer, to core infrastructure improvements like multi-arch in APT.

Beyond Sharing: Open Source Design (Mushon Zer-Aviv)

Mushon gave an excellent talk on open design. This is a subject I’ve thought quite a bit about, and Asheesh validated many of my conclusions from a different angle. I’ve added a new post to my todo list to go into more detail on this subject.

Some points from his talk which resonated with me:

• When collaborating on code, everyone must reason with one collaborator: the computer. This forces a level playing field and a common encoding.
• Collaborating on other types of creative work is more difficult in part because of the differences encoding/decoding information between different individuals
• Making this easier for design work requires improving motivational factors and language as well as tools and processes
• Many design decisions are actually rational, and are compatible with a group consensus project. Too often, I hear that design can’t be done collaboratively, citing “too many cooks in the kitchen” analogies, but I have never believed it.
• Mushon’s own project, shiftspace.org, seems to be a browser-plugin-based system for collaboratively remixing web applications. I haven’t looked at it yet.
• Leadership and openness are not mutually exclusive. This is another pet peeve of mine, and there are so many examples of open leadership in the free software community that I don’t see how anyone can think otherwise.
• Mushon’s presentation is available in revision control so that it can be freely used and improved

How Government can Foster Freedom in Technology (Hon. Gale Brewer)

Councillor Brewer paid a visit to DebConf to tell us about the work she is doing on the city council to promote better government through technology.

Brewer seems to be a strong advocate of open data, saying essentially that all government data should be public. She summarized a bill to mandate that New York City government data be public, shared in raw form using open standards, and kept up to date. It sounded like a very strong move which would encourage third party innovation around the data.

She also discussed the need for greater access to computers and Internet connectivity, particularly in educational settings, and a desire to to have all public hearings and meetings shared online.

Why is GNU/Linux Like a Player Piano? (Jon Anderson Hall, Esq.)

Jon is a very engaging speaker. He drew parallels between the development of player pianos, reproducing pianos, reed organs, pipe organs…and free software. He even tied in Hedi Lamarr’s work which led to spread spectrum wireless technology. To be quite honest, I did not find that these analogies taught me much about either free software or player pianos, but nonetheless, I couldn’t help but take an interest in what he was saying and how he presented it.

DebConf Opening Plenary (Gabriella Coleman)

Biella and company explained all the ins and outs of the event: where to go, what to do (and not do), and most importantly, whom to thank for all of it. Now in its 11th year, DebConf is an impressively well-run conference.

I’m looking forward to the rest of the week!

Syndicated 2010-08-02 00:50:44 from We'll see | Matt Zimmerman

Embracing the Web

The web offers a compelling platform for developing modern applications. How can free software benefit more from web technology, and at the same time promote more software freedom on the web? What would the world be like if FLOSS web applications were as plentiful and successful as traditional FLOSS applications are today?

Web architecture

The web, as a collection of interlinked hypertext documents available on the Internet, has been well established for over a decade. However, the web as an application architecture is only just hitting its stride. With modern tools and frameworks, it’s relatively straightforward to build rich applications with browser-oriented frontends and HTTP-accessible backends.

This architecture has its limitations, of course: browser compatibility nightmares, limited offline capabilities, network latency, performance challenges, server-side scalability, complicated multimedia story, and so on. Most of these are slowly but surely being addressed or ameliorated as web technology improves.

However, for a large class of applications, these limitations are easily outweighed by the advantages: cross-platform support, instantaneous upgrades, global availability, etc. The web enables developers to reach the largest audience of users with the most compelling functionality, and simplifies users’ lives by giving them immediate access to their digital lives from anywhere.

Some web advocates would go so far as to say that if an application can be built for the web, it should be built for the web because it will be more successful. It’s no surprise that new web applications are being developed at a staggering rate, and I expect this trend to continue.

So what?

This trend represents a significant threat, and a corresponding opportunity, to free software. Relatively few web applications are free software, and relatively few free software applications are built for the web. Therefore, the momentum which is leading developers and users to the web is also leading them (further) away from free software.

Traditionally, pragmatists have adopted free software applications because they offered immediate gratification: it’s much faster and easier to install a free software application than to buy a proprietary one. The SaaS model of web applications offers the same (and better) immediacy, so free software has lost some of its appeal among pragmatists, who instead turn to proprietary web applications. Why install and run a heavyweight client application when you can just click a link?

Many web applications—perhaps even a majority—are built using free software, but are not themselves free. A new generation of developers share an appreciation for free software tools and frameworks, but see little value in sharing their own software. To these developers, free software is something you use, not something you make.

Free software cannot afford to ignore the web. Instead, we should embrace the web more completely, more powerfully, and more effectively than proprietary systems do.

What would that look like?

In my view, a FLOSS client platform which fully embraced the web would:

• treat web applications as first-class citizens. The web would not be just another application, represented by a browser, but more like a native application runtime. Web applications could feel much more “native” while still preserving the advantages of a web-style user experience. There would be no web browser: that’s a tool for legacy systems to run web applications within a compatibility environment.
• provide a seamless experience for developers to build web applications. It would be as fast and easy to develop a trivial client/server web application as it is to write “Hello, world!” in PyGTK using Quickly. For bonus points, it would be easy to develop and run web applications locally, and then deploy directly to a PaaS or IaaS cloud.
• empower the user to manage their applications and data regardless of where they are hosted. Traditional operating systems act as a connecting fabric for local applications, providing a shared namespace, file store and IPC mechanisms, but web applications are lacking this. The web’s security model requires that applications are thoroughly sandboxed from each other, but a mediating operating system could connect them in meaningful ways, just as web browsers store cookies and passwords for various websites while protecting them from each other.

Imagine a world where free web applications are as plentiful and malleable as free native applications are today. Developers would be able to branch, test and submit patches to them.

What about Chrome OS?

Chrome OS is a step in the right direction, but doesn’t yet realize this vision. It’s a traditional operating system which is stripped down and focused on running one application (a web browser) very, very well. In some ways, it elevates web applications to first-class status, though its paradigm is still fundamentally that of a web browser.

It is not designed for development, but for consuming the web. Developers who want to create and deploy web applications must use a more traditional operating system to do so.

It does not put the end user in control. On the contrary, the user is almost entirely dependent on SaaS applications for all of their needs.

Although it is constructed using free software, it does not seem to deliver the principles or benefits of software freedom to the web itself.

How?

Just as free software was bootstrapped on proprietary UNIX, the present-day web is fertile ground for the development of free web applications. The web is based on open standards. There are already excellent web development tools, web application frameworks and server software which are FLOSS. Leading-edge web browsers like Firefox and Chrome/Chromium, where much web innovation is happening today, are already open source.

This is a huge head start toward a free web. I think what’s missing is a client platform which catalyzes the development and use of FLOSS web applications.

Syndicated 2010-07-26 10:43:52 from We'll see | Matt Zimmerman

Read, listen, or comprehend: choose two

I have noticed that when I am reading, I cannot simultaneously understand spoken words. If someone speaks to me while I am reading, I can pay attention to their voice, or to the text, but not both. It’s as if these two functions share the same cognitive facility, and this facility can only handle one task at a time. If someone is talking on the phone nearby, I find it very difficult to focus on reading (or writing). If I’m having a conversation with someone about a document, I sometimes have to ask them to pause the conversation for a moment while I read.

This phenomenon isn’t unique to me. In Richard Feynman’s What Do You Care what Other People Think?, there is a chapter entitled “It’s as Simple as One, Two, Three…” where he describes his experiments with keeping time in his head. He practiced counting at a steady rate while simultaneously performing various actions, such as running up and down the stairs, reading, writing, even counting objects. He discovered that he “could do anything while counting to [himself]—except talk out loud”.

What’s interesting is that the pattern varies from person to person. Feynman shared his discovery with a group of people, one of whom (John Tukey) had a curiously different experience: while counting steadily, he could easily speak aloud, but could not read. Through experimenting and comparing their experiences, it seemed to them that they were using different cognitive processes to accomplish the task of counting time. Feynman was “hearing” the numbers in his head, while Tukey was “seeing” the numbers go by.

Analogously, I’ve met people who seem to be able to read and listen to speech at the same time. I attributed this to a similar cognitive effect: presumably some people “speak” the words to themselves, while others “watch” them. Feynman found that, although he could write and count at the same time, his counting would be interrupted when he had to stop and search for the right word. Perhaps he used a different mental faculty for that. Some people seem to be able to listen to more than one person talking at the same time, and I wonder if that’s related.

I was reminded of this years later, when I came across this video on speed reading. In it, the speaker explains that most people read by silently voicing words, which they can do at a rate of only 120-250 words per minute. However, people can learn to read visually instead, and thereby read much more quickly. He describes a training technique which involves reading while continuously voicing arbitrary sounds, like the vowels A-E-I-O-U.

The interesting part, for me, was the possibility of learning. I realized that different people read in different ways, but hadn’t thought much about whether one could change this. Having learned a cognitive skill, like reading or counting time, apparently one can re-learn it a different way. Visual reading would seem, at first glance, to be superior: not only is it faster, but I have to use my eyes to read anyway, so why tie up my listening facility as well? Perhaps I could use it for something else at the same time.

So, I tried the simple technique in the video, and it had a definite effect. I could “feel” that I wasn’t reading in the same way that I had been before. I didn’t measure whether I was going any faster or slower, because I quickly noticed something more significant: my reading comprehension was completely shot. I couldn’t remember what I had read, as the memory of it faded within seconds. Before reaching the end of a paragraph, I would forget the beginning. It was as if my ability to comprehend the meaning of the text was linked to my reading technique. I found this very unsettling, and it ruined my enjoyment of the book I was reading.

I’ll probably need to separate this practice from my pleasure reading in order to stick with it. Presumably, over time, my comprehension will improve. I’m curious about what net effect this will have, though. Will I still comprehend it in “the same” way? Will it mean the same thing to me? Will I still feel the same way about it? The many levels of meaning are connected to our senses as well, and “the same” idea, depending on whether it was read or heard, may not have “the same” meaning to an individual. Even our tactile senses can influence our judgments and decisions.

I also wonder whether, if I learn to read visually, I’ll lose the ability to read any other way. When I retrained myself to type using a Dvorak keyboard layout, rather than QWERTY, I lost the ability to type on QWERTY at high speed. I think this has been a good tradeoff for me, but raises interesting questions about how my mind works: Why did this happen? What else changed in the process that might have been less obvious?

Have you tried re-training yourself in this way? What kind of cognitive side effects did you notice, if any? If you lost something, do you still miss it?

(As a sidenote, I am impressed by Feynman’s exuberance and persistence in his personal experiments, as described in his books for laypeople. Although I consider myself a very curious person, I rarely invest that kind of physical and intellectual energy in first-hand experiments. I’m much more likely to research what other people have done, and skim the surface of the subject.)

Syndicated 2010-07-12 12:57:39 from We'll see | Matt Zimmerman

We’ve packaged all of the free software…what now?

Today, virtually all of the free software available can be found in packaged form in distributions like Debian and Ubuntu. Users of these distributions have access to a library of thousands of applications, ranging from trivial to highly sophisticated software systems. Developers can find a vast array of programming languages, tools and libraries for constructing new applications.

This is possible because we have a mature system for turning free software components into standardized modules (packages). Some software is more difficult to package and maintain, and I’m occasionally surprised to find something very useful which isn’t packaged yet, but in general, the software I want is packaged and ready before I realize I need it. Even the “long tail” of niche software is generally packaged very effectively.

Thanks to coherent standards, sophisticated management tools, and the principles of software freedom, these packages can be mixed and matched to create complete software stacks for a wide range of devices, from netbooks to supercomputing clusters. These stacks are tightly integrated, and can be tested, released, maintained and upgraded as a unit. The Debian system is unparalleled for this purpose, which is why Ubuntu is based on it. The vision, for a free software operating system which is highly modular and customizable, has been achieved.

Rough edges

This is a momentous achievement, and the Debian packaging system fulfills its intended purpose very well. However, there are a number of areas where it introduces friction, because the package model doesn’t quite fit some new problems. Most of these are becoming more common over time as technology evolves and changes shape.

• Embedded systems need to be pared down to the essentials to minimize storage, distribution, computation and maintenance costs. Standardized packaging introduces excessive code, data and interdependency which make the system larger than necessary. Tight integration makes it difficult to bootstrap the system from scratch for custom hardware. Projects like Embedded Debian aim to adapt the Debian system to be more suitable for use in these environments, to varying degrees of success. Meanwhile, smart phones will soon become the most common type of computer globally.
• Data, in contrast to software, has simple requirements. It just needs to be up to date and accessible to programs. Packaging and distributing it through the standardized packaging process is awkward, doesn’t offer tangible benefits, and introduces overhead. There have been extensive debates in Debian about how to handle large data sets. Meanwhile, this problem is becoming increasingly important as data science catalyzes a new wave of applications.
• Client/server and other types of distributed applications are notoriously tricky to package. The packaging system works within the context of a single OS instance, and so relationships which span multiple OS instances (e.g. a server application which depends on a database running on another server) are not straightforward. Meanwhile, the web has become a first-class application development platform, and this kind of interdependency is extremely common on both clients and servers.
• Cross-platform applications such as Firefox, Chromium and OpenOffice.org have long struggled with packaging. In order to be portable, they tend to bundle the components they depend on, such as libraries. Packagers strive for normalization, and want these applications to use the packaged versions of these libraries instead. Application developers build, test and ship one set of dependencies, but their users receive a different stack when they use the packaged version of the application. Developers on both sides are in constant tension as they expect their configuration to be the canonical one, and want it to be tightly integrated. Cross-platform application developers want to provide their own, application-specific cross-platform update mechanism, while distributions want to use the same mechanism for all their components.
• Virtual appliances aim to combine application and operating system into a portable bundle. While a modular OS is definitely called for, appliances face some of the same problems as embedded systems as they need to be minimized. Furthermore, the appliance becomes a component in itself, and requires metadata, distribution mechanisms and so on. If someone wants to “install” a virtual appliance, how should that work? Packaging them up as .debs doesn’t make much sense for the same reasons that apply to large data sets. I haven’t seen virtual appliances really taking off, but I expect cloud to change that.
• Runtime libraries for languages such as Perl, Python and Ruby provide their own packaging systems, which manage dependencies and other metadata, installation, upgrades and removal in a standardized way. Because these operate independently of the OS package manager, all sorts of problems arise. Projects such as GoboLinux have attempted to tie them together, to varying degrees of success. Meanwhile, each new programming language we invent comes with a different, incompatible package manager, and distribution developers need to spend time repackaging them into their preferred format.

Why are we stuck?

I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.
– Abraham Maslow

The packaging ecosystem is very strong. Not only do we have powerful tools for working with packages, we also benefit from packages being a well-understood concept, and having established processes for developing, exchanging and talking about them. Once something is packaged, we know what it is and how to work with it, and it “fits” into everything else. So, it is tempting to package everything in sight, as we already know how to make sense of packages. However, this may not always be the right tool for the job.

Various attempts have been made to extend the packaging concept to make it more general, for example:

• Portage, of Gentoo fame, offers impressive flexibility by building packages with a custom configuration, tailored for the needs of the target system.
• Conary, from rPath, offers finer-grained dependencies, powerful revision control and object-oriented build recipes.
• Nix provides a consistent build and runtime environment, ensuring that programs are run with the same dependencies used to build them, by keeping the relevant versions installed. I don’t know much about it, but it sounds like all dependencies implicitly refer to an exact version.

Other package managers aim to solve a specific problem, such as providing lightweight package management for embedded systems, or lazy dependency installation, or fixing the filesystem hierarchy. There is a long list of package managers of various levels which solve different problems.

Most of these systems suffer from an important fundamental tradeoff: they are designed to manage the entire system, from the kernel through applications, and so they must be used wholesale in order to reap their full benefit. In other words, in their world, everything is a package, and anything which is not a package is out of scope. Therefore, each of these systems requires a separate collection of packages, and each time we invent a new one, its adherents set about packaging everything in the new format. It takes a very long time to do this, and most of them lose momentum before a mature ecosystem can form around them.

This lock-in effect makes it difficult for new packaging technologies to succeed.

Divide and Conquer

No single package management framework is flexible enough to accommodate all of the needs we have today. Even more importantly, a generic solution won’t account for the needs we will have tomorrow. I propose that in order to move forward, we must make it possible to solve packaging problems separately, rather than attempting to solve them all within a single system.

• Decouple applications from the platform. Debian packaging is an excellent solution for managing the network of highly interdependent components which make up the core of a modern Linux distribution. It falls short, however, for managing the needs of modern applications: fast-moving, cross-platform and client/server (especially web). Let’s stop trying to fit these square pegs into round holes, and adopt a different solution for this space, preferably one which is comprehensible and useful to application developers so that they can do most of the work.
• Treat data as a service. It’s no longer useful to package up documentation in order to provide local copies of it on every Linux system. The web is a much, much richer and more effective solution to that problem. The same principle is increasingly applicable to structured data. From documents and contacts to anti-virus signatures and PCI IDs, there’s much better data to be had “out there” on the web than “down here” on the local filesystem.
• Simplify integration between packaging systems in order to enable a heterogeneous model. When we break the assumption that everything is a package, we will need new tools to manage the interfaces between different types of components. Applications will need to introspect their dependency chain, and system management tools will need to be able to interrogate applications. We’ll need thoughtfully designed interfaces which provide an appropriate level of abstraction while offering sufficient flexibility to solve many different packaging problems. There is unarguably a cost to this heterogeneity, but I believe it would easily outweigh the shortcomings of our current model.

But I like things how they are!

We don’t have a choice. The world is changing around us, and distributions need to evolve with it. If we don’t adapt, we will eventually give way to systems which do solve these problems.

Take, for example, modern web browsers like Firefox and Chromium. Arguably the most vital application for users, the browser is coming under increasing pressure to keep up with the breakneck pace of innovation on the web. The next wave of real-time collaboration and multimedia applications relies on the rapid development of new capabilities in web browsers. Browser makers are responding by accelerating deployment in the field: both aggressively push new releases to their users. A report from Google found that Chrome upgrades 97% of their users within 21 days of a new release, and Firefox 85% (both impressive numbers). Mozilla recently changed their maintenance policies, discontinuing maintenance of stable releases and forcing Ubuntu to ship new upstream releases to users.

These applications are just the leading edge of the curve, and the pressure will only increase. Equally powerful trends are pressing server applications, embedded systems, and data to adapt as well. The ideas I’ve presented here are only one possible way forward, and I’m sure there are more and better ideas brewing in distribution communities. I’m sure that I’m not the only one thinking about these problems.

Whatever it looks like in the end, I have no doubt that change is ahead.

Syndicated 2010-07-06 15:31:59 from We'll see | Matt Zimmerman

Navigating the PolicyKit maze

I’ve written a simple application which will automatically extract media from CDs and DVDs when they are inserted into the drive attached to my server. This makes it easy for me to compile all of my media in one place and access it anytime I like. The application uses the modern udisks API, formerly known as DeviceKit-disks, and I wrote it in part to learn get some experience working with udisks (which, it turns out, is rather nice indeed).

Naturally, I wanted to grant this application the privileges necessary to mount, unmount and eject removable media. The server is headless, and the application runs as a daemon, so this would require explicit configuration. udisks uses PolicyKit for authorization, so I expected this to be very simple to do. In fact, it is very simple, but finding out exactly how to do it wasn’t quite so easy.

The Internet is full of web pages which recommend editing /etc/PolicyKit/PolicyKit.conf. As far as I can tell, nothing pays attention to this file anymore, and all of these instructions have been rendered meaningless. My system was also full of tools like polkit-auth, from the apparently-obsolete policykit package, which kept their configuration in some other ignored place, i.e. /var/lib/PolicyKit. It seems the configuration system has been through a revolution or two recently.

In Ubuntu 10.04, the right place to configure these things seems to be /var/lib/polkit-1/localauthority, and this is documented in pklocalauthority(8). Authorization can be tested using pkcheck(1), and the default policy can be examined using pkaction(1).

I solved my problem by creating a file in /var/lib/polkit-1/localauthority/50-local.d with a .pkla extension with the following contents:

[Access to removable media for the media group]
Identity=unix-group:media
Action=org.freedesktop.udisks.drive-eject;org.freedesktop.udisks.filesystem-mount
ResultAny=yes

This took effect immediately and did exactly what I needed. I lost quite some time trying to figure out why the other methods weren’t working, so perhaps this post will save the next person a bit of time. It may also inspire some gratitude for the infrastructure which makes all of this work automatically for more typical usage scenarios, so that most people don’t need to worry about any of this.

Along the way, I whipped up a patch to add a --eject option to the handy udisks(1) tool, which made it easier for me to test along the way.

Syndicated 2010-06-27 14:38:04 from We'll see | Matt Zimmerman