Older blog entries for StephanPeijnik (starting at number 20)

Android, Mythbusters and openness

I have been reading a great many posts about Android lately, some consisting of criticism, some of praise and some simply addressing issues in the Android "community". Let's have a look at those.

Matt Porter's Android Mythbusters presentation and Harald Welte's reaction


I haven't seen the presentation live, but I had a look at the slides. Impressing work done by Matt putting all this information together. However, we all knew that Android only (ab-)uses Linux, without making use of the GNU userland for a long time, didn't we?

In his presentation Matt has shown things such as Android's udev "replacement" that uses hardcoded values for device node creation and (on his blog) Harald has then come up with a statement I have found to be very strong:

The presentation shows how Google has simply thrown 5-10 years of Linux userspace evolution into the trashcan and re-implemented it partially for no reason. Things like hard-coded device lists/permissions in object code rather than config files, the lack of support for hot-plugging devices (udev), the lack of kernel headers. A libc that throws away System V IPC that every unix/Linux software developer takes for granted. The lack of complete POSIX threads. I could continue this list, but hey, you should read those slides. now!

Now both of these statements target technical details, but the root of the problem seems to be elsewhere.

Where is my Android 2.0?

Okay, that heading might not be making any sense in the context of this post at a first glance, but let me elaborate on that. Google and the Open Handset Alliance refer to Android as being an "Open Source" operating system, but the project is different from "real" Free Software projects: development takes place in a closed group and the results are shared with the community later on, when they are deemed to be ready.

This means that innovation also takes place behind closed curtains and that the community is not involved in the actual development process at all. Lately we have seen the result of that, as Motorola is bragging about working close with Google on Android 2.0 ("Eclair"), but the AOSP source trees, open for everyone to have a look at, show no signs of version 2.0. In fact no changes that might even remotely suggest the release of a new major version have been made public in the past few weeks. So where is the openess there?
Actually, the Motorola Droid has already shipped with Eclair on 6th, but still, there is no indication that Eclair will be made available to the broader public.

In short Android seems to be developed behind closed curtains, with hardly (read no) community input whatsoever and is sometimes released as Free Software, not what I would describe as an open development process.

The Android Market problem

As we have seen in the past Google is enforcing their copyright on proprietary applications that ship with pretty much every Android device, such as the Android Market. This has become really clear when Steve Kondik received  a cease and desist letter when packing the Google-proprietary applications into his ROMs. Okay, it's Google's right to enforce their copyright and there is nothing wrong with actually doing so, the thing I really have a problem with is something else: the Market is proprietary.

Now what this means should become rather clear. You can have an Android device without Google's proprietary bits, but with default settings you just do not have any way of installing additional software. In my opinion the Market should be freed by Google themselves, or the community has to react and come up with a free replacement to overcome the vendor lock-in. Oh, you might know a replacement called SlideMe (or Mobentoo) already. Well, that bugger is proprietary too, so not a solution at all.

Nokia and Maemo to the rescue

In most discussions about the openness of Android someone throws in Nokia and Maemo, as a solution to the dilemma. Reading all those positive comments I simply had to give it a try, but all my hopes were destroyed within a few minutes.

Let's start with the good news and let alone the reason why my hopes were destroyed for another minute or two. Maemo is based on Debian GNU/Linux and various Free Software components, such as GTK+, gstreamer, esd and friends. Most of the system is Free Software which is a good thing(tm) and reading all of this really got me into Maemo. Okay, some applications seem to be proprietary, but I am sure that could be fixed rather easily, so I could once for all use a truly open phone.

...and then came the SDK installer shell script:

#!/bin/sh
# Copyright (C) 2006-2009 Nokia Corporation
#
# This is proprietary software owned by Nokia Corporation.
#
# Contact: Maemo Integration <integration@maemo.org>
# Version: $Revision: 1110 $

Now there is one question you should ask yourself: Why would someone trying to promote his platform as being open make the *installer* script for its SDK proprietary? Come on, it's an installer script, how much of your secret juice could be in there? What's the problem with people modifying it and working on this installer script in an open development environment?

I had high hopes for Nokia actually doing a bit better than Google, but it seems they've failed to do so. It may be me overreacting, but a proprietary SDK installer shell script scares me enough not to install the SDK and have a look at it for now nor to think about buying a Maemo-based device in the near future. Please Nokia, either get the facts straight or provide us with a free SDK to your free & open platform.

So, in short, Google is bad at working with the community and creating a truly open development process, and Nokia simply fails in terms of not scaring off prospective developers for their open platform with the proprietary SDK installer. Do you have any solutions in terms of an open phone environment, apart from what OpenMoko has come up with?

Syndicated 2009-11-08 04:49:00 (Updated 2009-11-09 20:21:28) from sp

How to move panels in Gnome 2.28

I just installed Ubuntu Karmic Koala on my workstation and came across the problem of not being able to move/drag Gnome panels around in order to have the panels on my primary monitor.
On the Debian system that was powering the workstation before this was a non-issue as I could simply click, hold and drag both the upper and the lower panel, but this didn't work.

So, after a few minutes of googling I came across an entry at answers.launchpad.net[0] and a blog post, but I cannot seem to remember the URL to that one. I can imagine that some of you might be having the exact same problem, so the solution is holding down the ALT, whilst dragging as usual.

[0] https://answers.launchpad.net/ubuntu/+source/gnome-panel/+question/264

Syndicated 2009-11-04 10:04:00 (Updated 2009-11-09 20:21:28) from sp

Automagic bug reporting in Python applications for Debian

We all know this situation: a program crashes and you need to send a bug report to the DBTS. The damn bug however is hard to reproduce and you fail to do so and hence can't submit the report.

This has all changed for update-manager now. With the next upload to unstable update-manager will get automagic bug reporting. In short: there is code that detects uncaught exceptions, asks the user if he or she wants to file a bug report and then invokes reportbug. Nothing too special about this yet. There is one thing that should make lives of both bug reporters and developers easier though: the code automatically includes traceback information, that make finding the cause of the problem a lot easier.

Okay, enough of praising this feature of update-manager, this post is about something else. Ubuntu users and developers might think "apport" now, because apport is an application that provides exactly this, reporting of bugs on program crashes, for all users.

At least for Python applications and libraries in Debian providing this functionality should be easy. The only thing one has to do is create a sys.excepthook implementation that does the bug reporting, just as in update-manager.

The questions I have now are:

Do you think this feature would be a good addition to the Debian distribution?

Syndicated 2009-08-17 19:23:00 (Updated 2009-11-09 20:30:21) from sp

update-manager weekly update #6

So finally I have the time to provide you with a weekly update, instead of my usual bi-weekly ones.

Unfortunately I did not work on anything on last week's TODO list, but found other issues I worked on and corrected.  So let's have a look at what I've done.

Debian packaging update

I have done some work on the Debian packaging, which allows update-manager to be built using dpkg-buildpackage now. The way packages are splitted is not finalized yet and not up-to-date with my (and my mentor's) idea of how we should do that. You can expect an update to that soonish.

Automatically invoking package list reloading / update check

There is a command line switch (namely -c, or --check) now, that automatically performs an update check on startup. This gives other programs, like software-properties, a way of forcing a check when, for example, the package list sources have changed.

Checking/unchecking all updates in Gtk frontend

Finally the small feature of selecting or deselecting all updates works in the Gtk frontend. Special cases like "all updates already checked" or "no updates checked" yet are handled too, meaning that you can only use one of these methods if it actually makes sense.

Package dependencies in python-apt backend and Gtk frontend

Both the python-apt backend and the Gtk frontend are now aware of package dependencies. This means that when you select an upgrade that depends on another one that other update is selected too. The same works vice-versa too. Additionally the UI now lists all dependencies and dependencies on packages that are not installed yet and automatically deselects all updates that would requires new packages to be installed.

Displaying of overall download size in Gtk frontend

There has been a missing feature (ok, maybe a bug) so that the displayed download size would not be updated in the Gtk frontend. This has been fixed.

Install button being set sensitive correctly in Gtk frontend

In the past the install button would be set to either sensitive or insensitive at startup and not updated afterwards. That means if there were no packages to update when starting update-manager, then checking for updates where new updates are found, the install button would not be set sensitive again. I fixed that too.

Sorting of packages in Gtk frontend

In the Gtk frontend packages were not sorted at all, which meant that finding a specific package was rather hard. I added code that sorts the update list by package name now, which solves this issue.

Bugfixing humanize_size

The humanize_size method, which is responsible for human-readable size displaying in the Gtk frontend contained a major bug so that sizes were rounded. Again, I was able to solve this.

Next week's TODO list

As I didn't find time to work on last week's TODO list my new TODO list is in fact my old one, with additional "Bugfixing" and "Debian packaging" tasks:

• Downloading and installing of updates


• Bugfixing (?)


• Debian packaging


• Checking that everything is documented


• Even more unit tests


• Pylint checking


• If time permits and everything else works correctly: working on an aptdaemon backend

The next thing you can expect me to update is the Debian packaging and the documentation, which are my highest priority tasks for now, followed by support for downloading and installing updates.

Happy hacking!

Syndicated 2009-07-09 18:07:00 (Updated 2009-11-09 20:21:28) from sp

update-manager weekly update #5

Firstly I have to apologize again for not providing you with weekly update #4, but again I didn't have the time to write one, so this post is going to sum up everything that happened since my last update.

Let's have a look at my previous TODO list:

Documentation

Even though my TODO list entry contained a more detailed entry I have updated the UpdateManager documentation as a whole, leaving only a few blank spots right now.

Ubuntu distribution specific code

I implemented changelog fetching for Ubuntu, which works just as fine as its Debian counterpart now.

More unit tests

There are plenty of unit tests now, but not everything is being tested yet. I am especially proud of my Python interface validation code, that is being used in unit tests to check if handlers implement an interface correctly.

Update list downloading

Checking for updates is what caused me major trouble in the past few days. Basically I had all the code ready, but for some reason the UI froze, with no apparent reason.
However, today I was able to finally identify and fix the problem. As I expected my code was just fine, but python-apt was messing up. I am going to discuss the exact problem and its solution later on, but first: a screenshot. :-)

Note: As you probably noticed I replaced the default progressbar with a pulsating one, because we cannot get exact information on how many items/bytes to fetch and would likely get a progress bar moving backwards, which isn't beautiful.

Further changes

The TODO list was rather short and I did a lot of other work, which I want to elaborate on.

Dynamic selection of frontend, backend and distribution specific modules

Even though this is probably not of any interest to John Doe, it helps a great deal when debugging code as all three components can be selected via separate command line switches now.
Additionally some magic has been put in place that automatically detects the system's distribution and loads the corresponding distribution specific module. This is done via lsb_release and the newly introduced code in UpdateManager.Util.lsb.

Pylint cleanup

Just out of curiosity I decided to start a pylint run on the codebase and quite a few problems were detected, which I then fixed. To be honest though I added quite some code afterwards that probably needs pylint checking and fixes again.

update-manager IPC

My original plan and IPC design involved using callback functions and passing them between the different modules. Even though this worked out fine I had the feeling this wasn't clean enough and decided to ditch this approach and replace it with handler classes.
The handler base classes now provide an interface of methods that are called on certain events and their implementations act accordingly. The main benefit was that I could easily drop a lot of enums and rather have different methods handling different events.

Gtk, threads and python-apt

With the new IPC approach it became easier to use threads that do the actual work in the background, which I had implemented in next to no time, but a few problems showed up.
Whilst cache reloading from within a thread worked just fine checking for updates did not, and until today I didn't know why. I spent a good amount of time debugging this issue, even using python profiling, but nothing obvious showed up. The background process was running, whilst the UI froze.
Today I finally found the root of the problem: python-apt. Even though I assumed that the python-apt worker threads must be stealing CPU time from the thread running gtk.main I wasn't sure how this could be happening, having two completely independent threads.

Now, the cause of all this mess was that Python has a global threading lock and it seems as if this one is *LOCKED* when running C-code, such as the one python-apt comes with. The solution lies in calling Py_BEGIN_THREADS_ALLOW and Py_END_THREADS_ALLOW from within the C code, to release the global lock and let the Python interpreter do some work every now and then.

As with the python-apt acquire code I was able to allow other threads to work as soon as the fetching code starts working and only disallow threads when actually modifying Python objects or calling methods and/or functions. Surprisingly python-apt already made use of this in its cache loading code, but not the fetch progress code.
Fixing this problem took me less than half an hour and you probably can't believe how glad I was to finally get things working again.

UI updates & other changes


Some details in the UI were anything but optimal, like horizontal scrollbars in a few places, which I removed. Additionally I saw the need to move some code out of the Gtk frontend's __init__.py file and to a separate ui.py file.
A full list of all changes I made is available from the bzr changelog at bzr.debian.org.

A few more screenshots

Finally, I would like to provide you with two more screenshots (don't worry about my system being insecure because of not applied updates - this is a testing machine that is  not up-to-date on purpose):





TODO list

My TODO list for next week:

• Downloading and installing of updates


• Checking that everything is documented


• Even more unit tests


• Pylint checking


• If time permits and everything else works correctly: working on an aptdaemon backend

Syndicated 2009-07-02 10:29:00 (Updated 2009-11-09 20:21:28) from sp

Python interface validation

When I started working on update-manager I thought using zope.interface for my interfaces was a good idea, but soon realized that it lacked a way of actually validating a given interface against an implementation. The only thing it did was checking whether the implementation defined that it implements the interface.

Now, whilst writing some unit tests for update-manager I came up with a simple way of doing "real" validation, and I would like to share that Python code with you.

Firstly, I'd like to give you an overview of which checks my code carries out:

• Mandatory method (raises NotImplementedError in interface definition) is not implemented (also raises NotImplementedError in implementation)


• Optional or mandatory method is of correct type (static method versus instance method)


• Optional or mandatory method has a different signature (argument count is different)

I consider at least the first and last check viable for validation of an interface against its implementation. The second check I listed is not that useful, and may produce false positives when someone uses certain decorators, I did not carry out any tests on that myself though.

The code can be found in update-manager's repository (link) and (for now) is licensed under the GPLv2 or later. I am willing to distribute this code as a separate Python module (maybe under a more permissive license like the LGPL) if enough (let's say at least two) people are interested in it, so please let me know if you like it.

Apart from the code itself the unit tests in the file linked above should explain how this beast exactly works.

Happy hacking!

Syndicated 2009-06-22 21:30:00 (Updated 2009-11-09 20:21:28) from sp

update-manager weekly update #2

First of all: yes, I skipped update #1. I was rather busy with some assignments and exams at university and didn't work that much on update-manager the past two weeks.

Anyways, this update contains everything that has happened since update #0.

Changelog fetching

The changelog fetching code has been added to update-manager. This means that the changelog will be shown in the details section now and should look the same it looked before. However, I have only written that code for Debian so far, but the Ubuntu part is on my TODO list.

Documentation

The documentation has been updated and uploaded to alioth and can be viewed here. I have set up a python environment on alioth which allows building the documentation directly, rather than building it locally and uploading it then. Basically this works by having a separate python packages directory, containing some mock modules that are needed (think gtk and friends here), allowing us to build the docs without having to install all dependencies.
I am planning on elaborating on this method and how to create such an environment in one of my upcoming posts, so stay tuned if you could use something like this too.

Additionally to this environment the documentation has been updated a great deal, including more modules and containing documentation for previously undocumented methods and classes.

Application module

I have reworked some aspects of the UpdateManager.Application module, allowing me to do unit testing on pretty much every aspect of the class. The problem I fixed here is that Application directly called sys.exit when something went wrong and now raises exceptions, which contain the status code and are handled in the respective scripts (ie. "update-manager").

Gtk Frontend and updates from another thread

One thing I fixed was the problem caused by the changelog fetching code running in a separate thread and invoking a callback function that updates the UI. It seems as Gtk isn't that happy when you do this and the UI wouldn't be updated immediatly (it seemed that this only happened after some events, like scrolling the update list). This has been reworked and the callback function now checks if it was called from the main thread or not and calls gtk.gdk.threads_enter/_leave accordingly.

Changelog Viewer

After finishing the changelog fetching code I added the ChangelogViewer widget from previous update-manager versions again, supporting creation of links to launchpad and debian bugs (ie. LP:NNNNNN and Closes: #NNNNNN are now links) and displaying the version number in bold, among other things.

Weeding out UpdateManager.Frontend.Gtk.utils

Initially I just copied over the utils module from old update-manager to the new implementation, leaving every single function in there, but now I decided to weed out the module. The result is that only the functions actually used by this implementation remained in there. Related to this documentation of that module is pending and on my TODO list.

Version number

After a chat with my mentor we decided to bump update-manager's version to 0.200-pre. This should make it easier to distinguish from the old version and indicates that a lot has changed. The first release following the -pre series will be 0.200.0, which should then include all functionality old update-manager included.

My TODO list for next week

Ordered by priority

• Documentation of UpdateManager.Frontend.Gtk.utils and .ChangelogViewer modules


• Ubuntu Distribution Specific code


• More unit tests


• Update list downloading in Gtk frontend

Syndicated 2009-06-19 02:27:00 (Updated 2009-11-09 20:21:28) from sp

Should CLI debug output and error messages be localized in a GUI application?

Whilst working on update-manager I have been wondering whether I should use gettext for localizing debug output and error messages sent to stderr.
As for debug output itself I basically do not see the need for providing a localized version for each and every message sent to stderr, but as far as error messages are concerned I am uncertain.

The point is that update-manager (apart from its experimental text interface) is usually not launched from a terminal at all and so most users won't even see these messages ever. Also, I believe that every developer's English skills are good enough so that he or she is able to understand simple messages.
Error messages however might be useful to all users when they experience a problem with the software, but localizing those could make handling bug reports a bit harder, possibly having to translate the error message back to English before being able to see what has gone wrong.

So basically I am asking you: What do you think? Is it worth localizing these messages? What is your experience with localized or non-localized error and debug messages?

I would be glad if I could get some input from you, either as a comment to this article, via email to debian(dot)sp(dot)or(dot)at or through the update-manager-devel mailing list.

Syndicated 2009-06-02 06:55:00 (Updated 2009-11-09 20:21:28) from sp

sphinx-aware Enums in Python

As I promised to keep you updated on recent developments on update-manager I am writing this article. Just as a disclaimer: I am not going to write about any recent developments here, but would rather like to point at a piece of code I added to update-manager that could be useful in other applications too.

Now, as the title suggests there are sphinx-aware Enums in update-manager. Enums are common constructs in other programming languages like C and allow simple creation of constants with, for example, ascending values (first constant has value 0, second has value 1 and so on). Python unfortunately does not include support for Enums itself, but I found it rather easy to write classes that emulate such a construct.

Nothing is new about Enums in Python and there are probably quite a few different implementations out there, but I believe mine is different. The sphinx-aware part means that my implementation automagically updates the docstrings of the created instances and thus allows sphinx' "autodata" method to include sensible information in generated API documentation.

I could go on writing about and praising my method, but I believe a short example gives you a better idea how my implementation works and what I wanted to achieve with this. Have a look at this page, which is part of update-manager's new API documentation. You should see rather well-looking documentation of the UpdateManager.Backend.RELOAD_CACHE_STATUS NegativeEnum, the defined constants, their values and some additional information about each value now.

Still, nothing too fancy, HTML documentation generated from docstrings. What makes this special is the code from which it was generated:

RELOAD_CACHE_STATUS = NegativeEnum(
  BEGIN = "Started reloading package cache",
  DONE = "Finished reloading package cache")

This not only gives us a RELOAD_CACHE_STATUS enum, along with the RELOAD_CACHE_STATUS.BEGIN and RELOAD_CACHE_STATUS.DONE, but also some documentation, included in RELOAD_CACHE_STATUS' docstring, that can be used by sphinx.

You can find the Enum code, which is rather short and should be quite easy to understand, here. I hope you find this code as useful as I do.

Syndicated 2009-06-01 18:20:00 (Updated 2009-11-09 20:21:28) from sp

update-manager on alioth

As I noted in this weeks update-manager progress update one of my tasks was to create an alioth.debian.org project and get my branches uploaded to Debian.

I did not imagine that alioth admins (hi there, a huge "thank you" goes to you guys) would be this fast with reviewing and accepting the project and enabling bazaar support for me.
Anyways, the project has been accepted and its new home is on alioth. I have also already uploaded both my update-manager branch and python-apt branch to bzr.debian.org

Additionally I have generated the API documentation, which is also hosted on alioth, and created a development disccusion mailing list, update-manager-devel at lists.alioth.debian.org.

If you are interested in this project feel free to have a look at what I've done so far and join the development discussion. Comments, critizism and ideas are always welcome.

Syndicated 2009-05-28 11:44:00 (Updated 2009-11-09 20:21:28) from sp