Older blog entries for StephanPeijnik (starting at number 16)

update-manager weekly update #5

Firstly I have to apologize again for not providing you with weekly update #4, but again I didn't have the time to write one, so this post is going to sum up everything that happened since my last update.

Let's have a look at my previous TODO list:

Documentation

Even though my TODO list entry contained a more detailed entry I have updated the UpdateManager documentation as a whole, leaving only a few blank spots right now.

Ubuntu distribution specific code

I implemented changelog fetching for Ubuntu, which works just as fine as its Debian counterpart now.

More unit tests

There are plenty of unit tests now, but not everything is being tested yet. I am especially proud of my Python interface validation code, that is being used in unit tests to check if handlers implement an interface correctly.

Update list downloading

Checking for updates is what caused me major trouble in the past few days. Basically I had all the code ready, but for some reason the UI froze, with no apparent reason.
However, today I was able to finally identify and fix the problem. As I expected my code was just fine, but python-apt was messing up. I am going to discuss the exact problem and its solution later on, but first: a screenshot. :-)

Note: As you probably noticed I replaced the default progressbar with a pulsating one, because we cannot get exact information on how many items/bytes to fetch and would likely get a progress bar moving backwards, which isn't beautiful.

Further changes

The TODO list was rather short and I did a lot of other work, which I want to elaborate on.

Dynamic selection of frontend, backend and distribution specific modules

Even though this is probably not of any interest to John Doe, it helps a great deal when debugging code as all three components can be selected via separate command line switches now.
Additionally some magic has been put in place that automatically detects the system's distribution and loads the corresponding distribution specific module. This is done via lsb_release and the newly introduced code in UpdateManager.Util.lsb.

Pylint cleanup

Just out of curiosity I decided to start a pylint run on the codebase and quite a few problems were detected, which I then fixed. To be honest though I added quite some code afterwards that probably needs pylint checking and fixes again.

update-manager IPC

My original plan and IPC design involved using callback functions and passing them between the different modules. Even though this worked out fine I had the feeling this wasn't clean enough and decided to ditch this approach and replace it with handler classes.
The handler base classes now provide an interface of methods that are called on certain events and their implementations act accordingly. The main benefit was that I could easily drop a lot of enums and rather have different methods handling different events.

Gtk, threads and python-apt

With the new IPC approach it became easier to use threads that do the actual work in the background, which I had implemented in next to no time, but a few problems showed up.
Whilst cache reloading from within a thread worked just fine checking for updates did not, and until today I didn't know why. I spent a good amount of time debugging this issue, even using python profiling, but nothing obvious showed up. The background process was running, whilst the UI froze.
Today I finally found the root of the problem: python-apt. Even though I assumed that the python-apt worker threads must be stealing CPU time from the thread running gtk.main I wasn't sure how this could be happening, having two completely independent threads.

Now, the cause of all this mess was that Python has a global threading lock and it seems as if this one is *LOCKED* when running C-code, such as the one python-apt comes with. The solution lies in calling Py_BEGIN_THREADS_ALLOW and Py_END_THREADS_ALLOW from within the C code, to release the global lock and let the Python interpreter do some work every now and then.

As with the python-apt acquire code I was able to allow other threads to work as soon as the fetching code starts working and only disallow threads when actually modifying Python objects or calling methods and/or functions. Surprisingly python-apt already made use of this in its cache loading code, but not the fetch progress code.
Fixing this problem took me less than half an hour and you probably can't believe how glad I was to finally get things working again.

UI updates & other changes


Some details in the UI were anything but optimal, like horizontal scrollbars in a few places, which I removed. Additionally I saw the need to move some code out of the Gtk frontend's __init__.py file and to a separate ui.py file.
A full list of all changes I made is available from the bzr changelog at bzr.debian.org.

A few more screenshots

Finally, I would like to provide you with two more screenshots (don't worry about my system being insecure because of not applied updates - this is a testing machine that is  not up-to-date on purpose):





TODO list

My TODO list for next week:

• Downloading and installing of updates


• Checking that everything is documented


• Even more unit tests


• Pylint checking


• If time permits and everything else works correctly: working on an aptdaemon backend

Syndicated 2009-07-02 10:29:00 (Updated 2009-11-09 20:21:28) from sp

Python interface validation

When I started working on update-manager I thought using zope.interface for my interfaces was a good idea, but soon realized that it lacked a way of actually validating a given interface against an implementation. The only thing it did was checking whether the implementation defined that it implements the interface.

Now, whilst writing some unit tests for update-manager I came up with a simple way of doing "real" validation, and I would like to share that Python code with you.

Firstly, I'd like to give you an overview of which checks my code carries out:

• Mandatory method (raises NotImplementedError in interface definition) is not implemented (also raises NotImplementedError in implementation)


• Optional or mandatory method is of correct type (static method versus instance method)


• Optional or mandatory method has a different signature (argument count is different)

I consider at least the first and last check viable for validation of an interface against its implementation. The second check I listed is not that useful, and may produce false positives when someone uses certain decorators, I did not carry out any tests on that myself though.

The code can be found in update-manager's repository (link) and (for now) is licensed under the GPLv2 or later. I am willing to distribute this code as a separate Python module (maybe under a more permissive license like the LGPL) if enough (let's say at least two) people are interested in it, so please let me know if you like it.

Apart from the code itself the unit tests in the file linked above should explain how this beast exactly works.

Happy hacking!

Syndicated 2009-06-22 21:30:00 (Updated 2009-11-09 20:21:28) from sp

update-manager weekly update #2

First of all: yes, I skipped update #1. I was rather busy with some assignments and exams at university and didn't work that much on update-manager the past two weeks.

Anyways, this update contains everything that has happened since update #0.

Changelog fetching

The changelog fetching code has been added to update-manager. This means that the changelog will be shown in the details section now and should look the same it looked before. However, I have only written that code for Debian so far, but the Ubuntu part is on my TODO list.

Documentation

The documentation has been updated and uploaded to alioth and can be viewed here. I have set up a python environment on alioth which allows building the documentation directly, rather than building it locally and uploading it then. Basically this works by having a separate python packages directory, containing some mock modules that are needed (think gtk and friends here), allowing us to build the docs without having to install all dependencies.
I am planning on elaborating on this method and how to create such an environment in one of my upcoming posts, so stay tuned if you could use something like this too.

Additionally to this environment the documentation has been updated a great deal, including more modules and containing documentation for previously undocumented methods and classes.

Application module

I have reworked some aspects of the UpdateManager.Application module, allowing me to do unit testing on pretty much every aspect of the class. The problem I fixed here is that Application directly called sys.exit when something went wrong and now raises exceptions, which contain the status code and are handled in the respective scripts (ie. "update-manager").

Gtk Frontend and updates from another thread

One thing I fixed was the problem caused by the changelog fetching code running in a separate thread and invoking a callback function that updates the UI. It seems as Gtk isn't that happy when you do this and the UI wouldn't be updated immediatly (it seemed that this only happened after some events, like scrolling the update list). This has been reworked and the callback function now checks if it was called from the main thread or not and calls gtk.gdk.threads_enter/_leave accordingly.

Changelog Viewer

After finishing the changelog fetching code I added the ChangelogViewer widget from previous update-manager versions again, supporting creation of links to launchpad and debian bugs (ie. LP:NNNNNN and Closes: #NNNNNN are now links) and displaying the version number in bold, among other things.

Weeding out UpdateManager.Frontend.Gtk.utils

Initially I just copied over the utils module from old update-manager to the new implementation, leaving every single function in there, but now I decided to weed out the module. The result is that only the functions actually used by this implementation remained in there. Related to this documentation of that module is pending and on my TODO list.

Version number

After a chat with my mentor we decided to bump update-manager's version to 0.200-pre. This should make it easier to distinguish from the old version and indicates that a lot has changed. The first release following the -pre series will be 0.200.0, which should then include all functionality old update-manager included.

My TODO list for next week

Ordered by priority

• Documentation of UpdateManager.Frontend.Gtk.utils and .ChangelogViewer modules


• Ubuntu Distribution Specific code


• More unit tests


• Update list downloading in Gtk frontend

Syndicated 2009-06-19 02:27:00 (Updated 2009-11-09 20:21:28) from sp

Should CLI debug output and error messages be localized in a GUI application?

Whilst working on update-manager I have been wondering whether I should use gettext for localizing debug output and error messages sent to stderr.
As for debug output itself I basically do not see the need for providing a localized version for each and every message sent to stderr, but as far as error messages are concerned I am uncertain.

The point is that update-manager (apart from its experimental text interface) is usually not launched from a terminal at all and so most users won't even see these messages ever. Also, I believe that every developer's English skills are good enough so that he or she is able to understand simple messages.
Error messages however might be useful to all users when they experience a problem with the software, but localizing those could make handling bug reports a bit harder, possibly having to translate the error message back to English before being able to see what has gone wrong.

So basically I am asking you: What do you think? Is it worth localizing these messages? What is your experience with localized or non-localized error and debug messages?

I would be glad if I could get some input from you, either as a comment to this article, via email to debian(dot)sp(dot)or(dot)at or through the update-manager-devel mailing list.

Syndicated 2009-06-02 06:55:00 (Updated 2009-11-09 20:21:28) from sp

sphinx-aware Enums in Python

As I promised to keep you updated on recent developments on update-manager I am writing this article. Just as a disclaimer: I am not going to write about any recent developments here, but would rather like to point at a piece of code I added to update-manager that could be useful in other applications too.

Now, as the title suggests there are sphinx-aware Enums in update-manager. Enums are common constructs in other programming languages like C and allow simple creation of constants with, for example, ascending values (first constant has value 0, second has value 1 and so on). Python unfortunately does not include support for Enums itself, but I found it rather easy to write classes that emulate such a construct.

Nothing is new about Enums in Python and there are probably quite a few different implementations out there, but I believe mine is different. The sphinx-aware part means that my implementation automagically updates the docstrings of the created instances and thus allows sphinx' "autodata" method to include sensible information in generated API documentation.

I could go on writing about and praising my method, but I believe a short example gives you a better idea how my implementation works and what I wanted to achieve with this. Have a look at this page, which is part of update-manager's new API documentation. You should see rather well-looking documentation of the UpdateManager.Backend.RELOAD_CACHE_STATUS NegativeEnum, the defined constants, their values and some additional information about each value now.

Still, nothing too fancy, HTML documentation generated from docstrings. What makes this special is the code from which it was generated:

RELOAD_CACHE_STATUS = NegativeEnum(
  BEGIN = "Started reloading package cache",
  DONE = "Finished reloading package cache")

This not only gives us a RELOAD_CACHE_STATUS enum, along with the RELOAD_CACHE_STATUS.BEGIN and RELOAD_CACHE_STATUS.DONE, but also some documentation, included in RELOAD_CACHE_STATUS' docstring, that can be used by sphinx.

You can find the Enum code, which is rather short and should be quite easy to understand, here. I hope you find this code as useful as I do.

Syndicated 2009-06-01 18:20:00 (Updated 2009-11-09 20:21:28) from sp

update-manager on alioth

As I noted in this weeks update-manager progress update one of my tasks was to create an alioth.debian.org project and get my branches uploaded to Debian.

I did not imagine that alioth admins (hi there, a huge "thank you" goes to you guys) would be this fast with reviewing and accepting the project and enabling bazaar support for me.
Anyways, the project has been accepted and its new home is on alioth. I have also already uploaded both my update-manager branch and python-apt branch to bzr.debian.org

Additionally I have generated the API documentation, which is also hosted on alioth, and created a development disccusion mailing list, update-manager-devel at lists.alioth.debian.org.

If you are interested in this project feel free to have a look at what I've done so far and join the development discussion. Comments, critizism and ideas are always welcome.

Syndicated 2009-05-28 11:44:00 (Updated 2009-11-09 20:21:28) from sp

update-manager weekly update #0

It has been more than a month since I last wrote about my work on update-manager during this year's Google Summer Of Code and I am somewhat ashamed I wasn't able to provide you with updates more regularly.

So first of all, yes, I did do some work and yes, there has been quite some progress. Basically both private and university stuff have kept me from writing and that's why I'd like to start with this series of weekly updates today.
This series are meant to summarize what has happened during a week of writing code and give you an overview of what's happening. This first issue however will sum up the past month.

So let me begin explaining what has happened since my last post.

update-manager bazaar repository

All code I have written so far is available through a public bazaar branch on launchpad.net. My branch's page can be found here and provides you with its history and of course instructions on how to obtain the code. The location is only temporary though, as I am going to move hosting over to alioth.debian.org. This is on my task list for next week.

modular design

I have ripped apart nearly all of update-manager and put it together in a more modular way, which should implementing new frontends or backends more easy, whilst also simplifying code maintenance.

The new design consists of four major parts:

• The application class is responsible for parsing command line arguments, initializing all other components correctly and coordinate communication between the frontend and the backend.


• The backend itself is defined through the UpdateManager.Backend.BackendBase class and each implementation subclasses BackendBase. It is responsible for interacting with apt.


• The frontend is again defined through a base class, UpdateManager.Frontend.FrontendBase. This part of update-manager provides the userinterface, handles user input and starts operations accordingly.


• Lastly there is the distribution-specific part, which lives inside the UpdateManager.DistSpecific Python module and is defined by its own base class, DistBase.

backend implementation

When I started working on update-manager it heavily relied on synaptic and used it to do the dirty-work. However, together with my mentor, mvo, I decided to drop synaptic support and rather concentrate on using python-apt. This means that the only backend implementation right now is a python-apt backend.

The python-apt backend is currently a work in progress, but already includes some basic functionality. Right now it can (re-)load the package cache and package lists and is able to provide a list of packages which are upgradable to the frontend.
Whilst implementing these functions I noticed some shortcomings of python-apt itself, fixed those and got mvo included in his python-apt branch at launchpad.

frontend implementation

I started re-implementing the Gtk frontend as provided through current update-manager and right now it visualizes the package cache reloading process and provides users with a list of upgradable packages. However, that's pretty much all of the functionality it includes right now, which is why implementing more functions is pretty much on the top of my todo list.

Additionally I have ported the text frontend, as included in Ubuntu, to the new modular system, and this frontend's code really shows how easy adding a frontend with the new modular design is. This frontend contains the same functionality as the Gtk frontend.

distribution specific code

The core described above does not include any distribution specific code anymore, which is the main focus of this project. The implementations of distribution-specific functionality contains classifiers for update categories for both Debian and Ubuntu, whilst I focused on getting things right with the Debian implementation for now. These classifiers allow the frontend to let the user know which kind of update they are about to install, like a security update, a recommended upgrade or a third-party (unofficial) upgrade.

documentation

As update-manager was poorly (read: hardly at all) documented I started documenting the API using sphinx. However, right now the generated documentation cannot be found anywhere yet. This should change as soon as an alioth project for update-manager has been created.

next week's tasks

I would also like to provide you with my task list for the coming week. The list, ordered by priority, is:

• Register an alioth project and move the bazaar branch to bzr.debian.org


• Generate an HTML version of the API documentation and put it on alioth.


• Implement changelog-fetching for the Debian-specific module and make use of that from within the code and the Gtk frontend.

As you can see this list is rather short. This can mainly be attributed to a few university assignments, and instead of providing a long list of tasks which I probably won't be able to finish I rather keep the list short and hopefully get things not on this list done too.

Syndicated 2009-05-27 23:52:00 (Updated 2009-11-09 20:21:28) from sp

update manager to become more modular

In my last post I wrote about how I got accepted for GSoC09 and am going to work on update manager. Now I couldn't wait for the actual GSoC09 coding period to start and created my own update manager branch right away and started hacking.

So far I have only written a few lines of code, but my mentor Michael Vogt and me came to the conclusion that whilst working on the internals of update manager it might be a good idea to make the whole program more modular.
Right now all the different functions of update manager (being the UI/frontend and the package manager interface/backend) are mixed up in various files, which makes not only reading the code harder, but also extending update manager more difficult. This was reason enough for me to have a look into making update manager more modular in its design and some of my efforts can already be seen in my update manager branch.

If you have any comments on the proposed backend interface or see major problems with it, please let me know, I would really appreciate some input on that. Also, the UI and the distribution-specific code interfaces are next on my list, before beginning to actually move existing code around. I hope to be able to finish that work before the GSoC hacking period starts, so I can concentrate entirely on my task of making update-manager distribution independent.

Syndicated 2009-04-24 17:41:00 (Updated 2009-11-09 20:21:28) from sp

Summer Of Code 2009: Working for Debian

Yesterday Google announced the students and projects that have been accepted for Google Summer Of Code 2009 and guess what: my project was accepted. This means I will be working full-time on FOSS this summer.

So I guess it's about time to introduce my project to you: Distribution-independent update manager, mentored by Michael Vogt (mvo).

Okay, I believe some of you might wonder what this project is all about, as update-manager is in the Debian package archive already. There is a problem with update-manager though. As you see in the package's version number (it contains ".debian") update-manager has been adapted for use in Debian. Also, Debian contains update-manager 0.68 right now, whilst upstream (Ubuntu in this case) has released 0.111.6 (actually there were quite a few upstream versions meanwhile). The reason Debian is nowhere near being up-to-date with upstream is that right now a lot of effort has to be put into porting update-manager to Debian every time a new upstream release is made, because certain Ubuntu-specific functionality breaks update-manager in more or less severe ways on Debian.

This leads me directly to what my project is about: making update-manager (Ubuntu-) distribution-independent, but not package manager independent.
There are 6 main goals for this project, which I will be working on in the order below.

• Analyzing the code and identifying Ubuntu-specific parts.


• Creating a distribution-plugin interface and moving the Ubuntu-specific parts into a distribution-plugin, creating a core package that is distribution-independent.


• Creating a special notification for important/security related updates and providing the code that handles updates from security.debian.org as such.


• Creating a backend-plugin interface, moving the synaptics backend into a backend-plugin and optionally create a python-apt based plugin.


• UI redesign, providing a simpler interface to average joe, whilst allowing more experienced users to optionally display more information.


• Automatic downloading & installation of updates. This is still up to discussion, as automatic downloading is already provided by software-properties (-gtk and -kde) and automatic installation can be handled by unattended-upgrades. Both packages are part of Debian already.

Please note that this list should not be considered final and may be extended or modified over time. It exists to give you an overview of what exactly my project is about and how I am planning on carrying out the tasks.

Finally I wanted to let you know that I will keep you posted on the progress I am making, via this blog. Alternatively a blog aggregator for Debian's GSoC students has been set up over at http://soc.alioth.debian.org/feeds/blogs/, where you can not only find my posts, but those of all of Debian's students.

Syndicated 2009-04-21 07:10:00 (Updated 2009-11-09 20:21:28) from sp

Python everywhere: computer games

This is the second article in my series Python everywhere and covers the use of Python for in computer games. The first article of this series covered the use of Python for the conficker worm scanner tool and can be found here.
Games written in Python

As PyGame provides a nice library for writing games purely in Python it is becoming more common to use Python for this task too. The book "Beginning Game Development with Python and Pygame" is linked directly from the PyGame homepage, and thus is probably a good resource if you want to start writing games in Python.

However, I do not want to go into detail on how this library works, but rather provide you with a few examples of games written in Python. To provide you with a few examples I had a look at the PyWeek homepage. PyWeek is a Python Game Programming Challenge which invites everyone to participate, so the winners of this contest are of high-quality, and I'm showing you the latest two winners.

There are always two winners of PyWeek in for indivduals who have created games and teams. The latest winners are "Team Rambo" in the individual effort category and "Midnight Sun" with their two-man team.

PyWeek: Team Rambo's Stringrolled (individual)

Stringrolled makes use of the pygame library I mentioned earlier and is a platform game. In a mere 2377 lines of code, including comments and blank lines, Team Rambo created an impressive game, coming with a story, easy-to-learn controls and nice 2D-graphics, screenshot below.

PyWeek: Midnight Sun's Kite Story

Kite Story is yet another interesting game, with game mechanics I have not seen ever before. You are controlling a kite with your mouse and are trying to catch objects, such as bees and birds, with the kite's rope. So what you basically do you draw a loop around an object with your mouse and that way catch it. Every third cought object you advance to the next level, but keep in mind not to collide with the objects, because you will lose them and in turn be doing the previous level again, screenshot below. It should be noted that this game does not make use of PyGame at all, but rather relies on pyglet, and is 1997 lines of code in length, again counting blank lines and comments too.

Games using Python

You have seen now that it is possible to write a game completely in Python, but there's another use-case of Python in games: scripting.
Some (proprietary) games, such as Civilization IV, offer Python support in their editors and SDKs. This quote from the article at 2kgames.com should give you a good idea of what can be done using Python in Civilization IV:

The next level offers Python and XML support, letting modders with more experience manipulate the game world and everything in it. XML (eXtensible Markup Language) files can be edited in standard text editors or in special XML file editors that have ease-of-use features like a grid view. Editing these files will allow players to tweak simple game rules and change or add content. For instance, they can add new unit or building types, change the cost of wonders, or add new civilizations. Players can also change the sounds played at certain times or edit the play list for your soundtrack. NOTE: You can have custom soundtracks simply by adding music to the custom folder. You only need to edit the XML in order to assign certain pieces to specific eras or remove certain pieces.

The Python scripting language is fully integrated throughout the game and offers experienced modders a chance to really strut their stuff! People with some programming skills will be able to do things to alter the game in interesting and extraordinary ways. For instance, all of the game interface screens are exposed to Python, so modders will be able to change the information that's displayed, as well as how it's positioned on the screen. We also use Python to create and generate all of the random map scripts that are included in the game. So, players will now have the ability to add scripted events to the game like automatically generating units when a tile is reached, having specific situations trigger automatic war, or get this, bringing back Civil Wars caused by unrest, Civ II style!

EVE Online is another game making use of Python, as an article over at eveonline.com points out.

Python everywhere - also in compuater games

Even though I am sure you can come up with a lot more examples of Python being used in computer games I think I have proven my point. Python is being used not only to create computer games, but sometimes also to provide developers with a way of extending games. To me personally it feels as if adoption of Python for this very task is increasing too, and I expect Python to be used even more by the game development community in the future.

You can expect the third part of this series to be released in about a week, so please check back regularly if you like the series.

Syndicated 2009-04-02 17:35:00 (Updated 2009-11-09 20:21:28) from sp