Older blog entries for malcolm (starting at number 99)

Django Tips: Variable Choice Lists

Been a while since I added to this series. I've come across a couple of repeated questions lately, so it's time to give back to the knowledge pool again.

This time: using iterators to customise the options presented via the choices attribute on a model field.

Background

Before launching into the solution, let's consider the problem we are trying to solve. If you have a model field that is intended to hold only one of a number of limited values, Django provides the choice attribute. You can use it like so:

  class Document(models.Model):
    CHOICES = [(0, 'private'), (1, 'public')]
    ...
    status = models.IntegerField(choices=CHOICES)

When you use this in a form, only the two choices private and public will be presented and the database will store either 0 or 1, depending on the choice you made.

Aside: People often forget that when you retrieve such a model from the database, although the status field contains 0 or 1, you can get back the string version of the choice using the get_status_display() method of the model. Replace status with the name of the field for your own use. This is explained under get_FOO_display() in the Django documentation.

When Isn't This Enough?

There are two cases where the previous example falls a bit short.

The first case is when the list of choices is being updated regularly via changes to the database, or in some other way. In this situation, choices isn't the right approach to the problem. You are really talking about a dynamic relation to another data set. So model it that way: use a ForeignKey field to a table containing the list of choices and the values to store.

The second case is more subtle. Suppose you have a document presentation system. Documents on the production site are either public or private (more or less, this is the above example). However, the same code runs on a staging system as well, where documents are initially uploaded, reviewed and edited. On this system, the choices can include something along the lines of "ready for review" and "needs editing". This is a slight variation on similar systems I've implemented for a couple of clients recently, so it's not too unrealistic (although I've simplified a bunch of details).

In the second scenario, above, the list of choices is essentially static. So we are justified in using the choices attribute. However, the intiial values vary depending upon the system type — which we might reasonably control using a settings variable.

Now, it's generally a good idea to avoid referring to settings.* in the definition of fields and methods in Django. This way you can safely import the code without needing to have configured the settings module, which usually feels like neater code organisation (import everything, then configure, if you're using manual configuration). To my eye, using settings.FOO in declarations also looks a litle awkward (intuitively, it feels like a leaky abstraction, since we're delving into the depths of a module at the top-level).

For whatever reasons, whether you agree with me or not, I'm going to avoid using settings in my field declaration. Instead, I'm going to use a little-known (and not usually required) feature of the choices attribute: you can pass it a Python iterator instead of a sequence. So I can rewrite my example as follows:

  from django.db import models
from django.conf import settings

def status_choices():
    choice_list = [
            ('private', 'private'),
            ('public', 'public')]
    if hasattr(settings, 'STAGING') and settings.STAGING:
        choice_list.extend([
                ('review', 'ready for review'),
                ('edit', 'needs editing')])
    for choice in choice_list:
        yield choice

class Document(models.Model):
    ...
    status = models.CharField(maxlength=10, choices=status_choices())

You can see here that all the dependency on settings is inside the iterator function. So it isn't evaluated until Django needs to actually display the choices, which should be long after configuration has taken place. This relies partly on the fact that the Python compiler knows this is a generator function (because of the keyword yield) and consequently executes none of the code until the first value is retreived from the generator.

I would also draw attention to a couple of other implementation decisions I made in this code:

  1. The extra options only appear if the (optional) settings.STAGING setting is set to True. Note that this "fails safe", in the sense that if you forget to include the STAGING setting, it won't inadvertently expose the extra options and documents to the wider public. I made the setting optional, because I'm just a nice guy, and so had to first check that it existed using hasattr() before I tried to access it. You may or may not wish to be that flexible.
  2. I switched from storing integers, as in my first example, to storing short, readable strings in the database. I prefer this method, because it avoids the problems associated with having magic numbers in the database column. If you see the number '2' in the database, what does it mean? If you see the string 'review', things are a little more mnemonic. I've noticed a tendency for people to use integer values with the choices attribute; perhaps they are forgetting it works on pretty much any field and CharField fields are often a good choice?

Cavaet

If you are very familiar with Django, or tried to experiment a little with this example, you'll realise I have not told the entire truth here. The whole argument about using an iterator to avoid accessing the settings module too early is pointless. You cannot currently import django.db.models without configuring the settings module, so there's a chicken-and-egg problem there. However, I consider that to be a (very small) bug in Django and it's something I want to fix in the near future. You should be able to import modules without having done any configuration.

You probably won't need to use this technique very often at all. Every now and again, though, you will run across a configuration where being able to construct an intelligent choices list will help the code layout flow more smoothly.

Syndicated 2007-03-26 14:22:51 from Malcolm Tredinnick

After a long hiatus, I've started blogging again. Because I want to try out a few different things and not all of it is Open Source related, I've moved to my own site. Henceforth, all the real non-events will be over at the pointy stick.

GNOME Summit 2004

The summit was fun. Possibly surprisingly, this was my first dedicated GNOME conference and it was nice to finally attach faces to some people that I have been communicating with via email and IRC for three or four years (in some cases). It was also a productive three days for me in that I had a few conversations with people to discuss some things that would have taken dozens of emails to work through normally.

As noted already by so many other people, the Stata Center was quite a strange building, with some very odd ergonomic quirks. But it was a good venue for something like the GNOME summit in many ways.

So, good talks, good company, good fun. It was worth the trip over to the US to attend, I think. :)

Ankh: When I was starting to mess around with XML to the point that I wanted to read specs and write programs with it (early 2000, I guess), XLink seemed like one of the more interesting specs being produced. I could imagine a browser of some kind that would allow me to create annotated versions of documents that were otherwise read-only to me. I would just have to create a file of arcs between two external targets -- a point in the source document and the annotation document. I could write the annotation myself or just use this feature to connect up two different pages in a way that was useful to me.

Standard support for this sort of thing would be great. Instead of browsing document A that describes the connections between document B and other places, I could just go through B and have the links available as I go -- much more useful to somebody who works like I do, synthesising a lot between different information sources.

So much to write at some point; probably will never get it all down.

Just quickly, though: if there is anybody in the Sydney, Australia area who wants to work with Python on Linux doing "cool stuff", we are hiring. Get in quick to avoid the rush.

I have put up a brief write-up of my linux.conf.au week. It will only stay live for about a month, since it is not really GNOME related and that machine is a bit abused these days, so read fast.

(There may be some small issues with the photo captions on IE 5.x/Win, but the document is not important enough for me to bother tracking down some way to test that.)

17 Jan 2004 (updated 17 Jan 2004 at 12:54 UTC) »
linux.conf.au

And so another year's conference is over and we have to wait nearly 15 months until the next one.

Three days of interesting talks have gone past in a whirl. Every talk I went to was worth the time; speakers were well prepared, the equipment generally worked well and the rooms were comfortable enough, even when full. Havoc gave a fantastic keynote this morning about putting Linux on desktops everywhere. This is not to say Bdale and Maddogs' keynotes were not also interesting, but they said similar things to other talks of theirs that I have heard. I had not seen Havoc in full-flight advocacy mode before and it was an impressive performance.

I had a lot of conversations over the last three days of the conference with people who are interested in developing GNOME applications. One guy admitted to staying up until 0300 playing around with the Java bindings and discovering just how complete GNOME has become as a development platform. This kind of feedback is extremely rewarding and reflects well on everybody who is involved in GNOME development. Unfortunately, many of the conversations I had also highlighted the embarassing gaps in developer education materials we have, but that is kind of a known problem already. We just really, really need to fix it.

Extremely tired now, so tomorrow will be Recovery Day before returning to work on Monday.

linux.conf.au

Day two of the pre-conference program (yesterday) flew past for me. I gave a couple of talks at the Python mini-conference. The first one went fairly well (I have given it a couple of times now -- talking about using Python in business situations), the second one slightly less well, but that was entirely my fault; I could not think of enough practical Python tricks to talk about and it kind of meandered towards the end. Still, it seemed to be fairly well received and nobody threw fruit or anything. Wound up with a slightly impromptu talk at the GNOME mini-conference and then being roped into the question-and-answer session at the end.

Completed my speaking commitments today with a GNOME tutorial which attracted a reasonable audience who seemed to be interested enough to ask questions throughout. This is the kind of tutorial I think GNOME contributors should be trying to give at every single conference, so it was an interesting experience to see if I could talk about GNOME at a sufficiently high level to be interesting to "third-party" developers (Gstreamer guys: I pushed somebody in your direction who is interested in writing developer-level documentation. Don't frighten him away!)

I had a bit more time to talk to people today (and last night at the speakers' dinner). Much of my enjoyment at conferences like l.c.a comes from just catching with people I see once every year or two, so I am looking forward to just relaxing over the next three days, listening to talks and participating in the group discussions. I am completely blown away by the quality of organisation at this conference each year and the organisers this year have taken things to the next level again. Just little things like having areas set up under large tents so that we can sit out of the sun and talk makes the whole experience very enjoyable.

linux.conf.au

I am at linux.conf.au this week, along with a few hundred other people (glynn, jdub, hypatia, jamesh, mrd, havoc, ... the list of familiar faces is endless).

Gave my first (of four) talks (paper here) at the Linux and OSS in Government mini-conference this afternoon. I managed to avoid embarassing myself and some people stayed awake long enough to ask questions. One cannot ask for more.

The government mini-conference is really quite impressive. All day, it has been one speaker after another giving case study style presentations about the succesful use of Open Source software and ideas at both the state and federal level. Mine was the only talk all day that was not about a existing government installation or project (I was doing an advocacy talk). Normally, I suspect this kind of stuff would drive me up the wall, but the presentations have been very interesting and the between-talks talk (during coffee breaks) very motivating and intelligent.

18 Nov 2003 (updated 18 Nov 2003 at 01:23 UTC) »

QA in "other projects"

This posting by a gentleman (called Joe) inside Microsoft's testing division is interesting. It is a fairly well written blog entry about the testing and bug-fixing process inside Microsoft. A few things jump out at me from this post:

  • Firstly, more comments at the end than I would have expected are of the "Gee whiz!" variety. Are there really that many people who are taken aback by the process of fixing bugs and the fact that even when a problem is understood the fix does not come at zero cost? This is just as applicable in the open source world as in corporate environments, so maybe it is something we need to get the word out about.

  • Secondly, his comments about internationalisation are interesting, at least when you compare it to open source desktop projects. The application he is talking about is translated into a dozen languages. That is comparable to a minor GNOME application (for example) and far below the norm for a major one. Some of the problems he talks about such as checking the phrasing and translation accuracy are common to all projects. Other things, like making sure it works when the locale is set at runtime and making sure text is not clipped and the UTF-8 encoding is accurate are things that many open source projects handle as a matter of course: locale setting Just Works(tm) at the C library level and programmers know how to use it. Widgets and text handling libraries like Pango are designed to allow easy and correct text layout (you can still get into trouble if you use fixed width and height dialog boxes and they are small enough, but people know not to do that). The time required for translation is a bit less in open source projects due to the massively distributed nature of the various translation projects (gnome-i18n, kde-i18n, GNU Translation Project). In short, it appears that Microsoft have some problems that open source projects have already seen it is necessary to solve in this regard.

  • Finally, the rest of the entry: evaluating the impact of a bug (this was a reasonable low probability bug) versus the cost of fixing versus the proximity to the release date are issues that all projects have to solve. The views expressed in that article seem to be pragmatic and match what a project like GNOME does.

So, all in all, an interesting piece: both for confirming some of my suspicions (and probably prejudices) and for the information it contains.

I have some other things to say about Joe's writings, but I shall save them for later. For now, his follow-up posting is more food for thought and analysis.

"Professionals"(sic)

I saw a piece last night about performance enhancing drugs in US Major League Baseball. Nauseating. It pays to remember that professional only refers to the fact that the players and officials are paid, not to their level of behaviour.

Based on the figures mentioned in that report (five to seven percent of players use steroids and the like), if you are in the USA and go to a MLB game next season, on average, at least one of the players taking to the field in the first innings will be a drug cheat. It should be a fun game to play with your kid when they are there doing a bit of hero worship -- spot the slimeball.

Baseball administrators have adopted the attitude that it is better to be seen doing something than to actually do something. So they have carefully removed the possibility of punishments acting as a disincentive, random testing being used in a fashion likely to actually catch or discourage the cheats, or of testing being done by a credible organisation with experience and resources necessary to keep up with the ever-changing field.

90 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!