Older blog entries for jtauber (starting at number 206)

Storing HTTP_X_FORWARDED_FOR in Django

I occasionally get a

ProgrammingError: value too long for type character(15)

when people post to my blog. There aren't any fields in my model declared to have a max_length of 15 so I was always a little confused and in almost all cases, it was a spammer anyway so never took the time to investigate further.

But then someone just emailed me and told me they were getting a 500 when posting a comment to my blog. So I decided to investigate and that's where it started getting interesting...

Doing

./manage.py sql leonardo

revealed no sign of a field of length 15 either. So I went into the DB (in my case PostgreSQL)'s shell.

A quick

\d leonardo_comment

revealed

author_ipaddress | character(15) 

Like many blogs, I capture the IP address of the commenter so I can block spam.

In my model I have:

author_ipaddress = models.IPAddressField(null=True)

Which Django's ORM translates to:

"author_ipaddress" inet NULL

which PostgreSQL is obviously storing as a character(15).

Why would an IP Address be more than 15 characters, though?

Well, I went back to the error log and noticed this:

'HTTP_X_FORWARDED_FOR': '192.168.0.127, 12.34.56.78',

(note: I changed the second address to protect the original poster)

You see, because the Apache instance running django is behind another webserver (on the same machine), I can't rely on REMOTE_ADDR because it's always 127.0.0.1. So I log HTTP_X_FORWARDED_FOR.

What I didn't realise until now is that HTTP_X_FORWARDED_FOR can be a list.

I guess the best solution is to just change the field to a CharField.

Other Djangonauts who are logging HTTP_X_FORWARDED_FOR might want to heed this warning: don't use IPAddressField.

Syndicated 2008-11-11 04:36:52 (Updated 2008-11-11 04:46:36) from James Tauber's Blog

Chrome Overtakes IE

I was just looking at the analytics for http://pinaxproject.com/ and noticed the following browser stats for the last month:

Firefox                       68.79%
Safari                        13.84%
Chrome                        5.88%
Internet Explorer             4.31%
Opera                         3.26%

What stood out to me is that more people have accessed the website using Chrome than IE!

Syndicated 2008-11-10 18:53:54 (Updated 2008-11-10 18:55:13) from James Tauber's Blog

You Need More Than Equipment to be a Cinematographer

With its full frame 1080p video, I've seen a bunch of articles like this one that claim "The Canon 5D Mk II Will Turn Us All Into Professional Cinematographers".

Yes, the 5D Mk II, with a full-frame sensor and EOS lens system is potentially a game changer, but to suggest that we'll all be able to produce results like professional cinematographers is stretching it.

While a good understanding of photography helps with cinematography, it's necessary but not sufficient knowledge.

  • composition is important but you also need to know how to move the camera during a shot
  • you also need to understand concepts like "crossing the line"
  • with the narrow depth of field full-frame affords, you need to be able to pull focus while shooting—that's a whole new skill to learn (and you probably need to use a follow focus)
  • you can't rely on shutter speed or aperture to control exposure the same way you do with still photography—you have to learn to rely more on ND filters and lighting
  • colour correction becomes even more important as mismatches are more jarring

I hope that the Canon 5D Mk II will encourage people to learn more about the art and craft of cinematography. But the equipment is not the end of the story, it's just the beginning.

Syndicated 2008-11-10 14:11:50 (Updated 2008-11-10 14:13:50) from James Tauber's Blog

10 Nov 2008 (updated 10 Nov 2008 at 07:09 UTC) »

Groups, Tribes and Projects in Pinax

From fairly early on, Pinax had tribes and projects. The intention of the distinction was that tribes are a loose grouping of people with a common interest (see Seth Godin's great little book Tribes: We Need You to Lead Us) whereas projects were more focused around managing a group of people working on common tasks.

This distinction was reflected in the differences in implementation: projects are invite only, tribes are open to anyone. Projects have tasks, tribes don't. But there are a lot of similarities: both have threaded discussion, wikis and photo pools. There's also a lot of duplicated code.

It wasn't long before I realised that really projects and tribes we just two subclasses of the same class (or instances of the same metaclass—more on that in a moment). So one of the things we're working on for the next release of Pinax is to merge the two into a single model but then allow a site developer to create as many differently configured subclasses/instances of this model as they like.

We haven't worked out the details yet, but basically it would allow you to define a new subclass/instance of "group", pick the membership model and what apps and object types can be attached to it.

Quite separately, I've noticed that there is another kind of structure that tacitly exists between tribes on Cloud27. I started a Python tribe but others have started Ruby and PHP tribes. There are also geographic tribes and language-based tribes. So there is another sense in which a set of tribes could be labelled as "programming language" tribes, or "geographical" tribes or "natural language" tribes.

So we have:

  • the root group type (that will unify projects and tribes)
  • types of groups (e.g. projects, tribes, other combinations of features and membership patterns defined by site developers)
  • instances of one of these types (e.g. a particular tribe, like 'Python')
  • a category of instances (e.g. 'Programming Language Tribes')

To slightly complicate things, one could imagine an "Australian Pythonistas" tribe. Or various intersections of "movies", "food", etc with geography-based tribes (e.g. Italian Food, French Films). These sorts of semantic relationships seems quite orthogonal to defining a type of group that has a wiki but not threaded discussions or is invitation-only versus free-for-all.

I'm trying to wrap my head around which to view as instances versus subclasses. As a longtime data modeler, I thought I had a grasp on this stuff but my brain is hurting :-)

Any suggestions for a metamodel?

Syndicated 2008-11-09 22:11:44 (Updated 2008-11-10 01:46:22) from James Tauber's Blog

From Focus and Directrix to Bézier Curve Parameters

For reasons that will become clear in a couple of posts, I wanted to be able to calculate quadratic Bézier curve parameters from a focus and horizontal directrix.

A focus and directrix are enough to define a parabola (in fact a parabola is the locus of points equidistance from a point, the focus, and a line, the directrix).

A quadratic Bézier curve is a section of a parabola and is defined by three points, according to the formula:

B(t) = (1-t)²P₀ + 2t(1-t)P₁ + t²P₂, t ∈ [0, 1]

Here's how I came to my result...

Given the directrix is horizontal,

  • P₁ must have the same x-coordinate as focus
  • P₁ must have an x-coordinate midway between P₀ and P₂

and by definition:

  • the parabola's low point is midway between the focus and directrix

Now even though somewhat arbitrary and assuming the directrix is below the focus,

  • P₀ and P₂ can have a y-coordinate of 0

So,

  • the parabola's low point's y-coordinate is midway between P₀y and P₁y
  • ⇒ the y-coordinate midway between focus and directrix is midway between P₀y and P₁y
  • ⇒ the y-coordinate midway between focus and directrix is half P₁y

Therefore,

  • P₀ = (Fx - α, 0)
  • P₁ = (Fx, Fy + Dy)
  • P₂ = (Fx + α, 0)

for some α where (Fx, Fy) is focus and Dy is y-coordinate of directrix.

By definition,

  • P₀ must be equidistant between focus and directrix
  • P₂ must be equidistant between focus and directrix

But, given P₀ and P₂ have a y-coordinate of 0 and directrix is horizontal,

  • distance between P₀ and focus = Dy
  • distance between P₂ and focus = Dy

Therefore,

  • (P₀x - Fx)² + (P₀y - Fy)² = Dy²
  • ⇒ ((Fx-α) - Fx)² + (0 - Fy)² = Dy²
  • ⇒ (-α)² + (-Fy)² = Dy²
  • ⇒ α² + Fy² = Dy²
  • ⇒ α = √(Dy²-Fy²)

And so, assuming the directrix is horizontal and below the focus, the following Bézier curve parameters can also be used to create the parabola:

  • α = √(Dy²-Fy²)
  • P₀ = (Fx - α, 0)
  • P₁ = (Fx, Fy + Dy)
  • P₂ = (Fx + α, 0)

Syndicated 2008-11-08 15:55:26 (Updated 2008-11-08 15:55:27) from James Tauber's Blog

Song Project: LH Piano Riff

Having introduced the right hand of the main piano riff, let's introduce the left hand.

Firstly, here's what it sounds like by itself:


download if embed doesn't work

And together with the right hand:


download if embed doesn't work

And here's what it looks like in the score:

score

Now let's analyze it a little. Note that this is post hoc analysis, I didn't go through this thinking when I wrote it (at least not consciously)—it was improvised at the piano—but I find it interesting to go back and see why things worked or generated the particular effect they have.

One thing that immediately stands out is that the 3+3+2 rhythmic grouping found in the right hand is also found here (with one exception we'll come to in a minute). Also, if you look at what note is playing at the start of each of the 3+3+2 grouping: A A C♯ | D D D | B♭ B♭ - | F C E it is always the root of the current chord on the first two beats and either the root, the third or nothing on the final beat.

In the first bar, there is an additional passing note, B (notice it's natural despite the key, because the chord is A) and the C♯ is the leading note into the D chord in the next bar. The slur emphasizes this role.

In the second bar, the E is just a neighbour note between two repetitions of the root D. The final, non-accented A is in the triad of the chord but it's also the leading note of the following B♭ chord and again leading note is slurred.

In the fourth bar, the first E and D are just passing notes taking us from the F root to the C root. The final E is the third of the chord, the 5th of the A chord we return to if we repeat but it's also the leading note of the tonic F (which we'll take advantage of later). It's a great example of a note playing multiple functions both in terms of the current chord and what may follow.

The third bar (which I deliberately left until last to discuss) is a little unusual. If you imagine the second B♭ an octave higher, it's a little easier to see what's going on. In that case, the A and G are just passing notes down to the F in the next bar. But what is a little unusual is firstly that the A and G are not following the 3+3+2 pattern. It is almost as if the pattern has switched to 3+2+3 this bar. Secondly the A is quite dissonant, especially given it's only a semitone away from a note being played in the right hand. This rhythmic change coupled with the dissonance builds a nice tension that is then resolved with what I called a "triumphant" chord in a comment on the previous post.

You may notice a bit of chorusing in the piano sound of the recordings so far. Actually, there's some compression, chorusing and EQ on them, all preempting the mixing that is to follow once we add more instruments. I'll talk about each of these once we've added a few more tracks.

All material for this project is made available under a Creative Commons BY-NC-SA license so you are free to redistribute and remix with attribution but under the same license and not commercially.

Syndicated 2008-11-08 13:46:37 (Updated 2008-11-08 14:12:49) from James Tauber's Blog

Voronoi Diagrams

Back in Creating Gradients Programmatically in Python I presented some code for writing out PNGs according to some rgb-function of x and y.

The relevant write_png(filename, width, height, rgb_func) is here:

http://jtauber.com/2008/11/png.py

This enables one to quickly generate PNGs based on all sorts of functions of position.

For example, here's a Voronoi diagram:

Take any metric space and pick a discrete number of points in that space we'll designate "control points" (the black dots in the above example). Now colour each other point in the space based on which control point is closest. In other words, two points are the same colour if and only if they have the same "closest control point".

Here's how to generate such a diagram using write_png...

First of all, here's a function that given an (x, y) coordinate and a list of control points, returns a pair of:

  • which control point is the closest to (x, y)
  • what the distance was

def closest(x, y, control_points):
    closest = None
    distance = None
    for i, pt in enumerate(control_points):
        px, py = pt
        d = ((px - x) ** 2 + (py - y) ** 2)
        if d == 0:
            return i, 0
        if d < distance or not distance:
            closest = i
            distance = d
    return closest, distance

Now we can use this and write_png to generate a PNG Voronoi diagram for a given set of control points:

def voronoi(filename, size, control_points):
    def f(x, y):
        c, d = closest(x, y, control_points)
        # draw points in black
        if d < 5:
            return 0, 0, 0
        px, py = control_points[c]
        # just one way to generate a colour
        m = px * 255 / size
        n = py * 255 / size
        return m, 255 - ((m + n) / 2), n
    write_png(filename, size, size, f)

Of course, this is just a brute-force way of establishing the Voronoi diagram, but for just generating examples a few hundred points by a few hundred points, it's Good Enough.

Note the choice of colouring, based on the coordinates of the control point is just one of many possibilities. You could also just colour based on control_point number (i.e. c) The current approach has one disadvantage that two control points very close to one another can be given almost indistinguishable colours.

The example diagram was just randomly generated with the following code:

import random
from voronoi import voronoi
space_size = 200
num_control = 8
control_points = []
for i in range(num_control):
    x = random.randrange(space_size)
    y = random.randrange(space_size)
    control_points.append((x, y))
voronoi("voronoi.png", space_size, control_points)

You can read more about Voronoi diagrams on Wikipedia.

Syndicated 2008-11-07 20:33:58 (Updated 2008-11-07 20:57:34) from James Tauber's Blog

Song Project: RH Piano Riff

Most of my pop song ideas begin with either a chord progression voiced a particular way on piano or some bass line. The song we'll be talking about here falls in to the first category.

I remember when I first started composing in high school, I did a lot of songs that were just permutations of I, IV, V and vi chords (so in C, that would be C, F, G and Am).

I remember one instrumental I wrote in Year 10 (called "Mystical Movements in Green")—that my drama class choreographed a dance to—used the chord progression vi IV I V and in particular was voiced with the vi and I in the second inversion. I always liked the way it sounded.

A couple of weeks ago, I was improvising on my digital piano and took a liking to the following variation:

    III . vi . IV . I V

with the vi and I again in the second inversion. I was playing in F at the time with a driving 3+3+2 rhythm in the right hand, so the resultant riff was:

score

which sounds something like this:


download if embed doesn't work

This will form the basis for the song.

All material for this project is made available under a Creative Commons BY-NC-SA license so you are free to redistribute and remix with attribution but under the same license and not commercially.

Syndicated 2008-11-06 08:30:24 (Updated 2008-11-06 08:59:24) from James Tauber

Song Project

One series I thought would be fun to kick off this month is to talk about music composition and record producing and engineering by working through a song. I've chosen a song I just started working on last month and the idea is I'll go through the process from initial idea to final produced song over a series of blog entries.

All material for this project is made available under a Creative Commons BY-NC-SA license so you are free to redistribute and remix with attribution but under the same license and not commercially.

I'll get started with the music in a separate post right away.

Syndicated 2008-11-06 08:04:55 (Updated 2008-11-06 08:19:49) from James Tauber

Atom, Google Reader and Duplicates on Planets

For a while I've wondered why posts syndicated across multiple planets don't get picked up by Google Reader as duplicates (and automatically marked read when I've read it the first time around).

I wasn't sure whether the problem was:

  • the source feeds
  • the planet feeds
  • Google Reader itself

so I decided to investigate further with my own feed as the source and the three planets my site is syndicated to (that I know of).

Let's take my post Cleese and Disk Images.

My feed gives both an id and a link for both the feed itself and each individual entry. That makes it possible, at least, for planets and readers to do the Right Thing. So I don't think the problem is my feed.

On the Official Planet Python:

  • there is an RSS 1.0 feed
  • the rdf:about is not the same as the id in my feed but is the link URL (subtlety different but the planet is doing the wrong thing, IMHO)
  • there is no authorship or source information

On both the Unofficial Planet Python and on Sam Ruby's Planet Intertwingly:

  • there is an Atom feed
  • the entry Cleese and Disk Images has the same id as in my feed and a distinct link.
  • the source element gives appropriate information about my blog and the original feed
  • the author (which I only give on the source feed not each entry) is not inherited on to the entry in the planet feed, only included in the source

Note that the handling of the author by the latter two feeds is correct per the Atom RFC, although I have noticed that Safari's feed reader gets this wrong and, despite the author in the source element, uses the inherited author from the planet feed itself.

But, in short, the Atom-feed-based Planets do the Right Thing, although IMHO the RSS-1.0-based Official Planet Python does not. That may not be the Planet's fault. The RSS 1.0 Spec (or any RSS for that matter) may not make the distinction between id and link.

So given that my feed and two of the planet feeds do the right thing, I guess that places the blame with Google Reader.

Why does Google Reader not honour the entry id and automatically mark duplicates as already read when you've read it the first time. That's my pony request for Google Reader.

And by the way, the same thing applies to feeds themselves, not just entries. Feedburner, for example, does the right thing and passes through the id of a source Atom feed into its own Atom feed version. However, if you subscribe to both the source and Feedburner version of of a feed, Google Reader doesn't not identify them as the same feed. Of course, if either are RSS, I'd assume all bets are off.

So, in summary, Atom supports doing the Right Thing. The Atom-based Planets do the Right Thing. Google Reader doesn't take advantage of this.

Syndicated 2008-11-06 06:04:34 (Updated 2008-11-06 06:04:35) from James Tauber

197 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!