Older blog entries for nbm (starting at number 118)

At WordCamp South Africa 2008, held in Cape Town yesterday, we were given a brief overview of how Wordpress.com is set up to scale.

Matt Mullenweg set the scene with some idea of just how huge Wordpress.com is.  I may mess up a few numbers mentioned, but there've been something like 6.5 billion page views on Wordpress.com since the beginning of the year, there are 3.8 million Wordpress.com hosted blogs (only Blogger is bigger), and there are 1.4 billion words in posts created on Wordpress.com.

Warwick Poole then gave us some more in-depth numbers, although pointing out that Wordpress.com was bigger than AdultFriendFinder was a pretty good and well-understood indication from the audience's reaction.  In May 2008, Wordpress.com was served 693 million page views, but this rose to 812 million page views in July.  Over 1TB of media was uploaded in May, 1.3TB in July.  In May, 417TB of traffic left the Wordpress.com data centres.  These numbers are available in the "July wrap-up" post on the Wordpress.com web log.

Apparently, across the approximately 710 servers, 10 000 web requests and 10 000 databases requests are handled per second (I wasn't intelligent to write down whether this was the average).  110 requests per second are done to Amazon's S3 storage service, while 3TB of media is cached on their own media caches.  They output 1.5TB/s (I wrote TB, so it probably is TB and not Tb.  I'm guessing this is peak). They experience approximately 5 server failures a week.

How is it put together?  They use Round Robin DNS which determines the data centre (from testing, it seems there round robin six IPs - two IPs for each of three data centres).  There it hits a load balancer using some combination of nginx, wackamole, and spread.  They use Varnish for serving at least media, and currently use Litespeed web servers.  They also use MySQL and memcached.

They use (and developed) the batcache Wordpress plugin to serve content from memcached - according to the documentation, batcache only potentially servers stale content to first-time visitors - visitors who have interacted with the web log receive up to date content.

When new media is uploaded, its existence and initial location is stored in a table.  As necessary, the other data centres will create their own local copies of that media, and update that table.  The backup media stores in the data centres are write-only - apparently nothing is ever deleted from them.

That's about all I wrote down, but there's quite a bit of information about how Wordpress.com is set up and the sort of load/traffic it has on the Wordpress.com blog and on the blogs of various employees (such as this post on nginx replacing Pound, this one on Pound, and another on varnish) giving some useful information which will probably inform some technology choices we might make at SynthaSite.

Good version control system usage prevents many disasters, but that doesn't necessarily mean you won't make your own mistakes.  Today, I mistakenly included a file in a commit that I didn't want to commit yet.  I learned two new tricks while spending a few minutes puzzling the best way to get back to where I was before with that file.

First, make a mistake:

$ svn commit -m "..."
Sending        dev.cfg
Sending        gibe/plugin.py
Transmitting file data ..
Committed revision 114.

svn merge is the tool to use for this:

merge: Apply the differences between two sources to a working copy path.
usage: 1. merge sourceURL1[@N] sourceURL2[@M] [WCPATH]
       2. merge sourceWCPATH1@N sourceWCPATH2@M [WCPATH]
       3. merge [-c M | -r N:M] SOURCE[@REV] [WCPATH]

Trick #1: use svn merge's 3rd usage pattern with the -c option with the negative of the revision you've committed, and (here comes the trick) use . (the current directory) as the source of the merge:

$ svn merge -c -114 .
U    gibe/plugin.py
U    dev.cfg

With that your working copy is now where the repository was before your commit.  Commit that to the repository, and the repository is back where it was before your commit.

Now your working copy is where it was before you made any changes - but you probably want those changes back.  Easy enough:

$ svn merge -c 114 .
U    gibe/plugin.py
U    dev.cfg

Now your working copy is back where it was before you did the mistaken commit.

Trick #2: Of course, if your mistake is like mine and you only messed up one file and everything else is as it should be, you can just do this on one file, by using svn merge's 2nd usage pattern:

$ svn merge dev.cfg@114 dev.cfg@113
U    dev.cfg

Commit that, and your repository is back to normal.  Then run:

$ svn merge dev.cfg@113 dev.cfg@114
U  dev.cfg

Now the file is back where it was before your botch.

Some of the services in the SynthaSite service layer use Python, WSGI, and the Pylons web application framework (with some TurboGears 2...). Particular functions require authentication, while others do not. We had a few simple working parameters:

• We only currently need a single user name and password for these particular services, since we are authenticating the connecting application, not a particular user.
• The authentication details must live in deploy configuration, not code.
  
• We would like to easily be able to see a list of all entry points into the application, and see whether they require authentication.
• If we do not specify otherwise, assume that our functions require authentication.
• To simplify testing and development, be able to easily turn off the authentication requirement in deploy configuration.

We already have a list of all entry points into these applications, since they use Routes. In the Pylons layout, these live in config/routing.py, and look like this:

from pylons import config
from routes import Mapper

def make_map():
    """Create, configure and return the routes Mapper"""
    map = Mapper(directory=config['pylons.paths']['controllers'],
                 always_scan=config['debug'])

    # The ErrorController route (handles 404/500 error pages); it should
    # likely stay at the top, ensuring it can always be resolved
    map.connect('error/:action/:id', controller='error')

    # CUSTOM ROUTES HERE

    map.connect('users', '/users', controller='users', action='index')
    map.connect('user', '/users/:user_id', controller='users', action='user')
    # ...

While one can use Routes with controllers and actions that depend on the URL, I prefer being explicit.  This creates a single list of all the accessible URLs and their accompanying controllers and actions.

If you attach additional keywords to the map.connect method, then they are added to the defaults attribute of the Route object created for each of those routes.  The RoutesMiddleware WSGI middleware places the route that matches the incoming request into the routes.route key in the environ dictionary that drives WSGI.  So, we can just add _auth = True to routes that require auth and _auth = False to those that don't, and create our own simple authentication middleware.  It would look something like this:

from paste.auth.basic import AuthBasicHandler

class LocalAuthenticationMiddleware(object):
    def __init__(self, app, config):
        realm = config.get('localauthentication.realm', None)
        username = config.get('localauthentication.username', None)
        password = config.get('localauthentication.password', None)
        if realm:
            def authfunc(environ, username, password,
                _wanted_username = username, _wanted_password = password):
                if username == _wanted_username:
                    if password == _wanted_password:
                        return True
            self.protected_app = AuthBasicHandler(app, realm, authfunc)
        else:
            self.protected_app = app
        self.config = config
        self.app = app

    def __call__(self, environ, start_response):
        route = environ.get('routes.route', None)
        if not route:
            return self.app(environ, start_response)

        if route.defaults.get('_auth', 'False') == 'False':
            return self.app(environ, start_response)

        return self.protected_app(environ, start_response)

We use Paste's AuthBasicHandler WSGI middleware to optionally wrap our application.  We keep a reference to our application around, in case we don't want to apply authentication.  When our middleware is called, we check whether we want the AuthBasicHandler-wrapped application, or the plain application, and call the one we want as per standard WSGI middleware.

Specifying _auth = True and _auth = False for every route is going to be painful.  Instead, we created a simple wrapper function around map.connect that we use instead, and it does the defaulting to requiring authentication for us (amongst other things):

    def connect(route_name, route_url, *args, **kw):
        if 'method' in kw:
            method = kw.pop('method')
            if 'conditions' in kw:
                kw['conditions']['method'] = method
            else:
                kw['conditions'] = dict(method=method)

        # Unless otherwise specified, require authentication
        kw['_auth'] = kw.get('_auth', True)

        return map.connect(route_name, route_url, *args, **kw)

Over the past few days, I've been putting the final touches on TechGeneral before letting anyone know about it.  The process from development to deployment has been surprisingly simple.

TechGeneral runs gibe, the web log server application I wrote for my personal web log.  Gibe is written in Python, using the TurboGears 1 mega-framework.

When deploying Python applications, using virtualenv (or something equivalent) is the best way to go.  Each virtual Python environment contains the particular versions of libraries necessary to run the applications that run in that environment.  TurboGears 1 is getting a bit old (although that's entirely relative), and needs some older versions of libraries.  No problem accomodating that with virtualenv.

Gibe itself, its plugins, and themes written for it (which are just plugins) are all Python packages, and are most easily installed using easy_install within the virtual Python environment.

This was my first real-life use of mod_wsgi, which manages a WSGI application's lifecycle.  I created a simple .wsgi file using the TurboGears example, and set up a line or two in my Apache config, and I had a fully managed Python process running as a specified user and using the virtual Python environment I'd set up.

At this point, moving from development to deployment was just a matter of creating Python packages, uploading them, installing them with easy_install, and using the Unix command "touch" on the .wsgi file to tell mod_wsgi to redeploy the application.

If a mistake happens, I just remove the new version of the offending package and touch the .wsgi file.

I created a simple development-side alias to create a new Python source distribution of the current Gibe plugin (or Gibe itself), and upload it to my server:

alias tgu="python setup.py sdist && \
    scp `ls -trc1 dist/* | tail -1` \
    private:/srv/techgeneral.org/"

On the server side, I have a simple function (paths removed for simplicity):

tginstall() { easy_install "$@" && \
    touch mod_wsgi/techgeneral.wsgi ; }

(I supposed if I got really bored, I could create one command on the development machine to push it up, install it, and reload the server.)

I'm Neil Blakey-Milner, a technology generalist based in Cape Town, South Africa.  Welcome to TechGeneral, my new (at least at time of writing) technology web log, where I talk about my wide-ranging interests in technology.

Likely common themes are:

• open source (or Free Software, if you prefer),
• scalability,
• security,
• the Python programming language,
• technology events I'm interested in and/or attend,
• becoming a better developer,
  
• finding/hiring developers and other technology workers, and
  
• motivating and getting the most out of them.

Since April 2003, I've maintained a mixed-bag web log, Cosmic Seriosity Balance.  From today, that will be where I'll talk about thing other than technology, such as:

• books, games, movies, and other entertainment,
• events I'm interested in or attend,
  
• travel inside and outside of Cape Town,
• experiences with people and organisations,
• random opinions on topics such as politics, and
• that blogging about blogging thing.

I've realised that the people who read my technology posts (predominantly outside of South Africa) probably don't care much about the other stuff.  The people who read my other stuff (predominantly inside South Africa) probably don't care all that much about the technology stuff either.

I hope this split helps improve the subjective signal to noise ratio for those who follow what I have to say.