New net_flow package
My adventures in trustiness continue: Raph Levien, the author of the net_flow C code & the admin of the advogato site, gave me the actual configured capacities of his site. Woot. The code seems to reproduce the actual certification levels of advogato with fairly good accuracy; I suspect my scrape was a bit out of date, which could account for the discrepancies. (I nail the robots.net certifications exactly, and that site is a bit less active, I think.)
Grab it here.
New development server & tools setup: part 1 of inf.
I finally took the plunge and ordered another server from JohnCompanies. I went with the lowest-cost Linux VPS, so I share a machine via virtualization, with 256 mb of RAM guaranteed (up to 1 gb burst), 40 gb of bandwidth a month, and 5 gb of disk space. I ordered it with Debian 3.1. This machine adds to my small collection of servers: I already have a FreeBSD server with them, and I also run a local development server for my lab (RH 7!), an e-mail & Web page server for my family and friends (Debian), and a home MythTV server (Debian). I'm hoping to swap up the servers so that the e-mail server is moved onto a JohnCompanies server; that way I won't have to worry about hardware or backups.
I've had the FreeBSD server with JohnCompanies for several years now, and I share the cost with several people. (The total is something like $40/mo., after a discount because of my open-source work.) JohnCompanies is really great; I get the sense that they know UNIX as well as I do, if not better. Unlike me, however, they have the focus to actually do a good job of network adminning ;). They're incredibly responsive and their servers Just Work. It's exactly what I need. (Although if anyone has any suggestions for similarly priced off-shore servers, I'd appreciate a tip. I'd rather our government be able to legally wiretap me without a warrant, you see ;).
This new server is going to be a svn, darcs, Trac, Web site, domain, and whatnot hosting system. I'm hoping to make it into a bit of a co-op, where friendly like-minded people can host projects; part of the project will be to produce infrastructure to run all this stuff nicely. Grig is already planning to host two domains there. If you're a Python OSS developer who doesn't bite and wants a root shell, Trac+ setup, and a Python-friendly installation, e-mail me to join up. (Note, this is not-for-profit; we're just trying to share out the costs of all our little projects. It may even end up being free, if I can scrape together sufficient google ads revenue over the next few months)
After JC set up the machine for me, I had a fun-filled evening of setting up software. Briefly, I:
- upgraded to Debian 'testing' from stable;
- installed SCGI for Apache 2.x;
- installed tailor;
- set up Trac with the WSGI patches.
I had such a good time doing this that I thought I'd give a little "tutorial-style" introduction to two aspects of the setup for which I found relatively little documentation on the 'net.
Using easy_install to manage your Python packages
This is so easy that it's almost silly to write a HOWTO, but ... I didn't realize how easy it was 'til I'd done it ;).
Motivation:
I'd like to give users on this machine the ability to use multiple versions of a Python package.
Solution:
Use easy_install to install everything. easy_install will build eggs for each version of each package and allow the import of specific, distinct versions through the 'require' function.
To install easy_install, I downloaded ez_setup.py and ran it:
python ez_setup.py
This downloaded and installed setuptools.
To set up multiple versions for multiple Python versions, you can do something like this:
# install for 2.4 python2.3 ez_setup.py mv /usr/bin/easy_install /usr/bin/easy_install2.3# install for 2.4 python2.4 ez_setup.py mv /usr/bin/easy_install /usr/bin/easy_install2.4
# default to 2.3 ln -fs /usr/bin/easy_install2.3 /usr/bin/easy_install
You can now use easy_install to install all of the packages that you don't want managed by apt-get (or your package manager of choice).
To install the latest version of a package via PyPi:
easy_install package_namee.g.
easy_install twill
will install twill 0.8.1.
To install a .tar.gz that may or may not know about setuptools, try:
wget http://darcs.idyll.org/~t/twill-latest.tar.gz easy_install twill-latest.tar.gz
This will install the latest development version of twill.
To install a specific egg:
wget http://issola.caltech.edu/~t/dist/CherryPy-2.1.0-py2.3.egg easy_install CherryPy-2.1.0-py2.3.egg
And, finally, you can use to install files directly from an untarred distribution or a repository:
darcs get --partial http://darcs.arstecnica.it/tailor easy_install tailor
easy_install will go grab tailor/setup.py, grok it, and install both the library code and the scripts.
easy_install: it's that easy ;). Note that I have yet to run into any problems with using it on local files; occasionally it fails to find PyPi packages, or does strange things while scanning random Web pages.
So what drawbacks are there to easy_install? I've only run into two problems: one is that automated tests may not work for packages installed as eggs, e.g. 'tailor test' doesn't work. The other is that 'help' apparently doesn't work, either. Neither are big problems for me, because I don't use 'help' much (emacs can browse zip/egg files just fine, and I prefer reading the source code) and I'm not developing on 'tailor'.
I'll describe using the Python API to setuptools to import only a specific version of packages in a bit; haven't found the online docs for that, otherwise I'd just link to it ;).
Subversion to Darcs Mirroring with Tailor
The back story:
Devoted readers may recall that I complained about having to maintain my own set of patches to Trac. At the time I said I didn't want to have to convert the Trac Subversion archive to darcs (my versioning system of choice) with tailor, and I also didn't want to fork the Trac code.
A few days ago, Stan Seibert e-mailed me to point out that SVK does a fine job of making "private" copies of svn archives, although in further conversation we agreed that it might not be the best way to make your changes available to other people. (I referred to this practice as a "patch stream", and it's something darcs does very well.) Another obstacle was that SVK required a certain amount of Perl-fu, and my development machine wasn't package-managed any more.
Long story short, I decided to go with tailor on my new Debian server. It's written in Python, it's maintained in darcs, and (best of all!) the author uses it to maintain his own patchset for Trac.
Actual information:
First, I installed tailor:
darcs get --partial http://darcs.arstecnica.it/tailor easy_install tailor
I then upgraded subversion and darcs from Debian stable to Debian testing. You can do this in several ways with Debian; I chose to reset my system-wide preferences to testing and then do an 'apt-get dist-upgrade', but the simplest way to do it is this, I believe:
apt-get install -t testing svn darcs
(Warning: if you don't upgrade subversion, your first tailor attempt will end with the error that --limit is an unknown flag to svn.)
At this point you're pretty much ready to run tailor, believe it or not! The tailor README advises you to specify command-line arguments corresponding to the configuration you want, and then just use the '--verbose' flag to output a configuration file. That's what I did, but you still need to read quite a bit to figure out exactly what options to use. Luckily for you I've already done the reading -- and here's the configuration to pull a remote SVN repository into Darcs.
File 'trac.tailor':
[DEFAULT] verbose = True # CTB *4* encoding-errors-policy = ignore[project] target = darcs:target start-revision = INITIAL # CTB *1* root-directory = /tailor/trac state-file = tailor.state source = svn:source subdir = trac
[darcs:target] # CTB *2* repository=/tailor/trac/trac
[svn:source] # CTB *3* module = /trunk repository = http://svn.edgewall.com/repos/trac
There are four configuration points in this file that need discussion.
- First, at CTB *1*, the root-directory. Tailor puts all of
the files, including the repository, in this directory. It must be
writeable by the user running tailor.
- Second, at CTB *2*, the target repository directory. As
far as I can tell, this is entirely ignored.
- Third, at CTB *3*, you need to specify the source repository
independently of the module that you're importing. The module
doesn't need to be a top-level module, either.
- Fourth, at CTB *4*, you will get a unicode exception error midway through your import of Trac if you don't tell it to ignore encoding errors. I'm not sure how to fix this.
Once this configuration file is in place, run tailor trac.tailor, and watch & wait. (It took tailor about 30 minutes to pull in the entire Trac repository of over ~2000 patches -- not extraordinarily fast, but you only need to it once.)
At this point, you have a fully-functional darcs repository. I don't plan to modify it directly -- after all, I don't have check-in access to the Trac archive! -- but you can pull the darcs repository as usual:
darcs get /tailor/trac/trac
and work off of the downstream archive.
In summary, tailor "just worked". Try it out yourself!
I'll make my trac/darcs repository (with the WSGI/SCGI patches in it) available soon; there's still some machine configuration to do before I give out the hostname.
cheers,
--titus