I've been busy working on a bunch of stuff.

* The log analysis stuff is coming along nicely. Thankyou very much SQLite. I'll post a specific update or two about that when I've finished fixing the bugs I've introduced.

* I've modified the pygrub boot loader to understand FreeBSD disk labels. The hacking can be found at http://people.freebsd.org/~adrian/xen/ in the bsd_pygrub directory. It turns out that the pygrub/xen UFS code is (a) Solaris UFS, (b) UFS-1- only, (c) crashes very badly when fed a FreeBSD formatted UFS1 for some reason. I'll investigate that shortly. It is one more step towards sensible FreeBSD/Xen integration though!

* I've been fixing bugs and adding features to my Squid-2 fork, Lusca. I've found and fixed a couple of nasty bugs inherited from Squid-2.HEAD (especially one to do with 304 replies not making it back to the client!) and I've started documenting how all of the transparent hijacking/intercepting code works.

I'm doing some pretty heavy customisation of "Lightsquid", a GPL'ed squid logfile analysis tool. I'd like to be able to offer Xenion customers a decent set of management and reporting tools.

The Lightsquid interface is reasonably simple, fast and snappy. It captures the right amount of information for the average network/system administrator. There are a few problems though.

The HTML needs an overhaul. Its nested table hell. It is all done via a custom template engine so it shouldn't be too painful.

The parser seems to assume you're going to feed it all of the logs for a given day. If you feed it half a day at a time, the second import will over-write the first.

The data is stored in flat files, indexed by day. This is fine - a year is 365 directories - and trolling each directory to pull the daily stats isn't too bad. But the per-user statistics are kept in single files, one per user per day. Generating a monthly or yearly user report per user is a very, very expensive operation. Multiply that by a few thousand users and it just won't scale.

I'm going to have to abstract out the data storage and retrieval into a simple API; then implement a database backend for it. This API should implement an "add" functionality so I can handle adding data to an existing day repository.

There probably won't be much of the original Lightsquid code left when I'm eventually done with it.

Then once that is done, I can focus on some better monitoring and management tools.

I've been slowly trying to sort out the bugs I've been seeing in the FreeBSD Xen code. The first annoying crash(es) are in the netfront network driver.

I started digging in the driver and found that the bug in question is due to some reasonably brain dead behaviour in the TX mbuf ring allocation. I've committed code to FreeBSD-current which enforces that there's space in the TX mbuf ring. I'm about to commit further fixes to enforce that there's space in the Xenbus TX ring.

The current Linux code has the same flaw. It may not be so easily tickled in Linux due to the driver model but it should still be explicitly catered for. That said, I get plenty of random network interface hangs in Linux/Xen; perhaps this stuff is related.

I've spent the weekend building and testing a new VM server to deploy next week.

I've taken the oppertunity to test out FreeBSD/Xen support. I've put my images up here:

http://wiki.freebsd.or g/AdrianChadd/XenImages

It actually works, which is a good starting point!

.. hm, its been a while since I braindumped into here. I really should pick one or two blog sites (nerd, work, personal) instead of having dozens of sites spread around everywhere.

Anyway. I've just been bootstrapping FreeBSD-current/Xen and attempting to document the process so others can also test it out.

http://wiki.freebsd .org/AdrianChadd/XenHackery

I've also forked Squid-2 off into a separate project - lusca

I've also built a small open source CDN out of it which I need to spend more time on (but can't because I have to make money somehow..) - Cacheboy

I'm also doing bits of web/VPS hosting, squid/network/systems consulting and bits of other work - Xenion - this is how I'm trying to pay the bills so I can spend more time working on useful open source stuff.

More to come!