14 Feb 2004 Fefe   » (Master)

The event notification stuff has progressed quite nicely. It is the basis of some benchmarks I did on Linux and the BSDs, and it made quite a wave when I posted it on Slashdot.

The goal was to create a good abstraction for the various event notification APIs, but also around the various sendfile variants, so it would be easy to write a highly web server. The abstraction part is in libowfat, and the web server is called gatling (it's unreleased so far, but you can get the source code via cvs from :pserver:cvs@cvs.fefe.de:/cvs, check out "gatling".

gatling now uses sendfile on Linux, FreeBSD, Solaris, HP-UX, AIX, and uses mmap+write on the other systems (this provides zero-copy TCP at least on NetBSD and IRIX). gatling uses better-than-poll event notification APIs on Linux, the BSDs, IRIX, Solaris and MacOS X. So it should be the ideal basis to write highly scalable network servers and clients.

gatling is not just an HTTP server, it also is an FTP server. And the beauty of it is that it can be used in ad-hoc mode, i.e. you just run it and the current directory is exported read-only to the world. Uploads are possible via FTP, but only if the receiving directory is world writable. This is incredibly useful, much more so than I would have initially thought. I'm planning to also add SMB and NFS read-only export. Then gatling would be the ideal LAN party network publication tool.

Also, I started hacking on tinyldap again. A friend of mine reverse engineered the database format on a "telephone book CD" for Germany. I thought to myself: this would be a formidable data set to put into tinyldap to prove that it is not just a toy. So he wrote a converter that spit out tab separated data and I converted it to LDIF and it's over 2 Gigs worth of ASCII LDIF. Whew. tinyldap assumes that all your data fits into memory. Not so much tinyldap itself but the programs that create the data file tinyldap serves from. So I started hacking on parse (which creates the binary data file but no indices on it). I ended up completely reworking it and over days my local source tree would not create correct data files. It was a strange experience for me because normally I only do small changes on my local cvs checkout and check every change in immediately. On the other hand, I always wait until it at least compiles and survives very basic tests. But now I checked everything in.

The next step is to convert the index generation to work like sort(1), i.e. quicksort on smaller blocks and then mergesort on these blocks.

That's harder than it sounds because I am sorting offsets. The comparison function has to seek to the offset and compare the data there. So the merge sort would drown the disk in wild seeking around unless I attach the first n bytes behind the offset to the offset before the merge sort and throw it away afterwards. I'm curious how much seeking I will end up doing.

I am also seeking sponsors for my work on tinyldap. If you run company working with LDAP, please go to your LDAP admin when he is doing LDAP administration and see for yourself. Chances are, he will be cursing all the time. Why not spend some money on the development of an LDAP server that does not suck? Hey, you can even say what features you want implemented! If you are interested, please contact me.

Latest blog entries     Older blog entries

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!