Fefe is currently certified at Master level.

Name: Felix von Leitner
Member since: 2000-05-13 20:30:07
Last Login: 2008-12-09 18:04:38

FOAF RDF Share This

Homepage: http://www.fefe.de/

Notes:

Projects

Articles Posted by Fefe

Recent blog entries by Fefe

Syndication: RSS 2.0

A few small updates: The ACLs in tinyldap work, at least for the read case. I also added partial write support now. tinyldap can not only serve data from the read-only data structure, but also from a journal that is read in when the server process starts. Since the tinyldap model is to start one process per client, this works well, but I will add some code to check whether there's news in the journal during the run-time of the server processes, too.

I also added support for LDAP AddRequests, but no other writes yet. My blog software now writes to the journal directly, and it helped greatly. I initially wrote my blog to use LDAP, so I'd have a "customer zero" for tinyldap, and would be compelled to iron out kinks I found while using it. It worked in that it made me add the journaling code, but it also failed in that I now write to the journal directly instead of using the ldap modify requests and so on. Important lesson: using your own laziness to get things done does not always work.

A small update on gatling, too. I added a mode to have more than one process, and I expected it to speed up the cold cache case, because then more than one disk read request would be outstanding at a time, and the OS could use the I/O elevator to optimize the movement of the disk head. To my great surprise it turned out that that case did not get faster at all, but the hot cache case had a x2 speedup when using two cores. This is all on Linux, mind you, but the results are almost the same on other operating systems. See this page for details.

I have been drafted to do two talks at the CCC Camp. One will be about the dark side of C++ (why it sucks to audit C++ code) and the other one will be about what optimizations you should not do on your source code, because the compiler already does them for you, and it makes the code less readable.

27 Sep 2005 (updated 27 Sep 2005 at 10:30 UTC) »

The last half year has been quite successful from a business point of view, but it has cost me in terms of time spent on writing free software.

Nonetheless, I have found the time to write rudimentary ACL support for tinyldap, to find out whether the idea works at all. It is still far from finished: you can for example still query using attributes that you are not allowed to see in the ACLs, as long as you don't ask for the attributes in the answer record. And there is headroom for optimization.

Besides that, I have been thinking about adding threads to gatling, to make open() asynchronous. I have seen a production web server serving lots of small images which don't fit into memory at the same time, where gatling was limited by the serialization of the read I/O. Something needs to be done about this, obviously.

Anyway, I am now moving my diary to blog.fefe.de, where I mostly collect URLs that I find interesting, but you will also find a few articles between the links.

27 Jan 2005 (updated 27 Jan 2005 at 01:06 UTC) »

Another year has passed, and much has happened.

From my event notification framework a web server has risen. I named it gatling because it was designed to serve many small requests. gatling was meant not only as web server, but as generic file server. The idea is: you go somewhere, want to give someone some files, and you just go to the directory with the file and start gatling. gatling than exports the files using as many protocols as possible. So far only HTTP and FTP are actually implemented, and HTTPS. But SMB and NFS are planned. Both protocols are unfortunately very ugly and need more management infrastructure than HTTP and FTP.

I also wrote an anti-spam patch for qmail, find it at www.fefe.de/qmail/. It checks whether envelope sender addresses can be replied to, whether there is some known open proxy or relay on the connecting IP, and some other stuff. Surprisingly effective.

I started writing some ACL infrastructure for tinyldap. Some interesting questions pose themselves. What parameters do you allow for rule parametrization? Only by DN? From the qmail anti-spam patch I have code to match a string against a known list of (sub-)strings, so I could basically match a dn against an arbitrary list of rules with complexity bounded by the length of the dn to be checked, not by the number of rules. However, the more I think about it, the more I think rules should not be restricted to DNs. It should be possible to, for example, match by objectClass. Some further thought is needed on this.

Oh, and I finally bought myself that Athlon 64 I was talking about in the last entry. Surprisingly it is much less noisy than my last system. I watched out for quiet components without compromising on performance, and it is possible. Now that my computer is so quiet, I start to notice how noisy the rest of my environment is :-)

ARGH! I vastly underestimated the relative size of the strings in the LDAP database and the offsets to the strings. The ratio is about 2:1. I.e. for two megs of raw data, I need one more meg for the offsets. And that is not counting the indices (which are much smaller).

parse still can't handle the phone CD. The reason is that Linux will not let me mmap more than 2 gigs. This probably has to do with some stupid compile time constants like the relative position of code, stack, reserved kernel space and the mmap space. Or so. I'm just pulling this out of thin air, sorry. So if there ever was a compelling reason for 64-bit computing, this would be it. Time to get myself an Athlon 64 box. Anyone have a spare one for me? *sigh*

On the other hand, I could have foreseen it. I haven't counted them, but I expect the phone CD to carry at least 20 million records, which means at least 80 megs just for the pointers to the records. This is a lot of data. Maybe I can think of an easy way to be more space efficient. Let's sleep over it. Comments or ideas welcome.

File systems keep pointer sizes down by segmenting the data into smaller independent chunks. Maybe I should do the same. The data could be encoded with a static Huffman tree. Bad tradeoff here. Maybe I will just have to settle for the data for Berlin and Brandenburg or so. Too bad. I really looked forward to this.

The event notification stuff has progressed quite nicely. It is the basis of some benchmarks I did on Linux and the BSDs, and it made quite a wave when I posted it on Slashdot.

The goal was to create a good abstraction for the various event notification APIs, but also around the various sendfile variants, so it would be easy to write a highly web server. The abstraction part is in libowfat, and the web server is called gatling (it's unreleased so far, but you can get the source code via cvs from :pserver:cvs@cvs.fefe.de:/cvs, check out "gatling".

gatling now uses sendfile on Linux, FreeBSD, Solaris, HP-UX, AIX, and uses mmap+write on the other systems (this provides zero-copy TCP at least on NetBSD and IRIX). gatling uses better-than-poll event notification APIs on Linux, the BSDs, IRIX, Solaris and MacOS X. So it should be the ideal basis to write highly scalable network servers and clients.

gatling is not just an HTTP server, it also is an FTP server. And the beauty of it is that it can be used in ad-hoc mode, i.e. you just run it and the current directory is exported read-only to the world. Uploads are possible via FTP, but only if the receiving directory is world writable. This is incredibly useful, much more so than I would have initially thought. I'm planning to also add SMB and NFS read-only export. Then gatling would be the ideal LAN party network publication tool.

Also, I started hacking on tinyldap again. A friend of mine reverse engineered the database format on a "telephone book CD" for Germany. I thought to myself: this would be a formidable data set to put into tinyldap to prove that it is not just a toy. So he wrote a converter that spit out tab separated data and I converted it to LDIF and it's over 2 Gigs worth of ASCII LDIF. Whew. tinyldap assumes that all your data fits into memory. Not so much tinyldap itself but the programs that create the data file tinyldap serves from. So I started hacking on parse (which creates the binary data file but no indices on it). I ended up completely reworking it and over days my local source tree would not create correct data files. It was a strange experience for me because normally I only do small changes on my local cvs checkout and check every change in immediately. On the other hand, I always wait until it at least compiles and survives very basic tests. But now I checked everything in.

The next step is to convert the index generation to work like sort(1), i.e. quicksort on smaller blocks and then mergesort on these blocks.

That's harder than it sounds because I am sorting offsets. The comparison function has to seek to the offset and compare the data there. So the merge sort would drown the disk in wild seeking around unless I attach the first n bytes behind the offset to the offset before the merge sort and throw it away afterwards. I'm curious how much seeking I will end up doing.

I am also seeking sponsors for my work on tinyldap. If you run company working with LDAP, please go to your LDAP admin when he is doing LDAP administration and see for yourself. Chances are, he will be cursing all the time. Why not spend some money on the development of an LDAP server that does not suck? Hey, you can even say what features you want implemented! If you are interested, please contact me.

34 older entries...

 

Fefe certified others as follows:

  • Fefe certified Fefe as Journeyer
  • Fefe certified andreas as Master
  • Fefe certified LaForge as Master
  • Fefe certified uweo as Journeyer
  • Fefe certified andersee as Journeyer

Others have certified Fefe as follows:

  • Fefe certified Fefe as Journeyer
  • andreas certified Fefe as Master
  • bagder certified Fefe as Journeyer
  • oec certified Fefe as Master
  • lars certified Fefe as Master
  • thalunil certified Fefe as Master
  • doobee certified Fefe as Master
  • mitch certified Fefe as Master
  • jLoki certified Fefe as Journeyer
  • ErikLevy certified Fefe as Journeyer
  • cord certified Fefe as Journeyer
  • uweo certified Fefe as Journeyer
  • LaForge certified Fefe as Master
  • kju certified Fefe as Master
  • RobinS certified Fefe as Master
  • ths certified Fefe as Journeyer
  • mcmilk certified Fefe as Master
  • 2B certified Fefe as Master
  • Omnifarious certified Fefe as Journeyer
  • technik certified Fefe as Master
  • twistor certified Fefe as Master
  • caf certified Fefe as Master
  • andersee certified Fefe as Journeyer
  • alpha certified Fefe as Journeyer
  • maelstorm certified Fefe as Journeyer
  • fvw certified Fefe as Master
  • accdias certified Fefe as Journeyer
  • rojaro certified Fefe as Master
  • adulau certified Fefe as Master
  • fxn certified Fefe as Journeyer
  • AntonA certified Fefe as Master
  • ak certified Fefe as Master
  • Miod certified Fefe as Master
  • thrx certified Fefe as Journeyer
  • mbacarella certified Fefe as Master
  • ghaering certified Fefe as Master
  • Stevey certified Fefe as Master
  • walpuski certified Fefe as Journeyer
  • kelimek certified Fefe as Journeyer
  • jnewbigin certified Fefe as Master
  • hukka certified Fefe as Master
  • esden certified Fefe as Master
  • pphaneuf certified Fefe as Master
  • pasky certified Fefe as Master
  • realblades certified Fefe as Journeyer
  • 8088 certified Fefe as Master
  • frehberg certified Fefe as Master
  • acme certified Fefe as Master
  • swilde certified Fefe as Master
  • insom certified Fefe as Master
  • cdfrey certified Fefe as Journeyer
  • metaur certified Fefe as Master
  • garbeam certified Fefe as Journeyer
  • sqlguru certified Fefe as Master
  • schugo certified Fefe as Journeyer
  • uriel certified Fefe as Journeyer
  • uzume certified Fefe as Journeyer
  • chalst certified Fefe as Journeyer
  • argp certified Fefe as Master
  • shlomif certified Fefe as Journeyer
  • ld certified Fefe as Journeyer
  • jhw certified Fefe as Master
  • RiscTaker certified Fefe as Master

[ Certification disabled because you're not logged in. ]

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

X
Share this page