Older blog entries for fejj (starting at number 78)

8 Jun 2002 (updated 8 Jun 2002 at 06:06 UTC) »

Valgrind

tried playing with valgrind to debug evolution-mail at home here, on my celeron 400 w/ 256 mb ram and it was just a might bit slower than molasis flowing uphill in january while being debugged under gdb on a low-end solaris machine remotely over ssh with X forwarding (well, it would be if molasis was a unix X application).

just so you don't get your panties in a twist, I don't put the full blame of this on valgrind (which is why I mention I'm running a celeron - not to mention evolution-mail is a heavy weight champion on steroids).

valgrind has certainly come a long way since I last looked at it a few months ago - in fact, last time I looked at it, it wasn't able to handle any multithreaded application at all, and now it can even handle evolution-mail (albeit a bit slowly on my hardware).

overall, I'm still impressed to say the least. sure beats the pants off my libeaks LD_PRELOAD hack.

POSIX Threads

Let me just say... they screwed the pooch on this one.

I'm extremely disappointed that pthread_once() doesn't block until the callback has been completed. Rather, the first thread to get to the pthread_once() function gets to call the callback while the other threads return immediately and go on their merry way. The whole point of this function as far as I can tell is to provide a means for applications to initialise some static data for later use by the rest of the library/function/whatever. Unfortunately, the fact that all other threads return immediately makes this totally useless for that operation because initialising data in that pthread_once() callback function is not an atomic operation. So here, let me give you the way this should have been implemented in-my-not-so-humble-but-correct-opinion:

typedef struct {
	pthread_mutex_t mutex;
	unsigned int complete;
} pthread_once_t;

#define PTHREAD_ONCE_INIT { PTHREAD_MUTEX_INITIALIZER, 0 }

int pthread_once (pthread_once_t *once, void (*init_func) (void)) { if (!once->complete) { pthread_mutex_lock (&once->mutex); if (!once->complete) { init_func (); once->complete = 1; } pthread_mutex_unlock (&once->mutex); }

return 0; }


While I'm on the subject, I think I'm gonna write a library called ppthreads (Portable Portable Operating System Interface Threads) or maybe tppthreads (Truly Portable Portable Operating System Interface Threads) since pthreads are notoriously not portable at all. Well, at least not beyond the very basics. I've spent a lot of time stumbling over inconsistancies between Linux and Solaris pthreads (and apparently Mac OS X pthreads aren't so complete either) in the Mono project. Everytime I get time to hack on the SPARC port I find myself porting the code over to Solaris.

Isn't it sad when portable doesn't mean portable?

You know, I have one simple request...and that is to have messages with freakin' laser beams attached to their headers. Now evidently my MIME specification informs that that can't be done. Uh, can you remind me what I pay you people for? Honestly, throw me a bone here. What do we have?

18 Apr 2002 (updated 18 Apr 2002 at 04:50 UTC) »

Update: Hmmm, now that I am home and not running the tests over ssh, I'm getting much better performance readings. For example, the memory parser seems to be averaging about 0.945s and my on-disk parser was consistantly reading 0.713s before an optimization I just did that got it down to 0.612s. For those who care, it was a very simple change:

Old code:

inptr = priv->inptr;
inend = priv->inend;

while (inptr <inend) { start = inptr; while (inptr < inend && *inptr != '\n') inptr++;

...


Note that priv->inptr points to the start of data inside an internal 4k read() buffer and priv->inend points to the end of the buffered data within that internal buffer. By adding an extra byte onto the end that we vow never to use except for the following optimization, we can be sure that we will not overflow the internal buffer with the following changes:

New code:

inptr = priv->inptr;
inend = priv->inend;
*inend = '\n';

while (inptr < inend) { start = inptr; while (*inptr != '\n') inptr++;

...


By setting *inend to '\n', we can eliminate our inptr->inend check in the while conditional. This has the result of decreasing the number of instructions from ~7 down to ~4 - that nearly cuts the time wasted in half!

GMime: Just created a 38657516 byte message to get a feel for how my parser would deal with a message that large.

Here are the results, not too bad if you ask me.

ZenTimer: gmime::parser_construct_message took 1.069 seconds
ZenTimer: gmime::message_to_string took 1.731 seconds

For comparison, here are the results of my memory parser from gmime1:

ZenTimer: gmime::parser_construct_message took 1.075 seconds
ZenTimer: gmime::message_to_string took 0.716 seconds

That got me curious as to how long just reading the stream into ram would take, so I wrote a quick test:
ZenTimer: gmime::stream_write_to_stream took 0.542 seconds

Notes: While it looks like my new parser is a hair faster than the memory parser - remember that you need to consider disk buffering that may have been done by the OS. Subsequent calls to the memory parser seemed to yield an average time of 1.050s while subsequent calls to my newer on-disk parser seemed to average closer to 1.078 or so. Just so I don't leave you hanging on the average time for writing the message to ram, it consistantly took about 0.540s (never going more than about 0.002s in either direction).

The structure of the MIME message, in case anyone is interested, was:

Content-Type: multipart/mixed
   Content-Type: multipart/related
      Content-Type: multipart/alternative
         Content-Type: text/plain
         Content-Type: text/html
      Content-Type: image/gif
      Content-Type: image/png
      Content-Type: image/png
   Content-Type: application/x-gzip
   Content-Type: image/png
   Content-Type: image/png
   Content-Type: text/x-c
   Content-Type: application/pdf
   Content-Type: application/pdf
   Content-Type: application/pdf
   Content-Type: application/pdf
   Content-Type: application/pdf
   Content-Type: application/pdf
   Content-Type: application/pdf
   Content-Type: application/pdf
   Content-Type: text/x-c
   Content-Type: image/png
   Content-Type: image/png
   Content-Type: application/pdf
   Content-Type: image/jpeg
   Content-Type: image/png
   Content-Type: image/jpeg
   Content-Type: image/x-portable-anymap
15 Apr 2002 (updated 18 Apr 2002 at 00:30 UTC) »

GMime: Never underestimate the power of Mirwais - Disco Science.ogg. Started implementing that parser I talked about in my last diary entry at around 11pm last night, around 5am this morning the gnome community sees this:

<fejj_sleep> whooo!! my new gmime parser is fucking
fast!!!
<andersca> cool
<fejj_sleep> unf
<andersca> you are very cool fejj
<fejj_sleep> haha, thanks
<fejj_sleep> it's almost as fast as a previous parser
of mine which loaded the entire message into ram before
processing
<fejj_sleep> we're talking a hare difference in speed
<fejj_sleep> and my new parser parses off disk
<andersca> cool
* andersca certifies fejj as master
<fejj_sleep> :)
<jamesh> if this is what fejj can do while sleeping,
think of what he is like when awake

How fast is fast do you ask? Let me show you a comparison of my latest 3 parsers taking as input a ~1.1MB MIME message (a multipart containing a multipart of X screen dumps):

gmime1:
memory parser: 0.051s
on-disk parser: 0.173s (buffered)
on-disk parser: 5.060s (unbuffered)

gmime2:
on-disk parser: 0.057s (varies between 0.056s and 0.057s)

That, my friend, is what we say in hacker land is fast :-)

Update: Just got pointed to this article by a friend: http://www.nodewarrior.org/minorfish/mf-users-archive/archive-20021/0006.html

Gotta love hearing that stuff about your software, makes ya all warm n fuzzy ;-)

GMime: Since I haven't mentioned GMime in a while, here's what's been going down: I've now branched GMime developement. Tag gmime-1 is for glib1.2 development and HEAD is now being ported to glib2. So far I think I have everything but the parser ported. GMime2 also features a restructured class structure as well as being ported to use GObject for both streams and MIME objects.

You'll notice that I have a g_mime_object_register_type, this is for registering new MIME object types at run-time. Why would you want this? Well, maybe you want a special MIME object for pkcs7 encrypted or signed MIME parts or something. Who knows. The idea is that it will be extendable by the application. The parser will use the g_mime_object_new_type() interface to create new MIME objects as it parses using the content-type header so as to need to know absolutely zero about the internals of the MIME part. This is a major change from the way gmime1 worked.

Choke to death on your damn designer bagel from Baldusha's. No cholesterol, naturally. Fuck your big ol' Sunday New York Times. Fuck the Wallstreet Journal, and NewsWeek, and the lot. Stupid set of privileged mother fuckers... think it's fashionable to have an "alternative view". An "alternative view". Hah. And fuck, if you can, your pencil-thin, avian-drinking, calorie-counting, caffeine-limiting, sodium-sparing, neutra-sweet-sweetening, rear-view-mirror-preening, carrot-nibbling body. Go drown in a lake of diet coke, fuckah.

Went out last night after work with my cousin Craig, his fiance Shelly, and my cousin's roommate Joe to see Blade2. Afterwards we went to Boston Beer Works for refreshments.

29 Mar 2002 (updated 29 Mar 2002 at 01:07 UTC) »

Step AWAY from the crackpipe...

Had to deal with this loser today.

Yes, Qpopper is disabling the account because it can tell you are using telnet. Uh huh. Riiiight. Get a clue, retard.

My favourite bit is when he decides to prove that every other email client can access this disabled POP server by setting up fetchmail to connect to a completely different POP server altogether.

I've been having to deal with a ton of retard users lately. Yesterday I had to deal with some guy that claimed Evolution couldn't login to his POP server for some reason or another. I asked him to verify that he had his account settings right, specifically a particular part (which I forget) and he kept telling me I had no idea what I was talking about. He kept bitching to me all day. At one point he even sent me an email claiming that the reason for Tech Support was because developers were idiots and users like himself knew it all (I'm serious, I can dig it up if you like). Finally about an hour before I left for home, I get an email by him saying I was right all along, he had misconfigured his account.

Note to users out there: Please. The developer of the software knows more about the software than you do. If he/she asks you to try something, don't come back and tell them they don't know what they're talking about. First, you won't make them want to help you more and secondly, you're almost guarenteed to be wrong about them being wrong. This doesn't mean that you can't offer suggestions/ideas, but don't try to play like you're smarter than the developer.

Mono SPARC: For those wondering how my SPARC port is coming with the mono interp, it's semi-working at this stage. It passes 60 or 70 some-odd percent of the tests in the test suite. Unfortunately I haven't had a chance to work on it in the last 2 weeks, but I plan to do some SPARC assembly hacking action this weekend ;-)

Well... does he like butter tarts?

Hacking: miguel has done it again. He has somehow convinced me to port mono to SPARC. To do this though, I first had to make it build on Solaris 2.8. I've had to do this twice so far this week, once on Tuesday or Wednesday and again today due to so much code changing. Unfortunately this has meant that I haven't had time to work on making it work on SPARC yet ;-)

As it turns out, last weekend I had started reading some SPARC architecture manuals and had started to write sparc-codegen.h. As it turns out, Paolo had the same idea at the same time I did so he had also started hacking this. Maybe that was a good thing. I hadn't gotten very far with sparc-codegen.h anyway mostly because I wasn't quite sure what I was supposed to be writing since <person>miguel</person hadn't been to clear on that. By the time he explained what I was supposed to be writing, I had a massive headache (lack of food?) and so decided to give up for the night and head home.

Monday morning miguel told me that Paolo had also started porting to SPARC but that he didn't have access to a SPARC machine and so it would be my honour to finish the port :-)

Yay!

Okay, so you may have read that "yay" as being sarcastic, and it sort of is. But at the same time I am in some ways relieved that I get to work on the SPARC port because reading the manuals made me interested in the architecture and this would give me a great opportunity to learn something new.

So... where am I now? I've just gotten mono to build on Solaris (minus the jitter) and the disassembler works. I then tested mint (mono interpreter) and got this:

bash$ ./mint --trace ~/hello_world.exe 
  Entering System.String::.cctor ()
  Leaving System.String::.cctor
Sending dummy where System.AssemblyLoadEventHandler expected
Sending dummy where System.UnhandledExceptionEventHandler
expected
Sending dummy where System.CrossAppDomainDelegate expected
Sending dummy where System.AssemblyLoadEventHandler expected
Sending dummy where System.AssemblyLoadEventHandler expected
Sending dummy where System.UnhandledExceptionEventHandler
expected
Sending dummy where System.UnhandledExceptionEventHandler
expected
Sending dummy where System.CrossAppDomainDelegate expected
Sending dummy where System.AssemblyLoadEventHandler expected
Sending dummy where System.AssemblyLoadEventHandler expected
Sending dummy where System.UnhandledExceptionEventHandler
expected
Sending dummy where System.UnhandledExceptionEventHandler
expected
  Entering System.Threading.Thread::.cctor ()
    Entering System.Collections.Hashtable::.cctor ()
      Entering System.Object::.ctor ()
      Leaving System.Object::.ctor
      Entering System.Int32::.cctor ()
        Entering System.Type::.cctor ()
Bus Error (core dumped)

gdb tells me:

Program received signal SIGBUS, Bus error.
0x178e0 in stackval_to_data (type=0xa2400, val=0xffbee2e8,
data=0xa8c52 "")
    at interp.c:359
359                     *p = val->data.p;
(gdb) p val
$1 = (stackval *) 0xffbee2e8
(gdb) p p
$2 = (void **) 0xa8c52
(gdb) p *p
$3 = (void *) 0x0
(gdb) p val->data.p
$4 = (void *) 0xa6748
(gdb) p *p = val->data.p

Now what? :-)

69 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!