Older blog entries for freetype (starting at number 22)

21 Oct 2005 (updated 21 Oct 2005 at 09:08 UTC) »
FreeType

I'm still trying to get rid of big memory eaters in FreeType. I have found the following offenders to date:

  • the LZW decompressor, used to support pcf.Z and bdf.Z font files on legacy systems (i.e. SunOS) allocates a 400 KB memory block before reading anything from a file.

  • the PCF driver copies large portions of font files into memory, when it should really re-load them on demand.

  • the BDF driver allocates a 64 Kb heap block before reading anything. It also duplicates a lot of things in memory when it should re-read them from the file.

I've started working on the first one. It seems the code we use came from the BSD "uncompress" program. Its source code is cryptic, suffers from bad design and slow read performance. I rewrote it in a few hours into something much more elegant. Try to compare this and this.

The good news is that testing with timR24.pcf.Z shows that opening is now 27% faster, glyph loading is now 63% faster, and that the new code saves 330KB of heap memory with this font. not bad for such a small change :-)

However, this is for a deprecated format that nearly nobody uses. I'll probably work on the PCF reader later. Its source code comes from the XFree86 distribution, and probably needs a good rewrite as well.

I'm also wondering if there isn't a way to optimize the zlib source code used for .gz support. Last time I checked, it was horrible as well, but it seems they've done very good thing for version 1.2.

21 Oct 2005 (updated 21 Oct 2005 at 08:40 UTC) »
Pango, Text Layout and UI Performance

Kudos to Federico for starting to optimize Pango. I know fom experience that even a small performance difference in text layout alone (i.e. without touching glyph rendering) has a huge impact on user's perceived interface speed. That's because toolkits and applications tend to draw a _lot_ of text, even when they could trivially avoid it.

When I had to add Arabic to my company's products, the text layout operation timing initially doubled due to the support of the Unicode BIDI algorithm. It was a custom solution since we couldn't use either ICU or FriBidi. I didn't think it would be a big issue since, in the graphics architecture we were using back then, the time to transfer text glyphs to the screen was very important. My own tests didn't show any perceivable difference anyway when the whole application was running.

Well, I was wrong. It turned out that the application could switch dynamically between English and Arabic presentation mode, and the client complained that scrolling lists in Arabic was slower than in English. How much slower ? Well, a really _tinsy_ bit slower, you had to scroll several pages on the list to really show the difference. But they were horrified by this and considered this to be a top-priority issue.

After a quick analysis, it turned out that:

  • the list component redraw the whole page everytime you went from one item to the next, even though it only needed to update two lines

  • due to the component's design, this required a rewrite, and there was no time to do that (different team, different planning, etc..)

I ended up rewriting the BIDI implementation and the Arabic shaper, with all kinds of tricks. It's probably a bit slower than ICU's algorithm (which is insanely clever, by the way), but at least I knew it would be manageable by someone other than me at the company ;-)

The end result was that laying out Arabic text is now only 10% slower than English in our product, mainly because we don't use OpenType fonts !!.
Problem solved. Except we're probably never going to fix the list component. Ah ah...

Life

Alice is slowly getting better. She doesn't scream as much as she used too, and only wakes up once in the middle of the night. That's a huge improvement, and we're just beginning to recover though my brain is still terribly dizzy at the moment :-)

FreeType

FreeType 2.1.10 is out, this fixes a bunch of bugs, but the memory savings I have mentioned in a previous diary entry are disabled by default. This is because this breaks several libraries on Unix that unfortunately use internal FT2 structures which changed during the optimizations.

The next version will be called 2.2.0, use a major library version number of 7 (instead of 6), and will _not_ install the internal/private headers (this was our original mistake which created this situation). Instead, they'll be replaced by files generating #errors at compile time with a message explaining the situation.

What it means is that an undecided amount of code is going to break when compiled against this version of the library, which is why I've begun writing patches for Pango, FontConfig and a few other libraries. I'm currently working on SDL_ttf and Qt.

It turns out to be a lot simpler than what I expected; my problem is that I don't know the exact list of libraries and applications that directly depend on FreeType internal structures and functions. If you happen to work at at Linux distribution company, or simply have a lot of disk space, huge bandwidth and plenty of free time (which really isn't my case at the moment), I'd be glad if you could grep for some of the lines below and report which packages contain them:

grep "FT_INTERNAL_" sources
grep "freetype/internal/" sources

Of course, 2.2.0 isn't out until we get a sufficient number of patches and make them available on our web site. I don't expect to see that much problem however once the popular libs are out.

And yes, I've used Koders Search (www.koders.com) and it seems absolutely useless for this purpose.

Laptop

I've installed Ubuntu Hoary on my laptop, dual booting with XP, and I'm very impressed with it. However, trying to use it on an on-going basis is _very_ frustrating, because it lacks many of the niceties I'm used to on Windows. I'm not talking about Windows-specific items, rather about all the free cool-addons, most of them free software, that I regularly use.

Does anyone know an equivalent for the following on Gnome, or even, sigh..., on KDE:

  • TortoiseCVS
  • TortoiseSVN
  • Putty Pageant (I mean a Gnome or KDE applet)

And I can't get the damn thing to suspend properly, not even hibernate. I normally don't use the feature, because XP boots _very_ quickly on this machine, but startup is so slow on Linux that it's painful. For the record, I use this machine on the train twice a day, each trip being 30 minutes, it's not like a nearly 2 minutes boot isn't a problem for me.

Grosse Fatigue

I'm a father again. Alice was born on April 3rd this year. She's a magnificent baby, and more importantly, she's absolutely normal and doesn't have any of her sister's problems. Life is good, life is good. And Lucie doesn't even seem to be jealous at the moment.

On the other hand, Alice's still absolutely unable to sleep at night, and my wife and I have been seriously lacking any kind of decent sleep for more than a month now. I'm more lucky since I don't have to breast-feed her, but "a deep rest" is now a distant concept, at least for the next couple of months :-).

Language Wars

rmathew, you might want to take a look at the D Language: from my point of view, this is the best designed language currently available. The pros are:

  • All the niceties of C++ (OO, exceptions, templates, compiled, easy linking to C libraries) without the hurdles (atrocious syntax, atrocious compile times)
  • ultra fast compilation time, excellent execution times (easily beats C++ at many tasks)
  • dynamic arrays, strings, dictionaries and complex numbers as builtins, not ugly library/template hacks
  • the language is garbage-collected: the killer productivity enhancement. (Can be disabled if you want it).
  • built-in support for versioning, design-by-contract and much more.
  • much simpler and cleaner syntax.
  • a GCC Frontend is even available and works on Unix, Windows, MacOSX

There are still some cons at the moment though:

  • first of all, the language is still in beta, which means it's still slightly evolving.
  • the frontend isn't part of the standard GCC distribution, making it more difficult to distribute widely
  • GDB doesn't understand D name mangling yet, which makes debugging quite challenging sometimes. But you won't need it that much in my experience
  • Documentation exists but is scattered around several web pages.
  • It doesn't have the huge runtime libraries that come with Java or C# (though it can interface to C easily)

However, all these points can be solved in a short time frame. I predict this will happen once the language freezes. As for me, I've completely ditched C++ for my personal projects, it simply isn't worth the effort for what I do.

Hope this helps

1 Apr 2005 (updated 1 Apr 2005 at 12:37 UTC) »
Shared Libraries and Memory Usage on Linux

I've written a small program to measure the 'memory payload' of using shared libraries on Linux. What it shows is the amount of private process memory that is taken by the simple fact of _linking_ a program to a shared object. This corresponds roughly to the following data:

  • the specific position-independent-code tables, like the GOT, PLT and DLT. These are automatically generated by the dynamic loader. The more functions the library imports/exports, the bigger these tables are. They also explain why _any_ shared library uses at least 4 KB/process.

  • any writable static data used by the library itself.

  • finally, any absolute pointer constant found in the library, since these cannot be placed within PIC code. Any C++ shared library is going to have all of its vtables in the .data section for this reason.

Note that this does not include the memory used by the libraries when they're used (e.g. when allocating from the heap with 'malloc').

The code is available here. Python 2.1 or above required.

Results are interesting. On my work system (RedHat 7.3), the biggest "eaters" are the following:

 process     KB    KB per    library
   count  total   process
--------------------------------------------------------
       8   1312       164    /usr/lib/qt-2.3.1/lib/libqt.so.2.3.1
      59   1180        20    /lib/i686/libc-2.2.4.so
       8    768        96    /usr/lib/libkdeui.so.3.0.0
      18    576        32    /lib/i686/libpthread-0.9.so
       5    440        88    /local/dturner/opt/lib/libstdc++.so.5.0.5
      19    304        16    /usr/X11R6/lib/libX11.so.6.2
       8    256        32    /usr/lib/libkdecore.so.3.0.0
      59    236         4    /lib/ld-2.2.4.so
       8    224        28    /usr/lib/libstdc++-3-libc6-2-2-2.10.0.so
       4    192        48    /lib/libcrypto.so.0.9.6b
     ...
total payload = 10044 KB

This seems pretty reasonnable, though there are some surprises here (it's an XFCE desktop with only two KDE apps opened, no KDM !). I'd better run the script on more recent distributions to see how they compare however.

Dell Laptops

In October 2002, I was joking at the fact that Dell sells the same computer hardware with a slightly different casing for a lot more dollars. Well, it turns out that this is even more true than I imagined.

I discovered this because I wanted to buy a docking station for my Inspiron 8200, but couldn't find one at an affordable price. While at the same time, many companies on the web sell refurbished Latitude hardware for totally ridiculous amounts (and as you might guess, the Inspiron refuses to work when mounted on a Latitude station).

And then, I found this page which explains how to flash my laptop's BIOS to make it believe it is a Latitude C840. Following the steps detailed here, I could dock my laptop into a C/Port II that I bought for the shocking sum of 17€, shipping included (I'm not kidding) !!.

But this is not all. For some reason, the graphics card in my laptop had previously melted. Very fortunately, I could find someone to sell me a used graphics card for the laptop. It happens that this new card if a Quatro, instead of a GeForce. Because of this, and because I have more than 512 Mb of RAM, the new BIOS believes the machine is now a Precision M50, the "ultra-high-end" "version" of this computer when they were released.

Apparently, these things are known facts, but it's always pleasant to perform a little magic on this kind of hardware. This laptop is now used as a desktop computer, except that it's a lot more silent than the latest bunch of PCs :-)

gilbou, I think you didn't understand the sentence. Guillermito has only been fined for copyright infringement ("contrefaçon"), because he used a pirated copy of Viguard to do its research. This has nothing to do with the fact that he did publish a proof of concept that effectively showed that Tegam's marketing was essentially bullshit ("protects from 100% of all viruses, past, present and future").

Moreover, he has only been sentenced to a "sursis", which means he won't have to pay the 5000 euros, unless he's caught in the act of contrefaçon again. That's not exactly the most severe sentence you could imagine, even though there are still things that do not please me in this affair (but I prefer to see the final judgement before expressing myself on this).

One of the problem however, is that Guillaume Tena himself has a silly attitude where he claims that full disclosure is now illegal in France. This certainly not what has been judged here.

For more details, go to this page

2 Mar 2005 (updated 2 Mar 2005 at 16:54 UTC) »
FreeType & GTK/Gnome heap usage reduction

I've mostly finished optimizing FreeType memory usage when dealing with TrueType fonts, and the results are really interesting. In order to quantify the improvement, I run a fraction of the FreeType benchmark program "ftbench" to time various operations like face open/close, loading glyphs and parsing the charmap. The command itself is:

ftbench -p -b aij fontfile

heap usage was measured with FreeType's internal memory debugger, so the numbers only correspond to heap blocks allocated by the font engine.

The following are respectively the source font file, its size in bytes, and the number of heap bytes saved by the optimization:

  • vera.ttf (65,932 bytes): 53,483 bytes
  • arial.ttf (367,112 bytes): 159,016 bytes
  • times.ttf (409,280 bytes): 162,334 bytes
  • arialuni.ttf (24,131,012 bytes): 1,452,573 bytes
  • batang.ttf (15,619,828 bytes): 2,818,329 bytes

Please note that:

  • I'm comparing FreeType 2.1.9 and FreeType CVS, with memory-mapped files, which are used by default by FT2 when compiled on Unix platforms that support them.

  • The tested program loads all glyphs in a given font, and thus tends to use a lot more heap than typical GTK applications (especially when displaying CJK text). The savings displayed here are probably higher bounds, rather than averages.

  • The optimizations have only been performed on the TrueType and SFNT drivers of the font engines. This means that all other formats are probably eating too much memory. For example, the BDF driver starts by allocating a 64Kb block before reading the file. Even when the file to read is tiny. The Type 1 driver is known to be hungry as well...

I'm a bit ashamed by these numbers :o) I've tried to see if upgrading FT2 to the CVS version on my Mandrake 10.1 machine would result in any visible global memory usage reduction; Surprise, it didn't break it !! it's been unfortunately impossible to get any kind of stable numbers through 'free', 'top' and others. I suspect there is a 200-300 Kb global reduction when starting a fresh KDE 3.2 from boot (without any opened application), but that might come from something completely different. If anyone knows about a good way to measure this, I'd be extremely interested.

If you want to test it yourself on your machine, grab the latest CVS according to the following instructions. using the freetype2 module name, compiling as usual with ./configure --prefix=/usr --disable-static, then make and sudo make install. Then restart your X session. If this crashes, you can recover by following these instructions as root:

  cd /usr/lib
  rm -f libfreetype.so libfreetype.so.6
  ln -s libfreetype.so.6.XXX libfreetype.so
  ln -s libfreetype.so.6.XXX libfreetype.so.6
  restart X

Where XXX is the library minor number corresponding to the your existing installation. Mine was '7' on Mandrake 10.1, and the CVS installs '8' so just changing back the symlinks is enough to get back in a sane state.

Coucou

It's been more than a year since my last post, and this is a cool oportunity to recap what happened to me during all this time. Of course, it's not like anybody's supposed to be interested in this kind of things ;-)

Personal

We moved back to Paris, since my wife wanted to come back to work, since our daughter was getting extremely well (much sooner than the doctors planned). We couldn't both of us spend 3 hours a day in the train and correctly raise our kid, so we said goodbye to the countryside and went back to our 2-room.

After a few months, we found a larger appartment, which we paid a fortune for. Fortunately, the old one was sold at an even higher rate per square meter. Basically, 5 years of savings of two engineers will buy you an additionnal room and a small parking location in a basement in Paris. I think that's no so bad; and we have no mortage to pay. Neat

All in all, we moved three times in a little more than one year, and boy, this is exhausting. I hope we won't be moving until at least a century or two :-)

Now, all I need is a new PapaMobile to drive the family which is going to receive a new girly member in April.

FreeType

It is clear that I had very little time for FreeType this year, and was mostly absent from the mailing list during this period. My family and work, plus all the casualties of the movings (including looking for the new apartment) took all my time.

Things are slowly settling down now that we're "in the place". However, I don't know how things will be in the next few months when we'll have to handle two babies at the same time. Asking people about that is funny, some people will tell you that it's easier than one, others will tell you that it's 4 times more work. I guess it's a sort of lottery depending on the children's characters. Alea jacta est.

We're in the process of moving the FreeType server to different machines. That's mainly because the old server crashed due to a hardware failure, and our current sysadmin doesn't have the time to properly setup everything again on a different machine.

We've decided to move most things on Savannah. In one hand, it's very convenient for us, on the other, CVS is no awfully slow; fortunately, it's not like we need to perform checkouts and commits every half hours :-)

GTK/GNOME memory usage reduction

I've begun to tune the FreeType 2 code in order to reduce its heap usage as much as possible. This should benefit many desktops if done correctly. Interestingly, there are far more optimization opportunities that I initially though, and most of these are easy to do with rather immediate results. I'm however concerned about two things:

  • the first one is that I don't want to sacrifice performance for marginal heap usage gains. this isn't too difficult though.

  • the second one is that I'm modifying internal structures of the library, that are normally never accessed by client applications and libraries. Unfortunately, some popular libraries like fontconfig or Pango do peek at FreeType's internals to achieve various results.

What the second point means is that simply upgrading the library on your system with the new optimizations may break your system by making fontconfig and Pango behave erratically. Until complete testing has been performed, I'm unsure that the optimizations will be enabled by default in the next release of the code.

Of course, the correct thing to do do is fix Pango and fontconfig by mandating that they only use public APIs to do their stuff (and if they lack some API, they can ask for them, we already include some XFree86-specific stuff for example). I suppose that Owen and Keith didn't do that until now because they didn't want to depend on a too recent release of FT2. If you look at the Pango source code, you'll see that it does some pretty weird stuff to compile differently depending on the version of FreeType detected at compile time...

I suppose I'll finish by providing the right patches myself. Oh well...

Work

It's official, the company I work for has been sold to the competition. More exactly, to two distinct competitors. As for myself, I'm now one of Darth Rupert's minions. I can tell you that Microsoft doesn't look that evil from there :-)

22 Jan 2004 (updated 22 Jan 2004 at 14:19 UTC) »

Happy new year to everyone. I'm feeling that 2004 could hardly be worse than 2003 for most of us :-)

Life

Lucie got her second corrective surgical operation, and she's now an absolutely normal 11-month baby. Yep, yep, yep. It seems that the morphin she took at the hospital did disturb her sleep cycles though, and she can hardly fall asleep before midnight these days. I can live with that :-)

Books

Since I've been spending 3 hours a day in a train since last July, I had the time to read a lot of books lately. I have thus tried so far:

  • my father's collection (espionnage, crime, sci-fi). these are mainly "best-sellers" you'll find at airports and train stations. E.g. Ludlum / Crichton / Cornwell. Most of these books are midly entertaining, predictable, and lack humor. Oh well.

  • trying to ressurect my old sci-fi collection. This really means the whole works of Philip K. Dick :-) Also bought a few used books from Robert Heinlein (The Puppet Masters, and Starship Troopers). I recommend these because they're funny, and much better than the films that were written out of them.

  • trying to complete my collection of beloved French authors like Pennac or Benacquista. By the way, I recommend this book to any Frenchman that reads me. It's simply hilarious

  • finally, I've been hunting for new blood. I have discovered James Ellroy and fell in love with it (to the point that his books were the only thing I asked for Xmas :-). I'm currently completing my seventh book: I was expecting the novelty to wear off after some time, but I'm still stick on it.

13 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!