Advogato blog for salimma

ABI breakage and package naming

Tue, 15 Jul 2008 15:06:13 GMT

Daniel Béranger has raised the ABI issue surrounding Fedora and RHEL’s recent upgrade to Firefox 3. In short, RHEL 5.2 ships with Firefox updated to the new xulrunner-based Firefox 3, but its Eclipse and libswt3-gtk2 is still at 3.2, which depends on the old gtkmozembed interface.

This seems like a good argument in favour of adopting Debian-style package naming, when it comes to libraries: append the ABI version to the (sub)package name containing libraries that are linked to from other packages. So the old Firefox 1.5 would have a libgtkmozembed18 subpackage, that can be shipped with RHEL 5.2 without shipping the rest of Firefox 1.5 (technically speaking, the ABI version is 1.8.x.y, as prior to XULRunner 1.9 the ABI is ever-changing, but packages already handle this by depending on the specific version and release number anyway)

This is already done from time to time in the RHEL/Fedora world, in the form of compatibility packages, but making it the default would avoid this kind of breakage, where a package /has/ to be updated (due to upstream EOL) but parts of it are needed downstream.

Midori : NT/Vista :: NT : Win95

Wed, 9 Jul 2008 05:06:19 GMT

Looks like Microsoft is preparing for their next great leap forward in OS design. Just as Windows NT’s kernel is a clean room without any DOS baggage, Midori is based on the Singularity research kernel, that is written in .NET and utilizes a new compiler backend to output native code.

This will be rather interesting to watch. The idea of writing an OS kernel in a strongly-typed language makes sense — witness House and Singularity. The effort is not expected to be ready for years — this is not Windows 7, and I’d guess there will still be a traditional Windows 8, even if Midori is ready by then.

Incidentally, the OSS community already has a strongly-typed virtual machine designed for efficient native code generation: LLVM. If one takes a Unix kernel (or, more practically, microkernel) and get it to compile using LLVM’s C front-end, one then has the opportunity to gradually rewrite it one module at a time in any language with LLVM front-ends. In the time it will take for Midori to get ready, would there perhaps be an ML-like front-end to LLVM?

clipped from arstechnica.com

Every once in a while, an article gets posted somewhere in the blogosphere about how Microsoft needs to release a complete Windows rewrite, something along the lines of what Apple did with Mac OS X. Most people realize that Microsoft is in no position to pull a stunt like that at the moment; it’s hard to see Microsoft phasing out support for a billion-Windows-PC-strong user base, but that day may one day come, perhaps thanks to robust virtualization technology.

But in the meantime, Microsoft has settled on rewriting bits and parts of the Windows operating system as it sees fit, with Vista being one of the biggest rewrites (a fact which partially explains the many hardware and software compatibility issues XP’s successor experienced at launch). Speculation around a non-Windows operating system in the works at Microsoft has been present for years, but recent trustworthy tidbits of information have found their way into the hands of Microsoft ZDNet bloggers Mary Jo Foley and Ed Bott.

Wide Finder: take 2

Mon, 16 Jun 2008 04:07:21 GMT

Tim Bray’s revised Wide Finder project [ongoing.org] has been ongoing for a few weeks now, and I’ve finally took the time to design and prototype an implementation.

What
The goal is to evaluate the performance of middle-of-the-road, not embarrassingly parralelizable tasks on modern-day multi-core hardware. Such as the Sun T2000 servers. Fittingly, the task is to parse a multi-gigabyte web server log file and compile some aggregate statistics.

Design
The solution I came up with for the earlier iteration of the contest, coded in different versions (C++, OCaml and JoCaml) is fundamentally sound, though rather unoptimized (picking up two-and-a-half different languages in one weekend is a good way to find out how much there is to know about, say, C++ stream buffering). With the benefit of hindsight, and given that we are several weeks into the project and there are strong implementations already [wikis.sun.com], the idea is to find an unexplored niche.

Short recap of the main implementations:

OCaml: Fernandez is ahead of the pack again, the only solution in the 7 minutes
make+C+awk+sh: Perl is dead, but shell scripting is enjoying a renaissance with parallelizable tasks. 8 minutes
Java-based solutions: in the 13-17 minutes range are the various JVM solutions, from Java, Groovy and Scala to Fan, an interesting Ruby-like language for the JVM. Reminds me of .NET’s Boo.
Python, Ruby: in the 20+ minutes range. Python multiprocessing is not that efficient yet; I believe an improved Stackless Python solution might be forthcoming

I dabbled with a Common Lisp solution; it works and appears to be competitive, when tried on a partial log file. Exploring the available options for parallelism, however, revealed the disconcerting fact: no freely-available Common Lisp compilers have good multi-threading, or even multi-processing (without shared memory) on Solaris! Even worse, the SBCL incompatibility with GCC 4.3 means that even the Linux version on my Fedora machine is several months old, and does not have the threading library.

So it’s back to Java. Perusing the blogs of the Java and Scala programmers, it appears that the common complain is .. regular expressions. So the hunt was on for a good regular expression library. Joni, a port of the Ruby Oniguruma regex library to the JVM used by the JRuby project, appears ideal: low-level and supposedly very fast. Until one hits the total lack of documentation. So that’s off the table. Ended up using dk.brics.automaton, which appears to perform well enough, even when parsing Unicode strings.

The nice thing about using Java is that, if you hit a performance brick wall, chances are that many other people have been there before you. The problem I have, the need to have random access within a file (so different threads can start at different offsets (Java’s RandomAccessFile is good for this) combined with the need for buffered I/O (BufferedReader is good, but there is no RandomAccessReader !) is solved by the nice folks at Biojava.com. Great!

On my system (2 GHz Core 2, 2 GB RAM, 5400 rpm HDD, Fedora 9 x64), Ruby takes about 2.2 seconds, while my Java implementation running on OpenJDK 1.6 (64-bit) takes about 1.6-1.7 seconds with 1 thread and 1.4-1.5 seconds with 2 threads. Close to the 1.2+ seconds time that Alex reported for Python, but hey, we’re paying the Java start-up cost here.

Will update when I get an account on the test server. 40GB dataset, here I come! In the meantime, time to look for opportunities to use a JVM-based language better suited for the task. The Java code is a tad bit verbose.

C types 101

Mon, 9 Jun 2008 02:13:59 GMT

I was cleaning up the code of an application that I’m packaging for Fedora, and was Googling for information on size_t (in the code, a size_t variable was being printed as a normal integer (%d), which triggered a compiler warning, and I forgot what the relevant option is. Ended up finding it in printf’s manpage) when I discovered this rather well-written gem.

clipped from www.embedded.com

Using size_t appropriately can improve the portability, efficiency, or readability of your code. Maybe even all three.

Numerous functions in the Standard C library accept arguments or return values that represent object sizes in bytes. For example, the lone argument in malloc(n) specifies the size of the object to be allocated, and the last argument in memcpy(s1, s2, n) specifies the size of the object to be copied. The return value of strlen(s) yields the length of (the number of characters in) null-terminated character array s excluding the null character, which isn’t exactly the size of s, but it’s in the ballpark.

You might reasonably expect these parameters and return types that represent sizes to be declared with type int (possibly long and/or unsigned), but they aren’t. Rather, the C standard declares them as type size_t. According to the standard, the declaration for malloc should appear in <stdlib.h> as something equivalent to:

void *malloc(size_t n);

The ultimate Fedora 9 setup: Part 1 - UI

Mon, 2 Jun 2008 21:14:18 GMT

I tend to wipe off my old Linux setup and reinstall everytime a new release comes out — not that it is necessary, but it’s a good way to get rid of old cruft that have gotten installed, found not useful enough, and then forgotten (and there are lots of those, as df -u shows).

It also makes it easier to ascertain the state of Fedora — any (mis-)configurations that I might have corrected would be reset to the default.

So, now that Fedora 9 has been released, what needs to be added to / changed from the base setup? Surprisingly, not that many:

Compositing
Some people swear by Compiz; I personally find Metacity much more usable (Compiz does not support cycling through all windows of a given application — Ctrl+F6 in Metacity; Cmd+~ in OS X). Metacity now has a compositing manager that’s turned off by default; turning it on involves either using gconftool-2 (only for advanced users) or gconf-editor, and setting the /apps/metacity/general/compositing_manager key to true.

The support in the stable version is a bit flaky still; the metacity package in Rawhide is much better behaved and appears quite stable. Upgrade by issuing yum –enablerepo=rawhide update metacity. As of the moment it does not pull in any other Rawhide package so you can rest easy.

Try pressing the volume up/down/mute keys on your keyboard (if you don’t have a multimedia keyboard, change the bindings in System->Preferences->Personal->Keyboard Shortcuts) and be amazed at the translucency coolness (no, this is not bling). The brightness pop-up windows have not been changed yet, alas.

Firefox
Ever cursed Firefox’s font rendering in silence? Type about:config in the address bar, and add the following boolean keys:

font.FreeType2.autohinted = true font.FreeType2.enable = true

Keyboard
For the English-speakers among us specifically, and those who use the US keyboard layout in general (it’s the standard layout in Indonesia, for instance), the occasional times when one has to type an accented character is rather annoying.

There are various work-arounds — launch the character map (under Accessories), add the Character Palette applet to the panel (so that it consumes RAM even when you don’t use it!)…*or* you can just fix your keyboard layout. The die-hard command-line junkie would be able to tell you what option to pass to setxkbmap to achieve this. The rest of us can just use System->Preferences->Hardware->Keyboard. In the “Layouts” tab, select “Layout Options”. The option you want is “Compose key position”; I use Right Alt, but Caps Lock haters will rejoice to know that, yes, you can use that dreaded key as your compose key as well. To type an accented character, now the only thing you need to do is hit the Compose+accent followed by the letter you want to accent (using shift as necessary, e.g. for ^).

While you’re here, you might want to change the Alt/Win key behavior, and map either Meta, Super or Hyper to one of your Win-keys. The GNOME default is inexplicably for the Win-key to be a normal key and not a modifier (so it cannot be combined with other keys).

Coming up: Applications
Et voilà! You should have a nice-looking, and more importantly, functional desktop right now. In the next instalment, I’ll comment on the applications I use. Until then, à bientôt!

Programming languages, in a nutshell

Tue, 4 Dec 2007 09:14:06 GMT

Task: List the programming languages you consider important or interesting. Describe each of them in one sentence.

C	High-level assembler
Lua	Table-based, functional and embeddable
Python	Great RAD language crippled by dogma
Scala	Java done (almost) right, with Erlang- and Haskell-inspired features

Posting this from the new N810 using its built-in keyboard. Typing those tags was painful, and as a result, more languages will be added to the table tomorrow — from the workstation.

Why you should conditionally promise to buy the upcoming Nokia N810 tablet

Thu, 18 Oct 2007 04:08:58 GMT

It looks gorgeous
It runs Linux, and showcases what can be done with more vertical integration
Nokia has been improving their interaction with the developer community
Video camera and Skype (no Skype video support yet, though)
Rhapsody subscription service
New: Now with GPS, spacious internal storage, and sliding keyboard built-in!
New: More video codecs, Flash 9, Mozilla-based browser

So commercial software providers (Skype, Real Networks) will provide Linux ports if they judge that the userbase is big enough. Which is good news.

The same thing applies to Nokia itself, naturally, and sadly in this case, they do not think there is demand for Ogg Vorbis playback.

So if, like me, you find the product attractive, but have a personal collection of Ogg Vorbis files (or FLAC, which transcodes seamlessly to Vorbis), then this is what you can do:

E-mail Nokia about it
Inform outlets that stock the tablet (e.g. Best Buy, CompUSA)
Sign this pledge and pass it around

All the software for the new device (minus GPS — though perhaps it’s the same software that comes with the GPS kit for N800? Oh, and the ambient light sensor) will run on the N800, so holding back won’t be that painful.

Wide Finder: OCaml and JoCaml

Wed, 17 Oct 2007 18:09:40 GMT

Spent last night getting a crash course in using OCaml to do non-functional things (hash tables, file I/O, regular expressions) and the result is now up.

The JoCaml version does the file-partitioning trick used in the C++ implementation, with each finder workers being run inside a JoCaml channel; the channels share a single lock so they can update the hash table serially.

Interestingly, current implementation does not get a speed-up from the input file being cached (Ilmari’s wf.ml does). Will have to peruse his to see what’s slowing things down.

Lesson: not all techniques for processing a file line-wise are equally good!

Wide Finder: C++ update

Tue, 16 Oct 2007 22:09:22 GMT

Talked with a colleague about the slow single-threaded performance of my Wide Finder implementation, and we narrowed it down to two possibilities:

Boost regular expression is not compiled?
C++ strings have higher overhead than null-terminated c_str

First point can be ruled out: Boost compiles regular expressions when you assign them. Second point — well, reading in the file using std::getline turns out to consume the bulk of time.

I’ve reorganized the code a bit, using a multimap rather than a vector to rank the URLs by count, with no effect on speed. With two and four threads on a dual-core Intel notebook, the performance is at least on par with Ruby.

Alastair Rankine has a C++ implementation that is slightly faster, but uses Boost memory-mapped IO that I avoided for the same reason he put as caveat: that it will not scale to files that are too large. Which Tim’s log file might well be. Again, that is not significantly faster than the Ruby code.

Moral of the question: Perl and Ruby can be faster than C++! The C implementations out there are blindingly fast, but the way they do regular expression handling are really painful.

Will turn my (limited) spare time to doing a clean JoCaml implementation — it might not be faster but it definitely will look cleaner!

JoCaml

Tue, 16 Oct 2007 00:11:25 GMT

After turning in the C/C++ monster (cleanest C code I reckon it is possible to write, thus the total lack of memory-mapped I/O and other optimizations), I turned my attention to picking a better implementation language.

Requirements:

Functional
Good support for threading
If possible, support for distributed computing

As it turns out, JoCaml fits the bill perfectly. It’s an extension of Ocaml, so it combines a rich library with a familiar syntax (not to me, but having used both Scheme and Haskell, how different can it be) — and a very nice process calculus!

Example: this is a concurrent stack that blocks if there is no input available

let new_stack () =
def state (s) & push (v) = state (v::s) & reply to push
or state (x::s) & pop () = state s & reply x to pop in
spawn state([]);
pop, push
;;

This defines a private state channel, and then export the pop and push synchronous channels (that to the user behave just like ordinary functions)

and this is how you use it:

let pop, push = new_stack ();;
spawn echo(pop());;
push(1);;

Note that the echo channel will block, since pop can’t return a value until the stack contains something! This value is then pushed into the stack and ‘1′ printed.

More of this at the JoCaml site. And, as it turns out, there already is a JoCaml implementation of the Wide Finder, by Ilmari Heikkinen. Will have to grok the finer details from him.