Recent blog entries for salimma

In my comments on the recent LWN seed “Will LSB 4 Standardize Linux?”, I made the argument in favour of application bundles:

The advantage of bundles is that it contains more metadata than a simple tarball.
Compatibility problems might arise, but at least the application launcher could provide more
meaningful feedback to the user.

Or even integrate with the OS’ version control system, especially now that Linux has
PackageKit, to say “hey, the user wants to run this new bundle that needs libfoo-x.y, install
whatever package is necessary to provide that”.

You’d need a package management system that automatically scans built packages for what
libraries they provide; RPM does that, not sure whether DPKG does.

To provide the context: the discussion was on how application vendors can easily target LSB 4 with a single binary image. My post was in direct response to an argument by another reader, that complicated schemes do not add much real value over tarballs, to which I begged to differ, arguing that the metadata available in bundles make the integration of binary applications much easier.

The issue of binary distribution triggers an allergic reaction from some people in the FLOSS community, a reaction that is, in my opinion, rather unwarranted. Even Debian provides, in their non-free repositories, stub packages that will download binary packages and create a standard .DEB package out of them. There are clear advantages to making binary-only applications more well-behaved, in fact the same argument for having package management systems with graph-based dependency tracking in the first place: dependency, dependency, dependency. When installing/upgrading a package, you’d want all its dependencies to be pulled in automatically. When upgrading a library, you want to make sure that all its dependents will still work. When there is a security vulnerability, you want a non-techie end-user to be notified, preferably within a fixed period of the vulnerability being made public (through periodic updates), or the next time the user launches the application concerned.

There have been attempts to create a one-size-fits-all universal package format, that’s distribution-independent and vendor-friendly. This is a red herring, IMHO, for the same reason that the Unix market splintered in the ’70s and ’80s, and that we have a proliferation of Linux distributions — and multiple independent BSD operating systems, each of them with their own ports tree (DragonFly being an exception in that they share NetBSD’s pkgsrc system). It’s nice to control your own packaging format, or if it’s a shared format (like RPM is), to control the naming conventions, etc.

What application bundles can do is provide the best of both worlds: vendors can ship binary-only bundles that declare dependencies in a least-common denominator format that the LSB can standardize, for example:

<br /><Provides>
  <lib>libbaz-a.b</lib>
</Provides>
<Requires>
  <lsb-version>4.0</lsb-version>
  <bin>convert</bin>
  <lib>libfoo-x.y</lib>
  <lib>libbar-z.w</lib>
</Requires>

The first time the bundle is launched, the launcher can add it to its index of available bundles. If any of the dependencies are missing, the system-native package management (or a meta management infrastructure such as PackageKit) is triggered to install the missing dependencies. The bundles themselves can be placed anywhere (though library bundles — in NextStep/OpenStep/OS X parlance, “frameworks” should probably be placed in pre-determined paths, e.g. /Library/Frameworks, /System/Library/Frameworks and ~/Library/Frameworks).

The only problem is that the system-provided libraries might not be ABI-compliant with the specified LSB standard, for example, libraries written in C++ after a compiler ABI change. There would probably be a need for the native packages to declare their compliance, or non-compliance, with LSB standards.

And one last nice thing about bundles: fat binaries. It’s easy to provide multi-arch bundles, and stripping away unwanted architectures is a simple rm operation.

Daniel Béranger has raised the ABI issue surrounding Fedora and RHEL’s recent upgrade to Firefox 3. In short, RHEL 5.2 ships with Firefox updated to the new xulrunner-based Firefox 3, but its Eclipse and libswt3-gtk2 is still at 3.2, which depends on the old gtkmozembed interface.

This seems like a good argument in favour of adopting Debian-style package naming, when it comes to libraries: append the ABI version to the (sub)package name containing libraries that are linked to from other packages. So the old Firefox 1.5 would have a libgtkmozembed18 subpackage, that can be shipped with RHEL 5.2 without shipping the rest of Firefox 1.5 (technically speaking, the ABI version is 1.8.x.y, as prior to XULRunner 1.9 the ABI is ever-changing, but packages already handle this by depending on the specific version and release number anyway)

This is already done from time to time in the RHEL/Fedora world, in the form of compatibility packages, but making it the default would avoid this kind of breakage, where a package /has/ to be updated (due to upstream EOL) but parts of it are needed downstream.

Tim Bray’s revised Wide Finder project [ongoing.org] has been ongoing for a few weeks now, and I’ve finally took the time to design and prototype an implementation.

What
The goal is to evaluate the performance of middle-of-the-road, not embarrassingly parralelizable tasks on modern-day multi-core hardware. Such as the Sun T2000 servers. Fittingly, the task is to parse a multi-gigabyte web server log file and compile some aggregate statistics.

Design
The solution I came up with for the earlier iteration of the contest, coded in different versions (C++, OCaml and JoCaml) is fundamentally sound, though rather unoptimized (picking up two-and-a-half different languages in one weekend is a good way to find out how much there is to know about, say, C++ stream buffering). With the benefit of hindsight, and given that we are several weeks into the project and there are strong implementations already [wikis.sun.com], the idea is to find an unexplored niche.

Short recap of the main implementations:

• OCaml: Fernandez is ahead of the pack again, the only solution in the 7 minutes
• make+C+awk+sh: Perl is dead, but shell scripting is enjoying a renaissance with parallelizable tasks. 8 minutes
• Java-based solutions: in the 13-17 minutes range are the various JVM solutions, from Java, Groovy and Scala to Fan, an interesting Ruby-like language for the JVM. Reminds me of .NET’s Boo.
• Python, Ruby: in the 20+ minutes range. Python multiprocessing is not that efficient yet; I believe an improved Stackless Python solution might be forthcoming

I dabbled with a Common Lisp solution; it works and appears to be competitive, when tried on a partial log file. Exploring the available options for parallelism, however, revealed the disconcerting fact: no freely-available Common Lisp compilers have good multi-threading, or even multi-processing (without shared memory) on Solaris! Even worse, the SBCL incompatibility with GCC 4.3 means that even the Linux version on my Fedora machine is several months old, and does not have the threading library.

So it’s back to Java. Perusing the blogs of the Java and Scala programmers, it appears that the common complain is .. regular expressions. So the hunt was on for a good regular expression library. Joni, a port of the Ruby Oniguruma regex library to the JVM used by the JRuby project, appears ideal: low-level and supposedly very fast. Until one hits the total lack of documentation. So that’s off the table. Ended up using dk.brics.automaton, which appears to perform well enough, even when parsing Unicode strings.

The nice thing about using Java is that, if you hit a performance brick wall, chances are that many other people have been there before you. The problem I have, the need to have random access within a file (so different threads can start at different offsets (Java’s RandomAccessFile is good for this) combined with the need for buffered I/O (BufferedReader is good, but there is no RandomAccessReader !) is solved by the nice folks at Biojava.com. Great!

On my system (2 GHz Core 2, 2 GB RAM, 5400 rpm HDD, Fedora 9 x64), Ruby takes about 2.2 seconds, while my Java implementation running on OpenJDK 1.6 (64-bit) takes about 1.6-1.7 seconds with 1 thread and 1.4-1.5 seconds with 2 threads. Close to the 1.2+ seconds time that Alex reported for Python, but hey, we’re paying the Java start-up cost here.

Will update when I get an account on the test server. 40GB dataset, here I come! In the meantime, time to look for opportunities to use a JVM-based language better suited for the task. The Java code is a tad bit verbose.

I tend to wipe off my old Linux setup and reinstall everytime a new release comes out — not that it is necessary, but it’s a good way to get rid of old cruft that have gotten installed, found not useful enough, and then forgotten (and there are lots of those, as df -u shows).

It also makes it easier to ascertain the state of Fedora — any (mis-)configurations that I might have corrected would be reset to the default.

So, now that Fedora 9 has been released, what needs to be added to / changed from the base setup? Surprisingly, not that many:

Compositing
Some people swear by Compiz; I personally find Metacity much more usable (Compiz does not support cycling through all windows of a given application — Ctrl+F6 in Metacity; Cmd+~ in OS X). Metacity now has a compositing manager that’s turned off by default; turning it on involves either using gconftool-2 (only for advanced users) or gconf-editor, and setting the /apps/metacity/general/compositing_manager key to true.

The support in the stable version is a bit flaky still; the metacity package in Rawhide is much better behaved and appears quite stable. Upgrade by issuing yum –enablerepo=rawhide update metacity. As of the moment it does not pull in any other Rawhide package so you can rest easy.

Try pressing the volume up/down/mute keys on your keyboard (if you don’t have a multimedia keyboard, change the bindings in System->Preferences->Personal->Keyboard Shortcuts) and be amazed at the translucency coolness (no, this is not bling). The brightness pop-up windows have not been changed yet, alas.

Firefox
Ever cursed Firefox’s font rendering in silence? Type about:config in the address bar, and add the following boolean keys:

font.FreeType2.autohinted = true
font.FreeType2.enable = true

Keyboard
For the English-speakers among us specifically, and those who use the US keyboard layout in general (it’s the standard layout in Indonesia, for instance), the occasional times when one has to type an accented character is rather annoying.

There are various work-arounds — launch the character map (under Accessories), add the Character Palette applet to the panel (so that it consumes RAM even when you don’t use it!)…*or* you can just fix your keyboard layout. The die-hard command-line junkie would be able to tell you what option to pass to setxkbmap to achieve this. The rest of us can just use System->Preferences->Hardware->Keyboard. In the “Layouts” tab, select “Layout Options”. The option you want is “Compose key position”; I use Right Alt, but Caps Lock haters will rejoice to know that, yes, you can use that dreaded key as your compose key as well. To type an accented character, now the only thing you need to do is hit the Compose+accent followed by the letter you want to accent (using shift as necessary, e.g. for ^).

While you’re here, you might want to change the Alt/Win key behavior, and map either Meta, Super or Hyper to one of your Win-keys. The GNOME default is inexplicably for the Win-key to be a normal key and not a modifier (so it cannot be combined with other keys).

Coming up: Applications
Et voilà! You should have a nice-looking, and more importantly, functional desktop right now. In the next instalment, I’ll comment on the applications I use. Until then, à bientôt!

Task: List the programming languages you consider important or interesting. Describe each of them in one sentence.

C	High-level assembler
Lua	Table-based, functional and embeddable
Python	Great RAD language crippled by dogma
Scala	Java done (almost) right, with Erlang- and Haskell-inspired features

Posting this from the new N810 using its built-in keyboard. Typing those tags was painful, and as a result, more languages will be added to the table tomorrow — from the workstation.

• It looks gorgeous
• It runs Linux, and showcases what can be done with more vertical integration
• Nokia has been improving their interaction with the developer community
• Video camera and Skype (no Skype video support yet, though)
• Rhapsody subscription service
• New: Now with GPS, spacious internal storage, and sliding keyboard built-in!
• New: More video codecs, Flash 9, Mozilla-based browser

So commercial software providers (Skype, Real Networks) will provide Linux ports if they judge that the userbase is big enough. Which is good news.

The same thing applies to Nokia itself, naturally, and sadly in this case, they do not think there is demand for Ogg Vorbis playback.

So if, like me, you find the product attractive, but have a personal collection of Ogg Vorbis files (or FLAC, which transcodes seamlessly to Vorbis), then this is what you can do:

• E-mail Nokia about it
• Inform outlets that stock the tablet (e.g. Best Buy, CompUSA)
• Sign this pledge and pass it around

All the software for the new device (minus GPS — though perhaps it’s the same software that comes with the GPS kit for N800? Oh, and the ambient light sensor) will run on the N800, so holding back won’t be that painful.

Spent last night getting a crash course in using OCaml to do non-functional things (hash tables, file I/O, regular expressions) and the result is now up.

The JoCaml version does the file-partitioning trick used in the C++ implementation, with each finder workers being run inside a JoCaml channel; the channels share a single lock so they can update the hash table serially.

Interestingly, current implementation does not get a speed-up from the input file being cached (Ilmari’s wf.ml does). Will have to peruse his to see what’s slowing things down.

Lesson: not all techniques for processing a file line-wise are equally good!