Older blog entries for mikehearn (starting at number 8)

autopackage

So the project is growing. Lots and lots of nice comments on OSNews about the 0.3 release, which is really great. It's cool that people are so enthusiastic about the work we're doing.

The next release, 0.3.5, is time based. That's not because we're moving to time based releases, it's because in October I go to Durham university, so it'd be nice to kick out another release before then. It's mostly focussed on bugfixes and stability improvements (well, we so far do not crash, but by stability I mean working well on all distros).

There's still a long way to go, but at least the path is clear.

Wine

winecfg continues to make good progress. A lot of the work has involved cleaning up the codebase (which was a bit yucky, imho, hungarian notation all over the place, a wierd mix of styles etc), and rearchitecting it to be transactional. That means we can do instant apply Gnome2 style, which Alexandre seems to like. Unfortunately Win32 does not have great support for this style of config dialog, so we might have to add a few Wine extensions.

While we're at it, we're cleaning out some crack prefs. When even the developers don't seem to know what a preference does, it's time to worry :)

The most frustrating thing about Wine is that everywhere I turn, I find new things I want to do. Theming, making all my apps work, fixing InstallShield, merging DCOM, winecfg, better standards compliance, better desktop integration! I want to do it ALL but don't have enough time or energy (of course).

Wine

I've had pretty good productivity lately. In fact, I currently have 8 outstanding patches against Wine that haven't been committed yet. Alexandre has been busy with rearranging the 16/32 bit code, amongst other things, and commits from external contributors have slowed down a lot. I find myself wondering, not for the first time, if Wine would not proceed quicker with more committers. OTOH the Q&A Alexandre provides is certainly a benefit, especially for people like me who aren't hugely experienced with Win32, nor writing thread safe, OOM-safe code in C.

I've been mostly hacking on winecfg - the Wine projects official configuration GUI. The eventual plan is to scrap the config file, and move it all into the registry. In fact, this is a goal for the 0.9 "easy to use" release of Wine. Win32 isn't the easiest toolkit to work with, and it becomes doubly harder when you don't have an IDE to write the gobs of template code it needs. As a result, progress is not as fast as I'd like. Still, the end result should be good.

I started looking at MSN Messenger last night. If worst comes to worst, and Gaim and Jabber are forced off the network (I doubt that'll happen), then we might have a backup plan in being able to run MSN Messenger itself. Another interesting possibility is a build of Gaim that is a WineLib app and uses the MSN ActiveX control directly. More likely however, is that the Gaim team come to an agreement with Microsoft, or the plugin becomes "contraband" software like the DeCSS and MP3 plugins - we'll just shove it into the freeworld repos.

autopackage

We're getting ready for the 0.3 release of autopackage. This is the first release to feature a GUI, and though it's still incomplete I'm pretty pleased with how polished it's feeling. We've put a lot of work into thinking through how users are going to interact with software installation on Linux in the future - the long term goal is that packages can be dragged out of the webpage, direct onto your panel and all the complex machinery of package management takes place behind the users back, entirely automatically.

We hope that if we do some day build such a system, it'll be generic, ie not tied to autopackage.

Some of the spit-n-polish work we put into the autopackage GTK front end will make its way back to GTK eventually. For instance, we have developed a few patches for EggStatusIcon. Whereas before it blinked harshly on and off, it now gently pulses. Small details make big differences. Expect more such transition-animation work in the near future.

We've also got code now that will automatically download the autopackage support code and GUI if they aren't present on the users system. This works surprisingly well, and is now the preferred way for people to install autopackage (unless they are a developer). Our download times are currently satisfactory - about 8 seconds on dialup for the core code, and about 15 seconds for the GTK gui, meaning a total of about 20-25 seconds delay for dialup users. For ADSL/cable users, it's probably nearly instant. Those times will go up as we add more code, which might start getting problematic - it could be worth learning about optimizing code for space, though I doubt there's much we can do that will make a big difference.

That code was written not by me, but by Hongli and Curtis. I haven't said this before, but I feel I should - they are consistantly smart, helpful, reliable contributors who produce work of the highest quality. They're the sort of contributor every project maintainer wants about a hundred of, and I'm very lucky to have two. Still - the task is pretty big, even for three of us working on it as our primary projects. Our last release was on the 12th July.

bgeiger: I suspect if you have an even slightly modern x86 based PC and actually performed some speed tests, you'd find that the fact that a Mac feels fast doesn't mean it is fast.

Now feeling fast is a mighty spiffy trick, and mostly boils down to the way they double buffer the windows and use VSYNC to good effect.

To be honest, I've used MacOS X and wasn't impressed. It's great demoware, and a good OS, but nothing we can't beat the living snot out of in free software land given a bit of work. Anyway, unless you will enjoy writing Yet-Another-Cocoa-Instant-Messenger, there's not much to do there except wait and see what Apple does next. Hardly thrilling stuff.

Inspired by the ever interesting Chris Brumme, who blogs for Microsoft on .NET and Windows technical matters, I thought I'd steer this blog more towards technical writings on autopackage and Wine.

Though I do have a personal journal elsewhere, I haven't posted to it for a while, partly because now I'm back at home I'm just a phone call away from all my friends, so we have less need of it to keep in touch, and partly because since my hostelling trip around Europe ended unhappily for various reasons, not much is happening around here.

If you enjoy these mini-articles then please let me know, and if there's something you'd like me to write about, ditto.

Wine and InstallShield

One of the problems that has plagued Wine for a very long time now is (perceived) unreliability. Many people fail to get Wine working with their favourite apps, or even any apps at all. Part of the problem is that we recommend you use a fake windows drive, then install your applications into it, though as far as I know this recommendation is actually buried deep in the users guide.

The reasons for that are pretty simple - typically Windows installations have huge piles of rubbish that get in the way and can confuse Wine, not to mention that it stresses components that generally don't get much testing, like the code which reads and writes to the binary registry format. In the age of NTFS filing systems, the lack of writeability can likewise cause difficulties.

Unfortunately many people fall at the first hurdle, namely installing apps. Thanks to the near total lack of package management on Windows, there is a glut of installers all of which have different behaviour, and many of which are very complex under the hood. InstallShield falls into this category, and is a program that has given Wine problems for many years now. The difficulties with InstallShield basically fall into two areas:

1) Window management
2) DCOM

Window management has long been an achilles heel of Wine, as Win32 window management is close enough to X11 to make a mapping worthwhile and valuable, but different enough to cause serious problems in certain scenarios. The main difficulty lies in the location of the window management code.

In Windows, the application is responsible for performing its own management, or to be accurate it delegates via the default window procedure (message handling code) to libraries provided by Windows. Those libraries draw the window decorations, provide the default move/resize behaviour and so on. That simplifies the architecture somewhat, and means that opaque resizes don't suffer as much from loss of sync between the border and the contents (a problem which Havoc Pennington has been writing a patch for in metacity) but also means that if the app freezes, the window becomes impossible to move, minimize or close. In fact in XP the kernel has a hack for this, such that if a window stops responding it's silently replaced with a faked OS level window so you can still move it and resize it.

In contrast, on X11 based systems like Linux, the window management is performed on behalf of the application by a WM, a dedicated application. This program draws the borders for the app, controls its position, which buttons are available and many other things.

Typically, this distinction does not cause problems, but some apps take advantage of the implementation details. WinAmp for instance, actually creates a standard window then overdraws its own window borders with the skinned versions. Clearly this isn't possible in X11 so for times like this Wine has an "unmanaged mode", where the window is disconnected from the WM and Wine performs all its own management, which is closer to how Windows itself operates.

Unfortunately, that approach has several problems as well. For starters, the windows don't respect the virtual desktop you're on, nor z-ordering. This means that unmanaged Wine windows seem to be "nailed" to the screen, often making the app unusable in the real world.

The problems with InstallShield arise from the fact that typically, for reasons of intense LSD tripping on behalf of the developers, these installers take over your screen with a nice big blue background. Unfortunately, Wine cannot currently make managed full screen windows, as the protocols for that have only recently been agreed upon, so it defaults back to unmanaged mode. We now have a problem - with no real window management in use, the actual installer windows it opens on top of the blue background will appear underneath the background window, and worse there is no way to raise it to the top.

Because of this, for some time Wine has shipped in its default config file AppDefault entries which force all known installshield executables (they have seemingly random names) into "desktop mode". This, in theory, creates a window which contains the virtual Wine desktop, and means that Wine can accurately control and render the Windows without X interfering. Unfortunately for architectural reasons, desktop mode doesn't operate as you might think - each window gets its own desktop.

Until recently this wasn't a problem, but a bugfix elsewhere in the x11 driver now causes Wine to die with an X11 Client error when it attempts to shift multiple windows in desktop mode around. This problem renders InstallShield effectively unrunnable. There are two possible solutions to this problem - to fix desktop mode so all windows appear in the same desktop (which is what you'd expect), and to implement "half managed" mode, which is where the WM controls the window, but Wine is responsible for its position and border rendering. XMMS does something like this, and it works OK. Eventually both approaches will need to be implemented.

This leads to the second problem with InstallShield, namely its extensive usage of DCOM and RPC. DCOM, or the "Distributed Component Object Model", is the rough analogue of CORBA on Microsoft systems. It's been around for a very long time now, and is used by many apps that you would never even suspect. If you've seen a Windows 2000/XP box that's been hit by Blaster, you'll know how vital it is that RPC and DCOM are functioning correctly at all times, even seemingly unrelated things like copy and paste rely on it. That's why XP will reboot in 30 seconds if the RPC daemon terminates.

InstallShield uses DCOM primarily for inter-thread communication. While CORBA is intended for out of process communications, distributed across networks, DCOM is intended also for invoking object methods within a process. The process is fairly simple - you create a COM object that has interfaces, then marshal one of those interfaces into a stream (or alternatively, the global interface table, an object I partially implemented in Wine a few months ago). You then pass that stream to another thread, which unmarshals it. Behind the scenes, DCOM constructs a proxy/stub pair that switch the method call into the first thread context.

Exactly how it does that is a topic for another journal entry, but until recently Wine did not implement this technology. That caused InstallShield to have problems, especially with correctly redrawing its windows, although enough of the OLE/COM architecture was present to enable it to basically work. TransGaming developed a patch for WineX which implemented this part of DCOM, and thanks to the votes of its subscribers this work is now available to WineHQ. Part of my work that I talked about in my first journal entries was merging this work into the (significantly different to winex) WineHQ tree. That merge still isn't complete, but hopefully I'll get it done before going to uni in October.

At that point, the final missing piece of the DCOM puzzle will be the ability to generate binary TLB files using Wines IDL compiler, as Alexandre deemed it unacceptable to take Microsofts version (a large binary file) and stick it into CVS. Wine will then have enough infrastructure to deal with even complex usage of DCOM and OLE.

That's important because many programs use it behind the scenes, not just InstallShield. For instance, I worked on an app that embedded Internet Explorer in Java, then programmed it from another thread. That involves cross-language, cross-apartment method invocations: not a trivial thing to do, as it involves late-bound method calls via IDispatch (where you pass the numeric ID of the method and a variant array of parameters to the IDispatch::Invoke method). Sounds simple, in practice IDispatch is so hard to implement that Microsoft strongly discourage you from trying, you're told to rely on the implementations they provide instead.

Until the work is complete however, you can continue to install Microsofts OLE implementation by running the DCOM98 executable with a recent version of Wine. Make sure your config file is up to date. Fortunately this approach works reasonably well, although as the DCOM updates could be pulled at any time, and cannot be redistributed, it's important that Wine gets its own implementation. You need to run apps like this:

"wine --dll ole32,oleaut32,rpcrt4=n foobar.exe"

Life

The first week of my holidays have been pretty hectic as I try and get everything together for travelling around Europe for a month with Ken (an old school friend of mine). I leave on Monday, should be good fun! I've travelled before, but not for such a long time.

Geek

The server (theoretic.com) that I share with a friend has started acting up again. Apache eats all the memory, and I don't know why. Originally I thought it was mod_perl, so I removed that, and it seemed to fix it, but a few days ago it died again (linux sucks at dealing with apps that eat all the memory).

I'm going to try and move autopackage to sunsite.dk, which has proper hosting. I applied earlier, we'll see what the results are. Doesn't solve my own problem though :(

Code

autopackage has been bobbing along nicely, lots of arguments between me and Hongli about the GUI design of the gtk installer frontend. There's still a lot of work to do, but I think the end result is going to be nice.

One problem we have is the lack of a decent animation format. MNG is too heavy, and doesn't fit into the gdk-pixbuf API well (it's more akin to flash than gif). I've been toying with the idea of hacking up a super simple APNG format or something that maybe could get better adoption. I might get time after I get back from Europe.

OTOH I also have a metric ton of other things to do, like trying to reduce my differential against WineHQ CVS to something sane again, doing some of the boring under-the-covers work on autopackage and so on. So, like most things, it will probably suffer from not floating to the top of the TODO list for a while.

My (ex)boss just emailed me, and wanted to know why the app I ported using Wine won't open the apps associated with a given file from Internet Explorer. Theoretically adding code to do this is not so hard, the main problem of course is the lack of a solid freedesktop standard for it. There's one in the works, but who knows how quickly it will be deployed.

A close shave

That's what I call it when you finish your project at work 2 hours before you're meant to leave. Fortunately, at the end it seemed stable, though not quite as fast as I'd like.

So - I did it. I won. I didn't think I would, and I almost didn't, but I'm getting fast enough at debugging crashes, deadlocks and other nasty bugbears in Wine that I wasn't thrown when at the last minute I raised the fd limit and found it crashed in yet another way :)

Hopefully I'll be able to do a patch dump next week, so others can benefit from my work even if I don't get a chance to merge it in properly.

Life

Tonight I have to pack - I leave for home tomorrow. I've been putting it off for a while, and now it's 7pm and I've still not started. As you can tell, I'm still putting it off.

Play

Funny how work and play interlocked towards the end. I tried out apbuild today, which is a program that lets you compile the source in such a way that it will run on older glibcs. It's still rather untested, but can do Xlib and Wine (with a bit of encouragement, Wine is hard due to the games it plays with pthreads).

Of course, just after I imported it to CVS (very raw), Hongli discovers an obscure option to ld intended for platforms like BeOS that would let us get around the problems with using the LSB stub libs! ARGH! Well I need to evaluate them to see which is best, now.

We have GUIs for autopackage! Very pretty they are too, one of the nicer installer UIs I've seen for a while. Hongli (foobarwidget) is a really great coder :) There are screenshots in my home directory if you're interested, but I can't be bothered linking right now, as it's all changing so fast. I reckon people will be impressed when we do the next release though.

29 Jul 2003 (updated 30 Jul 2003 at 12:46 UTC) »
elanthis: Actually, I can't agree that compile times are a problem in such systems. My experience of them has been that such an approach massively improves compile times, because the compiler doesn't have to parse huge volumes of header file. Typically, these languages don't have a preprocessor which can speed things up a lot.

For instance, Object Pascal uses the module approach, except that logically the file is divided into "interface" and "implementation". It has no preprocessor of any real worth (but that doesn't seem to hurt the language much), and is lightning fast. If you are fixing syntax errors, it's quicker to simply hit ctrl-f9 in the delphi IDE to recompile than move your hands from the keyboard to the mouse and back again to move to the next bug.

Part of the reason it can be so quick, is that (a) Borland make very fast compilers and (b) the Object Pascal language is conducive to fast compiles. The built in dependency tracking usually works very well - there are no Makefiles, it (spew) Just Works.

There are limitations of course. You cannot tie the interfaces of two units to each other (ie if you declare class A in unit UA.pas and class B in UB.pas you cannot have class B have a class A as a member and also vice-versa). That's a "circular unit reference" and is illegal. It can be very annoying at times, but typically if that occurs it reflects some problem in the design of your code anyway, so might as well do some refactoring while you are there.

Life

Friday was our farewell do here. I've been living in Great Malvern for about a year now, at this gap year job working for an R&D firm. We're all starting to drift away now the year is almost over (I leave in a week), so last Friday for our goodbye night seemed like a good choice.

It was a lot of fun. We went to a very nice Italian restaurant and took over most of it (there were about 20 of us), then off to the pub for a bit, then some of us went to Tramps which is a rather dire but nonetheless enjoyable nightclub in Worcester. Needless to say, I slept in the following day :)

Work

Well, the app I'm porting using Wine no longer crashes, or hangs. We're so close! Unfortunately, typing in the page editor is far too slow, the cursor lags behind heavily and what you type isn't rendered until you stop typing. I have no idea where the bottleneck might be. Time to investigate profiling tools. For the first time, I'm starting to feel confident about making it work before I leave.

Once I've left, I have a couple of weeks before me and Ken go travelling around Europe, hopefully I'll at least be able to manage a patch dump of my work, if not actually being able to merge it all in.

Play

Work has started on the GUI front end to autopackage. You can see actual screenshots here and here. There's a mockup of a different UI design that we're going to try here.

I haven't had much time to work on this lately, but fortunately there are several really hard working contributors now, which is fantastic. I'm psyched about this project, we have a clear direction and goals, and I think we're going to make them, even though they are pretty ambitious.

So, this is my first entry on advogato. I've been vaguely aware of the site for some time, but seeing the GNOME guys blog here made me think maybe I should too. I share a non-geeky (mostly) blog with some friends, but one devoted to my free software work seems fun.

Wine

I join this site at an interesting time - I'm engaged in a race against the clock. I have precisely 1 week, and 1 day left to make my companies app (a groupware product in development) work completely under Wine before my contract is up and I leave. If it doesn't work before I go, the porting effort will be abandoned, as I'm the only one there who can do it.

I've been working on Wine for about a year now, no real area of specialisation yet. I've worked on the common controls library, core window handling, I wrote the XEMBED/freedesktop system tray integration code (currently working its way through the peer review process), and at the moment I'm battling with COM/OLE. Hopefully the results of my work (which builds on work done by Ove at TransGaming) will soon be meaning a major COM/OLE upgrade to Wine.

Seeing an opportunity to get paid to hack on Wine full time was great, and I've been having a fun time for the past few months. As I reach the end though, I'm getting pretty tired, both mentally and physically - the areas of Wine/Win32 that I'm currently working in are poorly documented, partly incomplete and deal with parts of Windows that are old, rickety and generally massively baroque, even moreso than usual.

Stay tuned to find out who wins - Microsoft, or free software. It cannot be both :/

What else do I do?

I am the principal designer, maintainer and author of autopackage. It makes packages. They are kind of unique, in that they are distribution neutral, can do interactivity etc. If people are interested, I'm going to blog more on autopackage, how it works and why it's so important in future. I noticed there was an articles system, perhaps I'll submit something there in future.

OK - compile over. I have to get back to work, no time to lose. Rock on my friends.

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!