Older blog entries for DV (starting at number 230)

Ouch, it seems I didn't wrote any blog entry for nearly 2 years, since our wedding. Well I would say that I have been fairly busy :-) !

Baby Cheng 程

Our son is born end of last august, so obviously this tends to keep the parents busy, but it's such a pleasure too !

libvirt project

this has been a wild ride ! The project has grown a lot, in community and contributor size, as well as code and features during the last couple of years. It acquired support for most well known virtualization hypervisors (QEmu/KVM, Xen, ESX, VitualBox, ...) the only notable exception being Microsoft Hyper-V I think. Our latest release is 0.8.0 so we are not at 1.0 yet, but we are really getting closer, for example this release now adds snapshotting support which was a big feature request.

About updates to Fedora

There was a discussion last month about forbidding direct built to the stable branches, and I really think this makes sense, except for security erratas, one really should not push directly an updated build onto the users without some testing. But on the other hand we entered beta on Fedora 13 and apparently it's too late to push libvirt latest release and features, meaning users who don't try to update either via the virt-preview special repository or by rebuilding from the source rpm, will see libvirt features when Fedora 14 is being released i.e around the end of the year. I see people bouncing between both extreme of the features vs. stability tradeoff, and I wonder if there isn't a better way than building absolute rules. Maybe a general profile per machine where the user indicates whether he want updates, just bug fixes or just security errata, and let him do the picking. This would require some policy from package builders, and some changes at the repo level I guess, but having the user ultimately in control might be worth it.

A new release: libvirt-0.4.3

After two months a lot of patches had accumulated, including a lot of improvements for Xenner and Linux container support. But this release brings a massive set of code cleanup, and just looking at the patches there there is a lot of obscure case failures which should now be properly handled (or at least better handled, like out of memory situations). I'm pushing testing updates for F-8 and F-9 if you have time and use virtualisation please review them, thanks !

A new name ?

I got married to Miss Wei LI last saturday, it was kept a simple family event with just a few close friends, everything went well except for a bit of rain ! For the name I wouldn't mind being called Mr Li, but it's probably not very practical at this point (ah and good luck getting the li.com domain, and I guess hijacking li.org would not be well accepted either ;-)

A very simple picture, people interested for more should know where to look for already. Oh it also made me fight with the Panasonic HDC-SD9 'new' MPEG-4 output, to get videos to render properly on that other OS, I didn't expect to learn so much about video format so quickly. I will post recipe and scripts later.

Releases

I pushed a bunch of releases on Tuesday, trying to catch the Fedora-9 train (I nearly missed it, it led to a not so fun curl_is_failing_to_upload debug session which led to nss3 for firefox3 is not compatible with nss3 for fedora8 curl), thanks to everybody who helped catch that train !

The releases are mostly bugfixes, libvirt-0.4.2 leading the pack, but libxml2-2.6.23 has a lot of fixes too thanks to various people reporting bug and giving patches, notably the Huawei team. Libxslt-1.1.23 includes the dozen or so fixes since last summer.

Developments

Clearly libxml2 and libxslt are in maintance mode, the focus is on libvirt, maybe I will just add support for the latest Proposed Recommendation of XML-1.0 in libxml2 before the Summer.

For libvirt, clearly we need to extend the number of hypervisor supported, maybe update and clean up the OpenVZ support too. IBM is actively contributing the Linux Container driver, I just commited a second set of patches today, you can expect good support in Fedora 10 I guess. On the high end side Sun just posted the patches for the lDOM virtualization on their Niagara based machines, lot of patch reviews those days. I also want to get a complete set of bindings for Java integrated, and now that Fedora java packaging guidelines are out, this is a good opportunity to add this.

History meme

that one is interesting, here is my contribution, as you can see I'm an old fashionned old fart, main workstation at home:

paphio:~ -> history | awk '{a[$2]++ } END{for(i in
a){print a[i] " " i}}'|sort -rn|head
319 vi
257 ssh
255 cd
156 cvs
130 make
125 ls
79 svn
60 scp
48 su
43 ping
paphio:~ ->

and on my second workstation in Annemasse:

wei:~ -> history | awk '{a[$2]++ } END{for(i in
a){print a[i] " " i}}'|sort -rn|head
362 vi
263 cd
262 make
136 cvs
115 svn
105 ssh
78 ls
67 scp
40 xmllint
38 grep
wei:~ ->

Happy new Year

yeah it's either really late or a bit too early for Chinese New year, proof that I should blog more often !

backlog

I have been away for the last month and a half, first vacation in China, and then in Paris for Solution Linux expo. As a result an awful lot of mail piled up in various folders that I didn't really had time to process, I am slowly trying to go though them, so if you get replies from a mail in November that's normal, or if you didn't get a reply, it's probably a good idea to ask again !

libvirt and Co.

libvirt development is progressing fast thanks to DanB, Rich, Jim and the many people porting, testing and using it. I really hope we will soon have easy support for most platform - including Windows and OS-X - at least as a client to remotely access the virtualization servers, Rich did most of the work. The development of the CIM provider seems to be progressing nicely thanks to DanS and the IBM'ers :-). During last week Solution Linux i got asked a few times about P2V tools i.e. tools allowing to automatically save the state of a server to allow to start it as a virtualized domain, Rich just released virt-p2v 0.9.1, it is still experimental, what it needs now is a wider range of testing to find and remove limitations, give it a try and provide feedback, thanks !

For libvirt core we still have a number of pending extensions, like support for new hypervisors, adding APIs for storage management which have been worked on by DanB, we also need to finalize a good XML scheme for 'partition' based virtualization, probably update OpenVZ, and add sound and more USB support. I will post on the list soon to try to clarify the roadmap, much work ahead !

Solution Linux

SL 08 was last week, it's the Linux event of the year where I'm sure to go, it was good seeing people around the Fedora and Gnome booth, and of course Red Hat one. My presentation on virtualization went well, even if it was more than 2 hour long, the PDF is available and a lot of people seems to have fetched it already, I assume it's a positive feedback from the audience :-)

China

I was for a month in China, what a blast! It's impossible to try to describe in just a few lines, but we went from the very cold Dalian and Beijing to tropical Xishuanbanna, most of the time within the chinese family and environment, picking only a few selected tourist spot. Things like Shanghai museum, the Great Wall or the Forbidden city are definitely must-go, but to me the most amazing was to stay and live with the chinese inside the family. The cultural gap is huge, there is so much to learn, too bad mandarin is so hard to an occidental brain (and ears), very challenging, but also very fun and enjoyable !

coral bleach

ncm, I think this may be an error of interpretation, the coral turns white when it dies, I mean when the organism dies. They form a symbiosis with alguaes and those give the color, white coral appears when the animal dies, you just have the skeletton left. White coral (with a few exceptions which are naturally white) is just like the bones left from a dead animal, it may result from increased acidity of water, but also from any other deadly condition. For example when overheating, if water temp reaches around 42 degrees, the coral rejects the alguae then dies, it can be quite dramatic because suddenly very large areas whithen suddenly, it's all dead and will take maybe half a century to get back in proper shape :-(

libvirt progresses

libvirt development progresses rapidly, as the traffic on the list can testify, we now have native secure remote operations, and there is ongoing work to add OpenVZ support. I hope to increase coverage of more virtualization engine, so far it looks good, we don't seems to have serious API design problems.

gamin

I made a new release 0.1.9 last week with some bug fixes and portability improvements.

One year in review

I didn't blogged for a year precisely. No I didn't hide in a hole or something but I just didn't really feel the need. I still did a few things:

• the libvirt project is my main work item, 15 releases in the last year, now supports QEmu and KVM, many contributors and well people are using it, so that's good.
• the libxml2 and libxslt libraries have seen their development slowing down, partly because they are mature enough, partly because I have less time, and Kasimier one of the main contributor lost his job and left the project. But Bill is still around and helps a lot !
• travelling, I had the opportunity to travel for work and also for vacations, went to Singapore twice, also Malaysia, had some fantastic time in Borneo, visited Bangkok, but didn't like Manilla too much. Also spent some vacations in Costa Rica, and went to USA and Canada for work. I really enjoy going to Asia and meet people there.
• I am getting crazy about orchids, my collection is growing fast, maybe a bit too fast, but I'm rewarded by fantastic flowers, and I'm trying more and more to travel to places where I can see them growing in their natural habitat, unfortunately they are seriously endangered, and it can be fairly hard to still find them in the wild.
• I also went to badminton training again this year but competition is really getting a bit too hard, and my body disagrees at time, I think I will slow this down a bit next year.

Overall quite a bit happened but no revolution, the most scary thing is that time flies faster and faster, so no promise about increase of blogging frequency !

On broken XML processing with libxml2

The XML spec is very clear about this:

Once a fatal error is detected, however, the processor MUST NOT continue normal processing (i.e., it MUST NOT continue to pass character data and information about the document's logical structure to the application in the normal way).

That's the default behaviour of libxml2, but for the purpose of helping recover occasionally corrupted data, I provided an XML_PARSE_RECOVER parser flag allowing to correct some of the trivial errors, for example if using xmllint --recover. Now the problem is that people start using this flag for default processing, and that's something I said many time I didn't want to happen. If blog generator can't output correct well formed XML, then they need to be fixed (and that's true for all other XML generators). The intent of the drastic rule in the XML spec is cristal clear, to avoid data corruption and data loss by forcing non conformant behaviour to be fixed at the source, not at the consumer level.

I am gonna give a hard time to those who abused this feature of libxml2, because while not completely disabling the option, I will make sure it's not used repeatedly, I'm still looking at the best way. Feedback welcome on this issue, preferably on the mailing-list.

half a decade of community knowledge gone

The archive of the xml@gnome.org and xslt@gnome.org are gone, I have no idea what happened, I didn't got a mail about it, just that going to the archives show all history is only the 4 last days. The only thing I have left right now are the words saved in the database used for the search engine on xmlsoft.org. The code is here, but the knowledge, the exchanges, all the process leading to it has vanished, I just can't express how sad this makes me...

[update] Apparently it was an indexing problem which should be relatively easy to fix and without serious effects. The post are still on-line and I now keep separate archives for more safety, the data is safe. To me OSS development is really about the process, i.e. the history, knowledge, feedback, the memory of that shared process is as important as the code itself. With that knowledge you can recreate the code, if you have just the code you miss out a large part of its semantic, and it can't be regenerated. That's why I don't believe in dumping large base of source code under a free licence can really compare to code coming from a genuine OSS/free software project.

libvirt 0.1.0

Yup, another release from the canned penguins dept :-) . The API got extended a bit but it's mostly a large revamp of the internals to follow a driver model, so it gets easier to keep the code clean, and contribute support for other virtualization engines than Xen.

There is still quite a bit of work needed, for example acces the Xen Daemon though XML-RPC for versions of xen >= 3.0.2 (which is out apparently). I started incorporating patches coming from various contributors, I like this !

French invades Fedora Planet

it's fun, but a bit disturbing to see 50% of the posts in French. Bienvenue a bord ! Oh I don't believe blogs billets should be limited to a single language even though I stick to english only usually.

Ekiga-2.0.1 and FC5

Ekiga-2.0.1 is a clear success from my point of view, it's installed by default on Fedora Core 5, and so far I received only one bug report: I didn't switch avahi support on (I was a bit conservative, I will switch it on when pushing 2.0.2 when released). In general Fedora Core 5 looks great to me, I miss my old friends xv and xsysinfo, but rhythmbox is a great improvement, audio support is better in general. Unfortunately I can't run my desktop as a Xen Domain-0 because X crashes randomly every day or so :-\ . I also don't activate beagled by default due to the near continuous drive accesses it generates, even after a couple of days (i.e. when I would have expected its database to be initialized). Yum and Extras support works well, that is great too.

Spring is late

really cold lately here, we got snow on the mountains just a few hundred meters above Grenoble. I planted a bunch of new flowers in the balcony, but they are growing slowly due to this. There is a rose bud which is showing up anyway I guess everything will start blooming as soon as I leave for vacations in May ... On the other hand the orchids are in good shape inside, it's nice to have them blooming next to the desk. The habaneros pepers seeds germinated well and I should have some burning hot but delicious pepers this summer :-). I look forward vacations in Asia, hopefully I will be able to import new orchids, well if I manage to get through the french importation administative hassles...

Ekiga-2.0.0

is out ! Congrats to Damien ! Packages are available for Fedora Core 4, and will be available for FC5 ... sometimes ...

libvirt

Slowly making progresses, next steps will be to drive QEmu instances and switch to XML-RPC for communication with the Xen daemon. Some serious internal code refactoring is needed.

libxml2

Went to the W3C XML plenary two weeks ago for the XML Core working group meeting, it was nice to see again a bunch people. Standard guys are a different kind of geeks, they don't hack code they hack specs (some do both !), and hacking the Web and XML can be really fun (or boring :-) ! I also went though all Coverity reports for libxml2 and libxslt, it was a PITA mostly due to the web interface to the reports, a couple of them would really lead to runtime bugs, but they forced me to be far more coherent about NULL pointers handling, static analysis is good because it can spot problem in code that your users will usually never run into (and if they do it won't be reproduceable !) So thanks to whoever paid for getting the analysis done :-)

Could someone explain locality trade-offs to Mozilla/Firefox developpers ?

Yes this may sound like a rant, it still is a very serious issue IMHO ...

Seriously guys before trying to push your stuff as a platform, fix your code. A program which balloons to half a gigabyte of virtual memory usage will not draw the web page I want to see any faster. But it managed to force me to reboot my box, linux handling of Out of Memory conditions is pathetic (I'm told there isn't good solutions to this, maybe), so give me my memory back when you don't need it for the computation of said page rendering. Oh and stop using X11 memory as a long term bitmap cache, this fools nobody, this DOES NOT HELP !

I have an http cache it's called squid and never uses more than 5Mbytes of RAM, I don't need another disk cache in Firefox to double I/O accesses and disk usage, I don't need to cache pages rendering, current processors are way fast enough to redraw them instantly WHEN NOT SWAPPING ! [Thot/Amaya was perfectly capable of instant rerendering on 10Mhz processors, large displays and complex page layouts]

For those frustrated like me by this nice looking memory pig, here is my recipe:

$  cat ~/.mozilla/firefox/default.*/user.js
user_pref("browser.cache.memory.enable", false);
$

restart firefox, you will see the following line when loading about:config in the URL :

browser.cache.memory.enable  user set   boolean  false

for good measure if you're using a cache like squid turn off browser.cache.disk.enable too by double clicking on the entry. It seems it make it a bit less glutton, but it still fuck with the X11 pixmap cache, X11 size still grows (and do not shrink when closing browser tabs !!!), if someone has a trick to beat gecko/mozilla/firefox in releasing the image pixmap as soon as they are not needed for display I would be very happy. An historical anectdote is that we had to phase out all the X11 terminals in the lab when Nescape started being popular because offloading of this pixmap cache onto the server memory led them to memory exhaustion and reboot of said terminals. This behaviour has been plaguing us for way too long, can't someone fix that code ?