Older blog entries for company (starting at number 10)

threading

Everybody that has ever tried to use GStreamer in a multithreaded way knows that it is horribly broken. If you have a pipeline running in a thread and want to unref that pipeline in an EOS callback when it's done, you can't. You need to put an idle handler into your application's main loop and unref the thread from there. Ask the Rb and Totem guys about how intuitive that is. So what are the requirements for a threaded media playing framework? I've come up with these (in order): - The obvious part: Doesn't crash, no unguarded variables, ... - It doesn't deadlock unless the app does something wrong and that something is well documented - or better: obvious. - The pipeline has a minimal overhead when running in non-threaded mode. - Every element in GStreamer runs all of its code in the same thread. Since an element is in essence a vtable that's filled by the plugin, this means that all of those vtable entries are called by the same thread. (This is done for mainly 2 reasons: It avoids the need to lock inside elements, which eases plugin development and some stuff, like OpenGL, expect to run in the same thread all the time.)

Now here are things people want to do: - as mentioned above: unref a thread when it hits EOS - be able to change the running pipeline in a different thread (think gst-editor here) - be able to change properties of elements in different threads (think volume in a media player) - be able to listen to signals from a running thread (think error signal or volume-changed in a media player)

marshalling

After discussing this quite a bit the last days (thanks Mike for the input), I came up with the solution of marshalling calls to an element into the thread the element belongs to and dispatch them there. Block the calling thread until the call is dispatched and then continue. Of course this doesn't work for a simple example that even exists in current code:

  • user changes volume
  • GUI thread wants to set the "volume" property on the volume element in the gst thread. This is marhsalled into the gst thread, the GUI thread blocks.
  • The gst thread changes the volume. This change triggers the notify::volume signal. The GUI thread has connected a signal handler for that to change the volume slider position. So the call is marshalled into the GUI thread. The gst thread blocks until done.
  • Noticed something? Right, at this point both threads are blocked waiting for the other to be done.
I quickly came up with the idea to not just block a thread when we wait for a dispatch, but to allow dispatching of other threads' marshalling requests while blocking. This nicely works in my example code, but unfortunately, it has an important issue.

reentrancy

What's this and why is it a problem? This is easily answered by Google. Unfortunately reentrant marshalling would make the following GStreamer pseudocode wrong in the general case:

if (gst_pad_link (pad1, pad2))
  g_assert (gst_pads_are_linked (pad1, pad2);
Since gst_pad_link is reentrant, the pads could have been unlinked by other code that happened to be dispatched before returning. I'm currently investigating if this is a real issue, or if there are constraints that make it impossible for such things to happen. Another thing I'm wondering is how reentrant our code already is (g_signal_emit requires reentrancy for example - or do you know what happens inside signal handlers?) The last thing I'm wondering is how much of a burden this is for GStreamer's target audience: application writers and plugin coders. Since GStreamer itself doesn't do much unless you ask it to, most reentrancy issues should be caused by application code. Plugins only handle their own stuff and should never call out anyway. Or am I missing something?

I'm really not sure on the big picture here, since it's a very hard problem where I don't have much experience. So if you happen to have run into problems with this that I'm missing, feel free to mail me. And if you're one of the bonobo/Orbit guys that knows URLs to the reentrancy issues those projects had, so I can read up on it, get in touch with me.

life

I've quit my student part-time job. Writing Office macros annoyed me so much I didn't want to anymore. And since I still can't convince louie to make Novell hire German students working on GStreamer, I'll soon go workless. Sucks. But not as much as writing Office macros.

automation

I've just read Glynn's post about an article open source issues. While the article's main points is nicely refuted by Glynn, one line got me thinking: "If you can't install it, you don't deserve to use it".

My point: Free software tools are incredibly bad at automatically doing the right thing - or at least don't aid. GStreamer's support for "just working" is a mess - everything requires manual intervention when in fact the GStreamer core should just do the right thing. Examples are autoplugging (There's a "let's just hope nothing breaks" mentality), plugin registration ("it works most of the time and if not, just delete all your config files and try again") or selection of audio and video outputs ("Just try another possibility in gstreamer-properties"). Nautilus still doesn't present me with "network places", my Linux kernel requires manual insertion of the right modules (or alternatively, distro-specific discovery tools) instead of a tool that jsut does the right thing. Even my Debian packages have let me down recently, when I tried to install icecast2 only to figure out that activating it requires editing a config file which it tells you in a one-liner on bootup. Next time, pop up a dialog during install, please? And to finish it all off, gtk-doc is one of the worst tools in the just works department.

The reasons I have identified why this doesn't just work so far are these:

  • We (the developers) all know how to get stuff done without automation. So why should we invest time in automation, when we can do without? I believe the fact that most people that are working on automation are company funded might be a good prove for this reason. Also there's a lot less duplication going on everywhere than in product automation. There's less mp3 players in Debian than distros available, which is another proof that automation (in this case installation) is tagged on and not designed into it.
  • Automation is hard. It does not only require code that does it, but it requires knowledge of the full system you are working with and adding enough introspection capabilities to the system, so the code can work from within. And especially providing introspection capabilities is an often neglected part, if only because it's boring or people don't know how to do it.

Gstreamer autoplug

Of course there's also a reason why I started to think about such things and it's the new autoplugger I'm hacking on, that's finally supposed to do everything our old autoplugger missed. While I had to write a lot of ugly code to get my automation working, all the features have been implemented and I'm using it on my local Rhythmbox for playback without problems and without all the old workarounds. I think I'll do a first release soon to get more people interested in it, when I have the metadata importing working without the one hack it uses, too. Especially because that will add an important feature: It will automatically add all sound files GStreamer knows about and not just those that are "enabled". I'm already afraid of the day when I'll try to import / just to see if it works good enough. I hope to get this stable enough (including the video part) so we can retire the old autoplugger for the next major Rhythmbox and Totem releases and Gnome 2.8. Hopefully that'll be the first release where Rhythmbox doesn't need to work around any GStreamer issue anymore and just can play any file that has audio in it.

Hi blog, long time no see. What's been happening?

GStreamer

0.8 is out the door. And I've already become annoyed with it. Good thing I'm the only one, people seem to like what we've done.
One thing that ds and I agree on though is that we're going to heavily modify the scheduling in the next big release. (Looks like we'll start that after we got Totem and Rb into Gnome 2.8) The good side: This allows you to wait on file descriptors, pads, the clock or conds in your element all at once and we don't need threads nor cothreads anymore (yay portability!). This will be liked by our video sinks (who will be able to listen to events even when no buffers arrive) or the people thinking about midi (who can send "nothing happened" even though nothing happened on the input). And there's of course the bad side, that rbultje is gonna flame me for: We'll need to rewrite large chunks of our elements. (Of course this has the plus side of finally going into a proper deep inheritance tree and really doing OOP this time.)

KDE

Due to ever increasing interest from KDE guys and them not getting a KIO wrapper (that's gnome-vfs for KDE) working, thought "that can't be hard" and deided to write one. Good things: 1) I know KDE a lot better. 2) There is a KIO slave in GStreamer now. Bad things: 1) It doesn't work reliably yet. 2) KDE
Now, before people start doing stuff to me for that second point, let me explain, why I dislike KDE and especially what I dislike. Gnome focusses on the model of separating functionality and using cleanly defined API for integrating all of that. This of course means that you can rip out the parts that you like and use them. You can write an app like the Gimp, that doesn't require libgnome. This doesn't work in KDE. KDE subscribes to the model of integration. In KDE everything is integrated. KIO only works if you have a working KApplication setup in your code. Since a KApplication uses the Qt event loop, KIO uses it, too. And since a Qt app has widgets, you can easily pop up an error dialog via the KIO API. This of course means that you now need X and a full blown widget set on your machine to open a webpage. Even if it's just a command line app. And now try wrapping this in an app that is not KDE.

Luis Ximian

Luis Ximian made me now write something about the issue I talked about with him last Guadec and that still gets worse: Companies vs Communities.
I still have a bad feeling in my stomach about the GNOME people getting too corporate and starting to a) raise the barrier to entry into the GNOME hacking world, b) hiding decisions from the community and c) steering GNOME into corporate waters far too much. You can see this for example on the Mono vs Java discussion. For a community this is a total non-issue, because a community just uses the tools that it likes to build great apps. Another example is the "we don't include code in Gnome that is possibly patented" (I probably should add "in countries where the corporations backing Gnome are interested in") viewpoint. Communities like Debian found a solution for that problem. Gnome just doesn't ship spring-loaded folders or usable video playback plugins.
But the most important thing for me is that the people I interact with are more and more employees that need to get a job done, not people that hack on their project for fun. (Add the bounties here if you want.) IRC nicks end with @redhat.com or similar instead of .edu, .pl or .net. And this is especially bad because the newbie help is missing. Because helping newbies offers no short-term benefits and people nowadays need to get a job done. But that reduces new blood coming into the GNOME world. And I believe that without new people joining, GNOME will not be able to scale. GNOME needs all those testing volunteers that run broken cvs builds and report bugs, categorize the bugs, do translations, create new artwork, provide patches or write new tools. You'd need to throw a lot of money in that direction if you wanted to pay for that. And it is my strong believe that you can only attract people to your cause, if you provide a community to attract the people. If someone is interested in GNOME but doesn't know how to get involved, joins #gnome and speaks a shy "hi" and then noone answers, GNOME might have just lost a valuable contributor. I still know this day three years ago when I first joined #gstreamer and said a shy "hi" and people did talk to me. I liked it more than the other places I tried, so I stayed.
You can say that in business terms, too: If noone had taken the 15 minutes to convince me to stay, Gnome would not have gotten contributions for free that would probably have cost a 6-digit Dollar-amount to get professionally.

random bits

  • Who are ebf and riggwelter? It's certainly funny when people show up certifying you at Advogato that you don't know and can't even classify.
  • Valgrind rocks. Not just using it, but adapting code for it, too. The header contain great documentation. Noticed that when adding support for it to GStreamer. It's slow though. Really slow.
  • Squashing memleaks like mad. Loading 1500 files into Rhythmbox only takes up 50MB instead of 500+ like before. Need to make teuf pound on it a bit more.
  • Rhythmbox plays mods. Yay. And if someone released GStreamer 0.8.1 I could start working on making it release the sound device on pause, which 0.8.2 will support.
  • hadess: Since you wanted opinions about USound, here's mine: Let them deliver first. There's a lot of sound servers that don't work already. Oh, and they should definitely write a plugin for what I the future sound API: alsalib.
  • Miguel is a Troll. Had to be said. Thx to tberman for listening to my rant on IRC so I didn't feel the need to put it somewhere permanent.
  • Anyone have an idea how much voodoo-fu it'd take to make this use a GStreamer backend?
20 Feb 2004 (updated 20 Feb 2004 at 17:21 UTC) »

Hm, bridging media frameworks is not easy. The curse of always trying to do the weird stuff I guess. I'm trying to wrap the plugins in Xine for GStreamer, so we can make use of their plugins until we get our own. The curse is the state of multimedia libraries in OSS. Some of them don't release code often enough, some of them don't even exist as libraries. So all those multimedia projects end up including the code directly with a few minor patches here and there.

So guess what happens when you get crazy enough to bridge two of those frameworks? That's right, symbol clashes en masse. Now how do I fix those? It doesn't make it easier that g_module_open for whatever reason insists on using RTLD_GLOBAL no matter what. Oh, wait it doesn't! Yay for the glib maintainers. Now to convince the GStreamer crew to require glib 2.4 ;)

Oh, and in case you're wondering why anyone would want to wrap Xine stuff for GStreamer:

  1. Because it's there.
  2. Xine has support for stuff GStreamer doesn't have support for. My pet peeves are win32 DLL support, lots of input plugins (CD reading, MMS, RT(S)P) and audio outputs for other systems (Solaris, DirectX, ...). It's nice to have at least a stupid solution. Better than none at all.
  3. I learn a lot about Xine and how it's done internally. There's one thing I know already: I like the GStreamer API better.
  4. GStreamer gets a lot of improvements because it is made capable to cope with weird ideas. Kinda like the obfuscated C contests.
And I'm not having time for anything really serious at the moment anyway because of work so I do light things like writing decoder plugins.

Seems like everybody is talking about metadata lately and I should be somewhere in there, too, since I'm the one who wrote the tagging backend for the upcoming GStreamer release. I wonder how hard it would be to make Medusa use those tags. It should be easy just from looking at it.

Now the next step would be to get Colin, his metadata storage in Rhythmbox, Thomas and his media-info and nautilus-media hacking and Medusa into the same boat, so there's not too much duplication going on. The only thing I know is that I won't lead the efforts, there's too many people insisting on their opinions already.

Oh, and the other thing I know is that tag extraction is working perfectly nice in GStreamer. The hard parts are automated tag writing and detecting if a given data stream contains audio and/or video (which is metadata in itself) and if GStreamer can play that file back. But that's an entirely different issue...

I've watched this movie around 50 times now and it's starting to get boring to see if when the snowman punches the glass my soundcard actually does this *dong* sound. Now it does. I don't want to debug realtime stuff again. Especially not A/V sync. It works well, even though technically it's still flakey, we'll need to figure out how to do sync correctly some day. It's a hard problem especially because synchronized elements do not know a thing about each other in GStreamer - yet.

I accidently deleted my Rhythmbox library. Now I'm too proud to use anything else but GStreamer to reimport it. Quite some work to do the next days - and without Rb...

On the musical front I'm totally sold to bad 80s pop at the moment. And there's very good 80s music imitations for free on the web. And then there's still the Dirty Dancing soundtrack...

Man, I'm sitting in front of vim, staring at a source file, knowing exactly what I want to do, but I can't concentrate. Someone fix that bug in my head, please.

Had a great Sylvester evening listening to Beethoven's 9th Symphony. Spent most of the boring part - the 3rd movement - thinking about GVADEC and if prople would be interested in getting a talk by me about GStreamer internals. I still don't like giving talks when they are scheduled.

Spent another part of my thinking about Ronald's "Do we want a perfect core or do we want applications?" It's bad to just change the internals of a big library too much to make them perfect but it's also bad to only make them do one thing and break for every other thing. I guess people that develop libraries are more of the "make it perfect" kind while application developers tend to "make it work". I'm definitely more a library developer than anything else.

On the coding front I started to play with my Christmas gift. And since I'm the library guy, I ended up inventing libgstui instead of just porting TVtime to GStreamer. Let's see how my "GNOME needs a TV application" quest ends...

You know what is cool? When other people tell you they like what you do and you don't expect it. I've had that twice the last days.

First it was Joshua Pritikin basically telling us GStreamer guys that we rule. That was the first happy bump in my stomach. Especially because he's a long time GStreamer user. And today a mperson at my job went into retirement. And she gave me a letter. That already confused me. It was a thank you for the great job I did with our software and with helping her. Included two tickets to the Sylvester concert of Beethovens 9th symphony. I haven't been happier for quite some time.

I should tell people more often when I think they rule...

Just posted that in IRC, thought it's worth putting here. My personal idea of the goals for GStreamer are these:

0.8 is about reading and playing every media file on earth without crashing. 0.8 is about reusing pipelines, too.

0.10 is about defining what you may and may not do, making calls signal errors if you do wrong stuff and make everything else not crash. 0.10 is about writing testcases that try all that, too.

1.0 is then about figuring out if everything works like it should and change where it's not

1.2 is about fixing bugs for the GStreamer renderman replacement, the new kernel module doing the netcode with gst and those Cubase replacements based on gst.

1 older entry...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!