Older blog entries for ensonic (starting at number 127)

GStreamer & Buzztard

As a big leftover promise from my talk at the GStreamer conference in 2011 I spend more time to understand latencies inside GStreamer. For plain playback the latencies are not an issue, but for anything interactive, be it entering notes or changing parameters while the song is playing, we want much lower latencies. For the talk I did some measurements by having a pipeline with two branches - src1 ! sink and src2 ! fx1 ! ... ! fxn ! sink. The fx elements where plain volume elements. The branches where panned hard left and right. For the experiment, I was pulling down the volume on the two sources down to 0 at the same time and recoding the audio on the sink. Looking at the wave, one could see the delay in the signal that went through the effect-chain. The longer the effect-chain the larger becomes the delay. Don't misunderstand this, GStreamer is properly handling A/V sync. The buffers that get mixed in front of the sink have the same time stamps, the problem is that we have a lot more buffers traveling the the effect-branch. This is because of the queue elements. In buzztard one can freely link generators to effects, effects to effects and effects to sinks. This includes diamond shaped connections. Here we need queues to not stall processing on one branch. It is sufficient for the queues to keep only 1 buffer. Still even knowing the buffer duration, I could not tell a formula to explain the measured delays.
This month I looked at the issue with a new idea. I wrote a small example that build pipelines close to what I have in buzztard, but stripped of many details. In this code I add buffer probed to all pads. The probe is comparing the time-stamp on the buffer to the pipeline clock. This tells how early the buffer is. Ideally we want buffers to be generated and processed as much in time as possible. When generating audio one needs to configure audio-buffer sizes on the source elements and two properties on the audio-sink - buffer-time and latency-time. A good scheme is to use buffer-time = 2 * latency-time. That configures the sink to have two audio-segments. Initially I also set the buffer sizes on the source elements to have a duration of latency-time. Now one problem with that is that we will have one buffer waiting on each queue. Thus if there are two queues the actual latency is (n-queues + 1) * latency-time. One way to improve that a bit is to half the buffer sizes on the sources, then the 2nd buffer is calculated when the first one is needed. As the first one won't be sufficient to fill the gap, the calculation of the 2nd buffer is scheduled right away. The disadvantage of this scheme is that one gets quite jittery latencies. In the end I settled on finishing the subtick timing in buzztard. Each tick will have n subticks, where n is setup so that we get down to the desired latency. So far I get nice low latency on all my machines (inlcuding a atom based netbook).

We also ported more buzzmachines and have 49 machines right now. When porting we often also fix bugs as gcc is quite good at warnings these days. Finally, the machines now also install docs (where available).

In the UI I got rid of the GtkRuler copy again. The analyzer window now has own code to draw the rulers. I think they look nice, a lot less noisy then the older rulers.

62 files changed, 1957 insertions(+), 1844 deletions(-)
buzztard

The first big change after releasing 0.6 was to move from svn to git. SourceForge offers git since a while, but does not provide any tips how to do the transition. Fortunately it is a popular topic though. I used svn2git for the main conversion. I fixed the author tags from svn commit messages (Patch-by:) manually using git rebase -i as I could not get a filter-branch script for it to work. A few times gitk got confused and I had to remove my .git/gitk.cache, one symptom are missing tags. I updated the enlistments on ohloh and tweaked the git hooks to have the same functionality as before.

A first code wise change was to move the bsl module into the buzztard one. It is a small module for the buzztard song-loader plugin and it is not needed if you don't want to load buzz songs. But it is small and has no further dependencies and thus it is easier to just include it. Especially as we plan to add other loaders in the coming cycle(s).

In the editor I bumped the required gtk version to get rid of some #ifdef and be able to use newer API. I replaced the ruler widgets in volume and panorama popups with with scale markers. Those look nicer and take the scale handle size into account. Unfortunately they were almost untested. I made the needed patches for gtk-2-24, gtk-3-2 and gtk-HEAD [1],[2]. I could also made workarounds for the issue for the time being. So for the time being, don't use inverted ranges on your scales and don't use an adjustment with fractions (use 0 .. 100 instead of 0.0 to 1.0).

Most of the work went in the the interaction controller. The controller assignment in the UI is a bit more discoverable (content menu not only on main widgets, but also on the label and value label). Controllers can be assigned to combo-boxes too (mapped to enums). We have controllers for note-on midi messages and the velocity that comes with it now.

The other bigger change is that we now have persistent audio sessions (right now only enabled for jack). We're basically keep the sink alive across songs and also keep it in a resource allocated state. This gives a little speedup on the playback startup, but the main motivation was to allow to configure linkage of buzztard with other jack client in qjackcontrol. This is also an enabler for transport sync support. I have landed initial support for this in gstreamer-git.

There were also a few smaller changes for user feedback on docs and new tips. The design folder got more experiments.

485 files changed, 2638 insertions(+), 2033 deletions(-)
buztard

Finally after a long time, I managed to release a 0.6 of buzztard. So far only one regression was found and bml-0.6.1 was released to fix it.

A few things happened before the release still. At first after updating my distro, I made a lot of changes to avoid deprecated gtk+ api. For now we ship a copy of the ruler widget (that got removed in 3.0). The internal ruler widget
is a lot saner than the upstream one too.

Another big change was to move from string parameter for notes to an enum. This is faster and lets us do things like blending of note ranges or transposing.

I also made quite a few bug-fixes - laspa effects work again, fluidsynth fixes, etc.

Now I look forward to a lot of new changes in 0.7.X.
buzztard


hi,

The buzztard team has released version 0.6.0 "black beats blue" of its
buzz-alike music composer. All modules got extensive improvements over the last
release from more than two years ago. Give it a try, join hacking and report bugs.

bml
Improved machine compatibility.

bsl
Several bug fixes and better compatibility.

buzztard
Main feature of this release is full undo/redo support. Related to it is the
journaling of edit action and the crash recovery. This way chances of losing
changes in the song are quite low. Other UI improvements are: tip of day,
improved spectrum analyzer, clipboard support, more commands in context menus
and many more). This release features a gstreamer decoder that enables playback
of buzztard songs in any gstreamer based media player.
We also kept the buzztard codebase clean and ported from deprecated APIs to the
successors (gnomevfs->gio, hal->gudev). The libraries and the applications got
performance improvements in many areas.
Also the docs have been improved a lot with tutorials, keyboard shortcut tables,
better coverage and man-pages.

gst-buzztard
Lots of code cleanups. Get rid of the temporary help interface. Switch from
liboil to orc. Performance improvements.

project-page: http://www.buzztard.org
screenshots: http://www.buzztard.org/index.php/Screenshots
downloads : http://sourceforge.net/projects/buzztard/files/
gstreamer & buzztard


Over the last 2 month I did quite a bit of work on the GStreamer side. Right now we're working on 0.10 and 0.11 in parallel. I worked on the audiovisualizers and opencv elements in 0.10. I added a freeverb port to 0.10. In 0.11 I updated the controller susbsystem. If is now a lot easier to use and faster too. I also ported the audiovisualizers and fixed a few elements here and there.

In the beginning of November I gave a seminar about mobile multimedia using GStreamer. For that I wanted to have a few more examples. I have been polishing some examples in GStreamer and adding some more (simple audio-mixer with xfade). One example is a nice showcase for how easy you can do some things. It is called tonematrix. It is 497 lines of c, including comments, a GTK ui with an own widget and the whole GStreamer handling. The toy supports different sounds, different speeds and different scales.

Buzztard allows live control of audio elements. Right now can use midi and hid devices. The software that exposes the wiimote as a hid device uses uinput. This is a kernel device with a simple api that allows user-space apps to create hid devices. I wrote a small toy (280 lines of c) that can use different GStreamer pipelines containing analyzer elements and map the detected features to virtual joystick controls. The simplest example is to map the loudness of the mic-input to the x-axis. A similar example is to map the brightness of the
camera input to a joystick axis. Then you can control e.g. the filter-cutoff of a synthesizer by shielding the light from the camera sensor. Now the fun starts with the recently improved facedetect plugin in GStreamer. This can not only detect the faces, but also the positions of eyes, nose and mouth within the face. Unfortunately the detection is not very stable when drawing faces. The idea here obviously is to make grimaces and control sound with that. A very expressive performance :)

I also spend a lot of time to track down issues with dynamic linking. A few more fixes are done on buzztard side and GStreamer side. I think on the buzztard side things are good for a release. Everything is reviewed (docs, demo-songs, ...) and make distcheck passes :)
buzztard, valgrind and ...


This month I focused on testing for the next release. My free time for the project was a big short anyway, as my family move to our new home and I had quite some janitorial work to do.

One lesson learned for the start. Some time ago I had problems with tests going crazy and eating memory. This can bring down the whole system which is bad. My solution at this time setrlimit(RLIMIT_AS,&rl) to cap the memory usage. While that works, that is also one way to shoot yourself in the foot. I was wondering why tests that work flawlessly on 32bit systems fail with "mmap() failed: Cannot allocate memory". Doh, 64bit platforms need more memory than 32bit ones. Now I raised the limits unconditionally and everything is working again. Would be good to check the difference to see how much more memory 64bit apps that in average - its not twice as much I think.

I did lots of valgrinding and obviously found and squashed a few leaks. It would be really nice if each library would ship a suppression file as part of the dev package. The file could be installed to /usr/lib/valgrind/.supp. The pkg-config file would have two variables: 'vg_supp' pointing to the suppression file and 'vg_env' setting extra environment variables easing valgrinding. This way one could just collect these from the libraries an application uses and be done. GObject based libs could suppress their singletons to improve the valgrind experience.

From time to time potential users show up on irc and get scared when I ask them to build the latest version from the sources. In the need for having an easy way to offer testers and translators the latest version, I took a look at glick. It is actually pretty straight forward. The resulting glick files where initially somewhat large, but after pruning development files, they are just about 6 Mb. I have two of them online - one for 32bit and one for 64bit systems. How to use them? Just download and run them - they don't install anything. The only downside I can see so far is, that the glick bundle would use the same settings as the properly installed one, which could be harmful if different versions are run. Also the path to the example songs in the bundle is somewhat cryptic - /proc/self/fd/1023/share/buzztard/songs :/. Having better desktop integration of such bundles would be great too, but less see how glick evolves.

50 files changed, 836 insertions(+), 373 deletions(-)
buzztard

After lots of changes I have switched the code permanently to the new sequence model. It saves about 170 lines of code in the sequence-page source. There is some potential to save more though.

I also did more work on state persistence. A song now contains more information (selected machines, pattern, options) and these things are restored when loading a song.

Another good change was moving the song-unsaved flag from the song (core-library) to the edit application. This make the core library more light-weight. The editor application can now determine whether there is something to save by combining the unsaved-flag and the undo/redo stack. The unsaved-flag is still needed as I feel not everything needs to go to the undo/redo stack (like selecting a machine). Undo/redo looks quite complete now. I found a solution for my last issue, although I am still not entirely happy with it.

I also made a few startup time improvements. Found a weird issue with the interaction-controller library probing joystick devices. One ioctl on joy devices for hdaps devices is hanging for quite a bit.

Finally I did a huge cleanup on or somewhat uncommon header file setup. We always had a header declaring the types and one the methods. This helped with include conflicts. But now we have 50% less files to care and I just needed 2 forward declarations.

Again several bugs were fixed. I will check remaining reported bugs and hope that everything is fine to consider releasing 0.6.

253 files changed, 27205 insertions(+), 26602 deletions(-)
buzztard

Another month with good progress. I've started with the undo/redo parts in the sequencer view. Most of the edits are handled now. I am still stuck with one problem though. So far the order of signal handler on object removal did not matter. Now it does :/ There are things that when they get removed remove other things. And that causes chains of modificatins that in the change log would overwrite (shadow) the save data. Not yet sure how to solve that. I'll probably confront someone at the desktop summit with the problem over some beer in the hope that I find a solution while I am explaining the issue :) As a good side effect of the undo/redo work, I could remove the warning dialogs that I was showing when removing things. Makes the editor more pleasant to use.

During my july trip to mountain view I brought two cheap midi controlers - korg nanokontrol and nanokeys 2. The nanokeys 2 is not so nice, but the nanokontrol seems to be okay. Unfortunately I didn't notice is is the old model, the nanokontrol 2 has more keys. I have played quite a bit with the nanokontrol in buzztard. Found and fixed a few bugs. One thing I definitely needed to do is to store the keys one has trained. Thanks to my long train journeys between munich and leipzig these days, that got implemented. For the nanokeys I need to still do a couple of things and those will have to wait for the next version.

As I was blocked on the undo/redo I did more small mini-songs. As usual found a couple of bugs. Buzztard stores timestamps in the song (like when it was created and when it was saved the last time). When closing a song with changes it warns you about loosing the changes for the last n hours/minutes. Thas was completely off if one loads a song and makes some changes. I need to set the last-saved to the time of the first change. A few other bugs were related to 64bit arch and using wrong int types in varargs functions. I also made some ui improvements. The settings hide more unusable audiosinks (apexsink, sfsink), especially probing apexsink caused long delays as I don't have the hardware for it.

One issue that I had no good explanation for but that was quite apparent when doing demos was small breaks when the song loops. Especially at the first loop it caused a bad glitch and later it sometimes played a bit too much or skipped something when wrapping around. I had made some test apps for it, but could not reproduce it there so far. Finally I looked at it again and found a small self-contained testcase that was reproducing it. On that case I could narrow it down to a combination of two gstreamer elements causing it. Still it was not straight forward to fix it, after all it is not crashing or such. I decided that the only way forward was to get a good picture of what is happening and verifying the events step by step. Voila, we now have a improved gst-tracelib and a gnuplot script for plotting event over time. Would be nice to have a more interactive UI for gnuplot though. I filed a bug and put the data there, if you are curious about the graphs. After some more nights I found the issue and the fix for the adder plugin is now upstream. A simillar change needs to be done for the videomixer elements (will look into that soon).

43 files changed, 2779 insertions(+), 496 deletions(-)
buzztard

This month I made great progress. I have been making several small demo songs and found+fixed quite many bugs and glitches along with that.

After a little break I took up undo/redo work and could make good progress. Now also pattern property changes (name and length) are tracked. In the sequence the boilerplate code is there, single edits, track and property changes are handled. In the machine view the initial machine position is tracked in the change-log.

In the middle of the month I did doc updates - api docs had some stuff missing and the user help finality has more beef and short-cut tables for all the keyboard accelerators.

OmniMancer on irc gave me some good first perspective usage experience. I added a blacklist filter to hide gstreamer elements from the menu that are known to be not useful for buzztard (e.g. dtmfsrc). Also some machines used GTypes not yet supported by the UI - this is now fixed. With that came some fixes in
pattern editing (blending parameters wasn't working in all cases). I also added a flip operation to the patterns. Also to make a few things easier for new users I added two items to help menu - file a bug and goto irc. The later fires up the freenode webirc as xdg-open seems to be unable to launch e.g. xchat for irc://
urls.

I have started to write the missing treemodel for the sequence view and used that changed to overhaul a few things. Even without the new model its quite a bit faster as we do less model rebuilds (e.g. when expanding the length). Also the row-shading code is simpler and with that the cell-data functions. Finally
the pattern usage tracking is now using a hashtable instead of rescanning the sequence.

With all those changes comes a bag of bug fixes - I'll skip listing them here - it is all in the change log. Also I did a bit of code cleanups and reorgs. Like using macros for the GType handling to save lines of code. Or bump the gtk+ version (2.10.12) to get rid of fallback code. Finally I rechecked a glib bug regarding mime-matching. It's fixed since a while, but I still saw it, as the .recently-used.xbel file had the wrong mime type from the time when glib had the bug in there. Maybe it would be good to trim that file from time to time. Imho
it should also be in $XDG_CACHE_DIR and not $HOME, but that is a different story.

48 files changed, 2712 insertions(+), 1250 deletions(-)
This is not Helsinki



Yes, that means 'bye bye Nokia'. The recent events have killed the last remains of trust in the companies leadership. It is quite sad to see how things get ramped down.

The new chapter is called Google Germany. I just started in the office in Munich three days ago. So far everything sounds cool. I'll write about it when there is something to tell :)

118 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!