Older blog entries for conrad (starting at number 55)

How oggz-validate works

oggz-validate is a tool for checking the conformance of Ogg files against the Ogg logical bitstream framing specification and RFC3533. It is used by validator.xiph.org an online conformance-checking service.

oggz-validate builds on the correctness checks imposed by liboggz when writing Ogg packets. Whereas the low-level libogg simply allows an application to construct arbitrary Ogg packets and push them into a stream, liboggz checks each packet against the basic constraints:

  • Packet belongs to unknown serialno
  • Granulepos decreasing within track
  • Multiple bos pages
  • Multiple eos pages
oggz_write() fails if any of these constraints are violated.

oggz-validate works by reading the input file and attempting to reproduce its sequence of packets. It creates both a reader and a writer and feeds the output of the reader into the writer; any errors in stream creation are reported as validation errors.

File -> oggz_read() -> packets -> oggz_write() -> discard

Additionally, oggz_write() checks the following higher-level constraints:

  • File contains no Ogg packets
  • Packets out of order
  • eos marked but no bos
  • Missing eos pages
  • eos marked on page with no completed packets
  • Granulepos on page with no completed packets
  • Theora video bos page after audio bos page
  • Terminal header page has non-zero granulepos
  • Terminal header page contains non-header packet
  • Terminal header page contains non-header segment

For example, the check for "packets out of order" uses liboggz's parsing of codec granulepos to interpret timestamps of many free codecs including Ogg Dirac, FLAC, Speex, Theora and Vorbis. Also, there is a simple constraint in the specification for Ogg Theora that the BOS (Beginning Of Stream) header packet for Theora must come before that for Vorbis (or another audio codec).

What oggz-validate does not do is check that the contents of the codec streams are valid for that codec. Such checking is left up to codec-specific tools such as vorbose, and flac --test.

Syndicated 2010-05-05 00:00:00 (Updated 2010-05-05 00:00:02) from Conrad Parker

Towards adaptive streaming for Ogg

Video streaming must be reliable and glitch-free. It must be possible for video hosting sites to allow clients to adapt to the available bandwidth, and for clients to be able to take advantage of this.

Adaptive streaming refers to a system which allows a video streaming client to request different versions of a stream according to the bandwidth it has available, and to change this selection on the fly, during the course of streaming. Such a system of course requires the streaming server to have various versions of a stream available, each in different bitrates. In order to allow the client to switch streams on the fly the content must be produced in such a way that corresponding video frames in the different representations can be easily accessed and decoded.

The first stage in building an adaptive streaming system is making it work for static content, ie. files on disk. The second stage is making it work for live content, ie. streams coming from a video production system consisting of cameras, mixing desks and random people in black tshirts. The first is mainly a technical problem; the second requires developing both technology and production processes.

Microsoft have a proprietary technology for adaptive bitrate streaming called Smooth Streaming, and an extension for Live Smooth Streaming. Apple are following a more open path, pursuing standardization of their specifications through the IETF, in the current form of the HTTP Live Streaming Internet-Draft. This extends the m3u playlist format with durations, sequence numbering, caching and stream information hints.

Ogg does not yet have an adaptive streaming specification; this should be developed in a way that is compatible with open specifications, and also taking into account the various quirks of Ogg. For example, the client must have access to codec setup headers for each bitrate representation, and the system must accomodate chained Ogg resources (as commonly used for streaming Ogg). In the W3C Media Fragments working group we are developing specifications for addressing fragments of media resources. The ongoing development of Ogg Skeleton allows Ogg to take advantage of these, allowing faster seeking through OggIndex and gapless playback through hints on presentation time.

Encouraging use of these features requires tool support and demonstrations of novel applications for video mash-ups. Video on the web should be a means of creative expression, allowing new applications that mash up parts of many videos and present the result seamlessly to the user. This goal makes Ogg fun, and brings us beyond thinking about video on the Web as just a different way of watching pre-packaged TV-style content.

Syndicated 2010-05-04 00:00:00 (Updated 2010-05-04 00:00:03) from Conrad Parker

Monday Music: Wannago by Kobi

A change of pace ... here's a little Ogg Vorbis track to test out the <audio> tag, seeing as your browser probably supports that now.

Made with AUBE on Linux, a few years ago, this is Wannago by Kobi:

AUBE/Metadecks Live is a music production tool designed for live use. A track like the above is made by setting up a bunch of sample, rhythm and effects units, playing them for a while and recording the result.

Syndicated 2010-05-03 00:00:00 (Updated 2010-05-03 00:00:05) from Conrad Parker

Stable release maintenance with git (liboggz 1.0.2 and 1.1.1)

I recently released two new versions of liboggz, liboggz-1.0.2 and liboggz-1.1.1. These are unremarkable maintenance releases, fixing some bugs but adding no new functionality.

Last year I released liboggz-1.1.0, which introduced a new oggz_packet type. This changed some of the public API while remaining binary compatible. As this was a fairly insidious change, I decided to also keep maintaining the previous 1.0.x version so that any distributions shipping that could easily upgrade without risking breakage. I do general maintenance work and bugfixes on the 1.0.x version as much as possible, and then adapt those to 1.1.x. Luckily this is quite straightforward to keep track of in git.

After committing a change to the 1.0-stable branch I merge that into master:

$ git commit # on 1.0-stable
[1.0-stable ccd2a2f] Fix regression introduced in 8c2da1
 1 files changed, 19 insertions(+), 7 deletions(-)
$ git checkout master
Switched to branch 'master'
$ git merge 1.0-stable
Merge made by recursive.
 src/liboggz/oggz_read.c |   26 +++++++++++++++++++-------
 1 files changed, 19 insertions(+), 7 deletions(-)

As these were just maintenance releases, the commit graph produced by git lol is quite well woven:

Lightweight branching makes it easy to keep track of these changes so that simple maintenance work is isolate from other development. The upshot is that these branches are ready for release at any time; if a critical fix comes along that requires a new release, then no backporting or cherry-picking needs to be done to get the code into shape: there is always a branch in releasable state.

Of course on top of that I also have topic branches for new features under development, and I periodically merge master into those. When the new features are ready for release they can simply be merged back into the master branch and shipped, without ever getting in the way of general maintenance work.

Syndicated 2010-04-30 00:00:00 from Conrad Parker

29 Apr 2010 (updated 9 May 2010 at 08:09 UTC) »

git lola

The best tip I learned at Scott Chacon's talk at linux.conf.au 2010, Git Wrangling - Advanced Tips and Tricks was this alias:

        lol = log --graph --decorate --pretty=oneline --abbrev-commit

This provides a really nice graph of your tree, showing the branch structure of merges etc. Of course there are really nice GUI tools for showing such graphs, but the advantage of git lol is that it works on a console or over ssh, so it is useful for remote development, or native development on an embedded board.

It is even nicer when you turn syntax coloring on in git, which also has the advantage of colorizing diff output to warn about bad whitespace.

To get an idea of a whole project structure, I found myself often running git lol --all, where the --all option says to show all branches. I used that often enough that I made a new alias, git lola:

        lola = log --graph --decorate --pretty=oneline --abbrev-commit --all

which has the added bonus of making me hum Lola every single day.

So, just copy the following into ~/.gitconfig for your full color git lola action:

        lol = log --graph --decorate --pretty=oneline --abbrev-commit
        lola = log --graph --decorate --pretty=oneline --abbrev-commit --all
        branch = auto
        diff = auto
        interactive = auto
        status = auto

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!