Older blog entries for conrad (starting at number 61)

15 Jun 2010 (updated 17 Jun 2010 at 04:09 UTC) »

Speeding up cross-compiling with ccache and distcc on Debian

The conventional way of doing embedded development is to cross-compile everything then copy it onto the target, but working natively allows you to use "normal" tools and workflows. We want to issue commands directly to a shell on the development board or phone prototype, and speed up the compilation step by distributing it to a faster machine such as your workstation. This isn't the usual way to do things, but I like working this way, and here's how to make it work faster.

This article explains how to configure a Debian PC host and a Debian target system so that development done on the target invokes the cross-compiler on the host. The advantage offered by this approach is a speed-up of compile times. Note that this does not speed up other aspects of building, such as source configuration (which can be slow for packages using GNU autotools), linking or installation.

We assume that a full Debian system is available for development on the target: packages can be built natively using gcc and a full toolchain (binutils, ld etc.), and tools such as automake, autoconf, libtool, version control systems etc. are available.

The setup we work with uses Debian on both the host PC and the target. The examples will use a debian-sh4 on the target, with the sh4-linux-gnu-gcc cross compiler installed on the build host. For other target architectures, simply replace all instances of sh4-linux-gnu- with the arch prefix, eg. arm-linux-gnueabi-.

In this article, commands executed natively on the target device will use the prompt target#, and commands executed on the x86 build host will use the prompt host#.

The first step is to ensure you can build software natively on the target. For GCC:

target$ gcc hello.c -o hello
and for autotools projects:
target$ ./configure target$ make


Next, install ccache:

target# apt-get install ccache

ccache keeps a cache of compiled object files, such that the same compilation does not need to be repeated. This cache exists outside of your source tree, so it persists across invocations of 'make clean'. It compares the pre-processed source files, so that compilation of a source file will happen if it or any of its included headers is changed. The usual way to use ccache is to simply set your C compiler to be "ccache gcc".

target$ ccache gcc hello.c -o hello
and for autotools projects:
target$ CC="ccache gcc" ./configure target$ make

Debian also sets things up so that if you put /usr/lib/ccache ahead of /usr/bin in your PATH, it will get used for native builds whenever gcc is invoked. That is useful to set up, but not necessary for this setup with distcc.

An aside about compiler naming

Before we move on to cross compiling, it's important to realize that the native compiler is also available with its full architecture prefix:

target$ ls -l /usr/bin/sh4-linux-gnu-gcc lrwxrwxrwx 1 root root 7 Mar 17 01:45 /usr/bin/sh4-linux-gnu-gcc -> gcc-4.4

The binary called sh4-linux-gnu-gcc does the same thing on both the host and target: you can simply think of it as a program that takes in a C file and produces an sh4 binary:

    C source -> | sh4-linux-gnu-gcc | -> sh4 binary

The distinction between "native" and "cross-" compiling is then just a matter of what machine you are running this compiler program on. If you run sh4-linux-gnu-gcc on an x86 machine, you are cross-compiling, but if you run sh4-linux-gnu-gcc on an sh4 machine then you are just compiling. Of course the compiler binaries are different; the point is that a shell script which calls the compiler by its full name would work without modification on either machine.


distcc allows you to use a compiler running on a different, faster machine. This involves running a server (distccd) there, and it is far easier to set up than it would seem.

First, ensure that we can cross-compile on the build host:

host$ sh4-linux-gnu-gcc hello.c -o hello host$ file hello sh4-linux-gnu-gcc hello.c -o hello host$ file hello hello: ELF 32-bit LSB executable, Renesas SH, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped

Next, we install distcc on the build host:

host# apt-get install distcc

To activate the server and tell it what clients to allow, edit /etc/default/distcc:


and restart it:

host# /etc/init.d/distcc restart

You can check that it is running:

host# netstat -pant | grep distcc tcp 0 0* LISTEN 16142/distccd

So that we can ensure that compilation is running on the host, watch this log file in a separate window:

host# tail -f /var/log/distccd.log

Then, on the client (ie. the target system) we also install distcc:

target# apt-get install distcc

We do not need to modify the distcc configuration on the target as it will not be running the server, so Debian's defaults are fine. However, we do need to set an environment variable to specify which machine[s] to compile on.

target$ export DISTCC_HOSTS='host'

You run distcc in a similar manner to ccache, by simply setting your C compiler. Note that we are only distributing compilation, not linking, so we just run the compilation step:

target$ distcc sh4-linux-gnu-gcc -c hello.c

This should turn up in the host's distcc logs:

host# tail -f /var/log/distccd.log distccd[16390] (dcc_job_summary) client: COMPILE_OK exit:0 sig:0 core:0 ret:0 time:46ms sh4-linux-gnu-gcc hello.c

And back on the target, we have the hello.o file which was generated by the sh4-linux-gnu-gcc cross-compiler on the build host:

target$ ls -l *.o total 16 -rw-r--r-- 1 conrad conrad 884 Jun 11 07:28 hello.o target$ file hello.o hello.o: ELF 32-bit LSB relocatable, Renesas SH, version 1 MathCoPro/FPU/MAU Required (SYSV), not stripped

The C file was transferred over the network to the host, where distccd invoked the cross-compiler and then sent the results back to the target. The end result is the same as if sh4-linux-gnu-gcc had been run directly on the target, but we avoided using the slower CPU of the target system.

To fully take advantage of distcc, you can run distccd on multiple build hosts, and specify all their names in the DISTCC_HOSTS environment variable on the target. Then use eg. "make -j 10" to run multiple compiles in parallel, which will each then get farmed out to different build hosts.

Combining ccache and distcc

You can quite simply put these two tools together, by calling:

target$ ccache distcc sh4-linux-gnu-gcc -c hello.c
You can quite simply put these two tools together, by setting CCACHE_PREFIX to "distcc" before calling ccache:
target$ export CCACHE_PREFIX="distcc" target$ ccache sh4-linux-gnu-gcc -c hello.c
(Thanks to Joel Rosdahl for the correction).

The first time we run this the code is cross-compiled on the build host and sent back to the target, and ccache keeps track of that. The second time we run this, ccache notices that it already has a stored copy of the output hello.o, and decides to use that rather than calling the compiler. (From ccache's point of view, the compiler is "distcc sh4-linux-gnu-gcc").

For autotools project, you can simply do the following before calling ./configure:

target$ export CCACHE_PREFIX="distcc" target$ export CC="ccache sh4-linux-gnu-gcc"
After which the ./configure step will write Makefiles which specify to compile with ccache, so the rest of your build (ie. make -j 10) just works as normal without any new settings or any other change to your workflow.

For more discussion of combining distcc with ccache, see the distcc(1) man page.


By combining both ccache and distcc we can:

  1. avoid redundant compilations, and
  2. distribute required compilations to a faster build host.
The result is faster build times, which speeds up your development cycle and allows you to work more efficiently on the target system itself.

Syndicated 2010-06-15 00:00:00 (Updated 2010-06-17 03:56:52) from Conrad Parker

Monday Music: Heyoo by Kobi

Made with AUBE on Linux last November, this is Heyoo by Kobi:

AUBE/Metadecks Live is a music production tool designed for live use. A track like this is made by setting up a bunch of sample, rhythm and effects units, playing them for a while and recording the result.

This post uses the HTML5 <audio> tag. If the audio controls are not present then the problem may simply be that your browser does not support HTML5 <audio> with Ogg Vorbis (in which case upgrade to one that does). If you are reading this in a feed reader or via a planet aggregator, then the problem may be that the reader or aggregator strips the HTML5 <audio> tag -- in which case you might want to switch to a more modern reader, or upgrade your planet.

Syndicated 2010-05-24 00:04:00 (Updated 2010-05-24 00:06:21) from Conrad Parker

Monday Music: Deika by Kobi

Made with AUBE on Linux a few years ago, this is Deika by Kobi:

AUBE/Metadecks Live is a music production tool designed for live use. A track like this is made by setting up a bunch of sample, rhythm and effects units, playing them for a while and recording the result.

Syndicated 2010-05-17 00:00:00 (Updated 2010-05-17 00:00:04) from Conrad Parker

Streaming Ogg Vorbis with sighttpd 1.1.0

I just released Sighttpd version 1.1.0, which includes support for streaming Ogg Vorbis from standard input. In an earlier post introducing a new HTTP streaming server (sighttpd 1.0.0), I described how sighttpd could be used to stream raw data, such as plain text:

$ while `true`; do date; sleep 1; done | sighttpd

and H.264 elementary video streams but not Ogg, because an Ogg stream needs to have setup headers prepended for each codec stream. "Instead, we would need to do something like Icecast: buffering these headers and serving them first to each client that connects before continuing with live Ogg pages".

So, that's exactly what version 1.1.0 introduces with a new <OggStdin> module. The sighttpd.conf setup is similar to the normal <Stdin> configuration:

Listen 3000

# Streaming Ogg Vorbis from stdin, using the special
# OggStdin module that caches Ogg Vorbis headers
        Path "/stream.ogg"
        Type "audio/ogg"

You can run this with a shell pipeline like:

$ arecord -c 2 -r 44100 -f S16_LE -t wav | oggenc -o - - | sighttpd -f examples/sighttpd-oggstdin.conf
And you can connect to it as an Ogg stream, eg:
$ ogg123 http://localhost:3000/stream.ogg

At the start of an Ogg Vorbis file or stream are three mandatory header packets:

  1. The Vorbis BOS (beginning of stream) header, which describes basic information like the number of channels and the samplerate of the audio.
  2. Metadata in VorbisComment format, which basically consists of text values like "ARTIST=Richard Feynman".
  3. The setup header, which includes "codec setup information as well as the complete VQ and Huffman codebooks needed for decode".

We can view the raw contents of these packets with oggz dump:

$ oggz dump Kobi-Birk_20011125.ogg |head -n 30
00:00:00.000: serialno 0639825516, granulepos 0, packetno 0 *** bos: 30 bytes
    0000: 0176 6f72 6269 7300 0000 0002 44ac 0000  .vorbis.....D...
    0010: 18fc ffff 00f4 0100 18fc ffff b801       ..............

00:00:00.000: serialno 0639825516, calc. gpos 0, packetno 1: 94 bytes
    0000: 0376 6f72 6269 7320 0000 0058 6970 686f  .vorbis ...Xipho
    0010: 7068 6f72 7573 206c 6962 566f 7262 6973  phorus libVorbis
    0020: 2049 2032 3030 3130 3831 3303 0000 000a   I 20010813.... 
    0030: 0000 0074 6974 6c65 0042 6972 6b0b 0000  ...title.Birk ..
    0040: 0061 7274 6973 7400 4b6f 6269 0d00 0000  .artist.Kobi ...
    0050: 6461 7465 0032 3030 3131 3132 3501       date.20011125.

00:00:00.000: serialno 0639825516, granulepos 0, packetno 2: 2.820 kB
    0000: 0576 6f72 6269 7325 4243 5601 0040 0000  .vorbis%BCV..@..
    0010: 8020 9a19 a7b1 945a 6bad 1d72 9a42 abb5  . .....Zk..r.B..
    0020: d65a 6bad 2594 5a5b adb5 d65a 6bad b5d6  .Zk.%.Z[...Zk...
    0030: 5a6b adb5 d65a 6b8d 81d0 9055 0000 1000  Zk...Zk....U....
    0040: 0021 0c55 0651 c99c d65a 6b44 1064 0649  .! U.Q...ZkD.d.I
    0050: e920 d65a 6be8 a0a5 105a 4cad d65a 6bad  . .Zk....ZL..Zk.
    0060: b5d6 5a6b adb5 d61a 6320 3464 1500 0004  ..Zk....c 4d....
    0070: 00c0 1863 8c31 0619 6410 5248 21a5 9452  ...c.1..d.RH!..R
    0080: 8c31 e618 74d2 5147 9d76 da71 6821 9594  .1..t.QG.v.qh!..
    0090: 5acc 2de7 9c73 ceb9 d61a 080d 5905 0024  Z.-..s..... Y..$
    00a0: 0000 a838 8664 5886 0584 86ac 0200 3200  ...8.dX.......2.
    00b0: 0004 1024 4353 34c7 d554 cf34 5d55 0542  ...$CS4..T.4]U.B
    00c0: 4356 0100 4000 0002 8000 0a18 4451 1445  CV..@..... .DQ.E
    00d0: 5114 4551 1445 d1f3 3ccf f33c cff3 3ccf  Q.EQ.E..

When a client connects to a stream somewhere in the middle of a song, these headers from the beginning are required in order to decode the audio data. sighttpd writes the pages containing the 3 header packets to a temporary file (created with mkstemp(3)). When a new client connects, the contents of that file are sent to it with sendfile(2) before jumping into the current contents of the stream.

I'm not trying to make a replacement for Icecast, but instead building a more general streaming server -- and of course I want it to have good Ogg support! So, please try it out, and leave some feedback in the comments or in email to me or ogg-dev :)

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!