Run-time linker interface
If you’re debugging an application that loads thousands of shared libraries then be sure to read the LinkerInterface page on the GDB wiki.
Run-time linker interface
If you’re debugging an application that loads thousands of shared libraries then be sure to read the LinkerInterface page on the GDB wiki.
r_debug
I’ve been trying to figure out how to get information about libraries loaded with dlmopen
out of glibc‘s runtime linker and into GDB.
The current interface uses a structure called r_debug
that’s defined in link.h
. If the executable’s dynamic section has a DT_DEBUG
element, the runtime linker sets that element’s value to the address where this structure can be found. I tried to discover where this interface originated, but I didn’t get very far. The only mention of it I found anywhere in any standard is in the System V Application Binary Interface, where it says:
If an object file participates in dynamic linking, its program header table will have an element of type
PT_DYNAMIC
. This “segment” contains the.dynamic
section. A special symbol,_DYNAMIC
, labels the section…
and later:
DT_DEBUG
No help there then. In glibc, r_debug
looks like this:
struct r_debug { int r_version; /* Version number for this protocol. */ struct link_map *r_map; /* Head of the chain of loaded objects. */ /* This is the address of a function internal to the run-time linker, that will always be called when the linker begins to map in a library or unmap it, and again when the mapping change is complete. The debugger can set a breakpoint at this address if it wants to notice shared object mapping changes. */ ElfW(Addr) r_brk; enum { /* This state value describes the mapping change taking place when the `r_brk' address is called. */ RT_CONSISTENT, /* Mapping change is complete. */ RT_ADD, /* Beginning to add a new object. */ RT_DELETE /* Beginning to remove an object mapping. */ } r_state; ElfW(Addr) r_ldbase; /* Base address the linker is loaded at. */ };
With glibc, r_version == 1
. At least some versions of Solaris have r_version == 2
, and when this is the case there are three extra fields, r_ldsomap
, r_rdevent
, r_flags
. GDB uses r_ldsomap
if r_version == 2
; the other two seem to be the interface with librtld_db
. That’s not documented anywhere to my knowledge, and may not even be fixed: applications are supposed to use the external interface to librtld_db
as documented here.
Here is the problem: r_debug
, as it stands, has no way to access more than one namespace. The objects in r_map
are the default namespace, directly linked, or opened with dlopen
, or opened with dlmopen
with lmid
set to LM_ID_BASE
. The r_ldsomap
field in Solaris’s r_debug
gives access to the linker’s namespace, opened with dlmopen
with lmid
set to LM_ID_LDSO
, but you still can’t see any other namespaces.
glibc uses multiple r_debug
structures internally, one per namespace. It would be trivial to add a “next r_debug
” link to r_debug
if it were possible to extend the structure, but to do this you’d need to set r_version > 2
. Applications could arguably expect a runtime linker with r_version > 2
to support the version 2 interface in full, but it wouldn’t be possible to do that in glibc without reverse engineering Solaris’s implementation. glibc is therefore stuck at r_version == 1
, and the r_debug
structure is effectively immutable for all time.
Garden office
Since two weeks ago my office has been this hillside hut. It’s a bit of a change from the pokey junkroom I’ve worked from this past twelve years. It’s a bit spartan inside right now–just a desk, a chair and a computer–but I’ll post some pictures later when I’ve jazzed it up a bit.
VM networking tip
If you are setting up VMs using libvirt then it’s a good idea to change the address of the virtual network to something other than the default. Why? Because if you don’t, and you create a guest which itself starts up libvirt and uses NetworkManager then at least some of the time your VM will start up with its networking hosed.
If the host is using the default network (192.168.122.0/24) and the guests also want to use that network then there is a race between NetworkManager bringing up eth0 and libvirt bringing up virbr0. libvirt checks for existing interfaces using the network it is configured for before starting up virbr0, so if NetworkManager brings up eth0 first then virbr0 will not be set up on the guest and everything will be fine. But, if eth0 is not set up by the time libvirt runs the check, then virbr0 will take 192.168.122.0/24, then eth0 will come up on 192.168.122.something, and you’ll have a VM with two separate interfaces connected to two separate networks that both have the same address range… and it won’t work!
The easy way to solve this is to not install libvirt on the guest, but you may not be able to change this until after the guest is running, and if libvirt starts up during a guest’s installer then you may need to complete parts of the installation with no networking. This may or may not be ok for you and your OS. I’m using VMs to set up clean test environments for GDB, and at the moment I’m setting up three or four new “machines” every day (and throwing them away when I’m done) so I want the process as streamlined as possible. If you only occasionally set up new VMs then some extra tasks during the installation may not be a problem, but it is pretty simple to change the network on the host and you only have to do it once:
virsh net-edit default
192.168.122
to something elsevirsh net-destroy default
virsh net-start default
It’s a shame this can’t be fixed more conclusively elsewhere, but NetworkManager brings up the interfaces asynchronously at boot time which makes it impossible to definitively schedule libvirt’s startup to happen after NetworkManager.
Thank you Laine Stump for helping me out with this.
Only dest dir longer than base dir not supported
Hello,
Have you experienced the mysterious error, “Only dest dir longer than base dir not supported”?
I have.
The Problem
When you build an rpm, the code is built in %{_builddir}
, which usually evaluates as %{_topdir}/BUILD
, which in turn evaluates as something like /home/you/rpmbuild/BUILD
. The built code (if built with GCC) ends up with loads of /home/you/rpmbuild/BUILD
paths embedded in it, and the script /usr/lib/rpm/debugedit
rewrites these paths to /usr/src/debug
so that the debuginfo rpms work. /usr/lib/rpm/debugedit
cannot extend strings, it can only shrink them. If you are seeing “Only dest dir longer than base dir not supported” then, somewhere in your build system, %{_builddir}
is defined as something that expands to a string shorter than /usr/src/debug
.
Example
In my case, I was building a glibc rpm in a VM that turned out not to have enough disk. I created a new disk, mounted it on /mnt
, and added the line %_topdir /mnt
to my ~/.rpmmacros
. The result? “Only dest dir longer than base dir not supported”. I fixed it by editing ~/.rpmmacros
to say %_topdir /mnt/rpmbuild
.
Briefly
/usr/src/debug # the reference /mnt/BUILD # too short! /mnt/rpmbuild/BUILD # plenty long enough
Thank you,
The Mgt.
Saving money
I have a pair of set-top box PCs I’ve been using as always-on servers. I used them because they’re silent, but lately I’ve been thinking about power consumption. They were pretty good when I bought them in 2006 and 2008, but there’s much better stuff available now. I spent £60 on a Raspberry Pi and some supporting bits; given that it uses roughly a tenth the power of one of the set-top boxes it will have paid for itself in about two months.
While reorganising everything I also decommissioned an old Netgear switch which was likely costing £100 a year to run. Maybe it’s time you looked in your networking cupboard too!
ath9k_htc drivers for RHEL 6.4
This morning I packaged up ath9k_htc drivers for RHEL 6.4. This isn’t anything official, just something I knocked together. I’m using it with a TP-LINK TL-WN722N and kernel-2.6.32-358.2.1.el6.x86_64; it seems to work but your mileage may vary! RPMs here.
Because I never remember how to use OProfile
sudo opcontrol --reset sudo opcontrol --start # the thing you want to profile sudo opcontrol --stop opreport -l | less
Easy things to do with GDB #1
I’ve been meaning to write some introductory articles to GDB ever since back in February at FOSDEM, where I was surprised by just how many people do not use GDB–or even know what it is! It’s taken me a while to figure out a nice example, but I finally found them.
Last week I wanted to extend ogg123
, a tool for playing music files on the commandline. It supports several formats–Ogg Vorbis, FLAC and Speex–and the way it abstracts that is by passing around a struct format_t
full of function pointers to various callbacks. I was adding a new format, and wanted to know what kind of arguments it was calling its callbacks with. I could have tried to figure it out by reading the code, but it was easier to just run it in GDB and see what was happening. Here’s how:
wget http://downloads.xiph.org/releases/vorbis/vorbis-tools-1.4.0.tar.gz tar xf vorbis-tools-1.4.0.tar.gz cd vorbis-tools-1.4.0 CFLAGS="-g -O0" ./configure make
The important bit here is the CFLAGS="-g -O0"
. This causes configure
to pass these extra options to GCC in the makefiles it generates. -g
instructs GCC to include debugging information in the files it generates to allow GDB to relate the executable files you’re debugging to the source files they were generated from. -O0
instructs GCC not to optimise the code, which makes for easier debugging.
ogg123
you just built:
gdb -args ogg123/ogg123 ~/music/Various/Disco\ Demands/4-08\ -\ Give\ It\ Up.flac
Everything after the -args
option is the command you’d normally type to run the program. GDB will print out some stuff as it starts up, then present you with a (gdb)
prompt and wait for you to enter some commands:
GNU gdb (GDB) 7.4.50.20120313-cvs Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /home/gary/vorbis-tools/ogg123/ogg123...done. (gdb) >
flac_read
. I’d like to run the program until flac_read
is called, so I set a breakpoint on flac_read
by entering break flac_read
and then using the run
command to set ogg123
going:
(gdb) break flac_read Breakpoint 1 at 0x40dde5: file flac_format.c, line 253. (gdb) run Starting program: /home/gary/vorbis-tools/ogg123/ogg123 /home/gary/music/Various/Disco\ Demands/4-08\ -\ Give\ It\ Up.flac [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x7fffeb863700 (LWP 10312)] [Thread 0x7fffeb863700 (LWP 10312) exited] Audio Device: PulseAudio Output Playing: /home/gary/music/Various/Disco Demands/4-08 - Give It Up.flac Breakpoint 1, flac_read (decoder=0x637560, ptr=0x6182a0, nbytes=30240, eos=0x7fffffffe37c, audio_fmt=0x7fffffffe380) at flac_format.c:253 253 flac_private_t *priv = decoder->private; (gdb)
ogg123
stopped at the very start of flac_read
. Pretty jazzy huh? You can see the values of the arguments in the second-to-last line it printed, so for example you can see that the caller has requested to read 30240 bytes of data. The various structures (decoder=0x637560
etc) aren’t so useful as they’re pointers, but we can deference them with the print
command:
(gdb) print *decoder $1 = {source = 0x637060, request_fmt = {big_endian = 0, word_size = 2, signed_sample = 1, rate = 0, channels = 0, matrix = 0x0}, actual_fmt = { big_endian = 0, word_size = 2, signed_sample = 1, rate = 0, channels = 0, matrix = 0x0}, format = 0x617ec0, callbacks = 0x7fffffffe3a0, callback_arg = 0x0, private = 0x6375d0}
That’s pretty nice, we can see for example what format it wants the data in: little endian, signed, etc. Also, notice the line of code it stopped on:
253 flac_private_t *priv = decoder->private;
That’s the next line of code GDB will execute if we set the program going again in some way. We can advance over just that line with the next
command:
(gdb) next 254 decoder_callbacks_t *cb = decoder->callbacks;
Now it’s initialised the local variable priv
, which we can also print out:
(gdb) p *priv $2 = {decoder = 0x637000, is_oggflac = 0, channels = 2, rate = 44100, bits_per_sample = 16, totalsamples = 13337016, currentsample = 0, samples_decoded = 4096, samples_decoded_previous = 0, bytes_read = 24576, bytes_read_previous = 0, comments = 0x638240, bos = 1, eos = 0, buf = 0x637020, buf_len = 4096, buf_start = 0, buf_fill = 4096, stats = { total_time = 0, current_time = 0, instant_bitrate = 0, avg_bitrate = 0}}
Did you see what I did there? Most GDB commands have abbreviated forms, and the abbreviation for print
is p
. The commands I’ve introduced in this article are some of the most commonly used, and their abbreviated forms are all simply their first letter: b
for breakpoint
, r
for run
and n
for next
. You can also repeat the previous command by pressing Return, which is handy for doing a load of next
commands one after the other, for instance.
Ok, that’s all for today. Go and have a play!
Breakpoints on inlined functions
I just committed a patch that makes GDB able to set breakpoints on inlined functions by name.
New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.
Keep up with the latest Advogato features by reading the Advogato status blog.
If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!