Older blog entries for acme (starting at number 159)

Back Home

The dwarves presentation at OLS went pretty well, people even seem to have liked it. The paper is now available, use it as the documentation.

Implemented –expand_pointers, that unfolds the pointer types in the same way that –expand_types expand non-pointer types. Should be useful in getting a bigger picture of a project data structure maze of relationships.

It should be a good first step on helping with checking ABI breakage, just use it in the old and new binary and use plain old diff to see what changed, perhaps something down three pointer levels. abichk will probably be a combination of this and what codiff does.

Here is an example:

Lets look at struct request_list in the Linux kernel:

$ pahole -C request_list fs/super.o
struct request_list {
        int                        count[2];       /*     0     8 */
        int                        starved[2];     /*     8     8 */
        int                        elvpriv;        /*    16     4 */
        /* XXX 4 bytes hole, try to pack */
        mempool_t *                rq_pool;        /*    24     8 */
        wait_queue_head_t          wait[2];        /*    32    48 */
        /* --- cacheline 1 boundary (64 bytes) was 16 bytes ago --- */
        /* size: 80, cachelines: 2 */
        /* sum members: 76, holes: 1, sum holes: 4 */
        /* last cacheline: 16 bytes */

Now lets expand its pointers, just one, mempool_t, and one that doesn’t have lots of pointers, to fit into this blog entry:

$ pahole --expand_pointers -C request_list fs/super.o
struct request_list {
        int                        count[2];        /*     0     8 */
        int                        starved[2];      /*     8     8 */
        int                        elvpriv;         /*    16     4 */
        /* XXX 4 bytes hole, try to pack */
        /* typedef mempool_t */ struct mempool_s {
                spinlock_t         lock;
                int                min_nr;
                int                curr_nr;
                void *             *elements;
                void *             pool_data;
                /* typedef mempool_alloc_t */ void * (*alloc)(gfp_t, void *);
                /* typedef mempool_free_t */ void (*free)(void *, void *);
                wait_queue_head_t  wait;
                /* --- cacheline 1 boundary (64 bytes) was 8 bytes ago --- */
        } *rq_pool; /*    24     8 */
        wait_queue_head_t          wait[2];         /*    32    48 */
        /* --- cacheline 1 boundary (64 bytes) was 16 bytes ago --- */
        /* size: 80, cachelines: 2 */
        /* sum members: 76, holes: 1, sum holes: 4 */
        /* last cacheline: 16 bytes */

It even seems to naturally avoid expanding opaque types, i.e. struct forward declarations, etc, so if header file with structs in the ABI is well written it just doesn’t goes down the non-ABI type rabbit hole, which seems the right thing to do.

Use it with –expand_types and the complete picture can be seen. And in the Linux kernel it is, humm, big:

acme@mica build]$ pahole --quiet --expand_pointers -C inode fs/super.o | wc -l
[acme@mica build]$ pahole --quiet --expand_pointers --expand_types -C inode fs/super.o | wc -l
[acme@mica build]$

One explanation for this last example is needed tho: –expand_types expands a type possibly many times, as many as there are members of its type, –expand_pointers, on the other hand, only expands any given type once, for the first member that is a pointer to this type.

This is because –expand_types was implemented to help in finding what was the field at some offset from complex data structures with a deep hierarchy, –expand_pointers, on the other hand, used most of the –expand_types code, but was implemented to help in finding ABI breakage deep inside the type.

I’ll eventually implement a config option to tell that only the first type should be expanded in the –expand_types case, make that a shell script that takes as parameters two files, call pahole for the two, do a diff and show where the ABI stopped being a virgin 8-)

Syndicated 2007-07-07 18:38:03 from Arnaldo's Ramblings

Inheriting namespaces

I’d prefer real state space, but for now I’ll have to seattle for C++ namespaces as the only inheritance to tap into. The dwarves now are able to show the wonders of goat guts, oops, of modern C++ techniques hidden behind layers of type within types within templates within namespaces represented in DWARF tags. This time there will be no tool sample output, they wouldn’t be pretty. Just be warned, pull from my kernel.org repo and be sure you have already digested everything you ate in at least two hours. Remember: you’ve been warned.

P.S. yes, there is a typoish message to friends in the northern hemisphere, far from the brazilian winter :o )

Syndicated 2007-05-26 01:07:25 from Arnaldo's Ramblings

No song for 1.0

As part of the review process to get dwarves into Fedora I released 1.0. Numbers are not that much meaningful anyway. In a previous life, as a package pac-man (sorry about the pun, too lame…) in a jungle distro I always looked forward for one dot zero. Even for nfs-server! So, here it goes, one-dot-zero! In a Fedora mirror some day soon!

P.S.: rpms in the usual place.

Syndicated 2007-05-22 19:37:03 from Arnaldo's Ramblings

Virtual doesn’t hurts as much as Real Life: !wii session

Today went with friends to do some non-Wii bowling. What can I say? My fingers are hurting. Yeah, I suck in real life. Nah, back at home listening to some torrents, nevermind I have the CDs for 20 years, just that those Metallica songs are so buried in layers of dust I found it easier to pound my ears with something freshly downloaded 8-) My 5 years old niece still knows nothing about torrents and stuff, but she beats me hands down on the not-so-heavy-as-real-life Wii bowling. She will learn, but lets not hurry her…

Syndicated 2007-05-19 03:27:41 from Arnaldo's Ramblings


- Paper for OLS about the dwarf debugging tools written, will be the badly needed documentation for these tools as soon as it is released. Will continue improving it even after presentation at OLS.
- Several new pahole options, for instance look at this
- libdwarves now uses libdwfl, so works well with RELA architectures such as x86-64

Syndicated 2007-05-10 22:16:57 from Arnaldo's Ramblings

Kernel hacking mostly…

The “mostly” allows me to celebrate my most recent non-kernel-hacking, at least personally, interesting news: I’m keeping up with jogging, now trying to get past the confortable part of jogging, over-achiever, not really… only 17:44.06 on a 3.3 KM circuit at Barigui, if only da beers would keep some distance from myself…

Syndicated 2007-03-30 20:50:37 from Arnaldo's Ramblings

libdwfl conversion

Finally I found time to convert dwarves to use the more active libdwfl library found in elfutils, thanks to Ulrich Drepper and Roland McGrath we can forget about most of the low level details related to RELA architectures and get the dwarves to work with kernel modules on architectures such as x86_64, where we need to do relocation work on the .debug_info section, now just enjoy pahole & friends working on more targets.

There is still some work to do on two of the dwarves that handle more than one target, codiff and ctracer, where dwfl_default_argp is not enough as it only handles one side of the game needed for these two dwarves, we need to open another set to compare in the case of codiff and another set for resolving the kprobes functions in the case of ctracer.

Syndicated 2007-03-30 20:39:56 from Arnaldo's Ramblings

Some hops thru the USA

So I’ll be flying two hours from now, with a little layover at Sao Paulo trying to be cautions about the brazilian hellish flying russian roulette, that should be past news by now, but would you take any chances on a _real_ russian roulette? me neither.

So I’ll be on the east coast for some 4 days and then on another hop, close to the canadian border in the west coast for 4 days more, then back home, so expect no news (or just a few, to pester ya with news about you know what ;) ).

Syndicated 2007-02-03 16:14:11 from Arnaldo's Ramblings

News fom dwarves land


Now we can also move bitfields around to combine it with other fields, moving a list of members with the same offset (the definition of a bitfield) to after other members that have holes bigger than the size of the (possibly type demoted) list of members comprising a bitfield, also there is a new pahole command line option to show all the steps involved in reorganizing a struct, –show_reorg_steps, look here for how it looks like.

Also related to this pahole news is the fact that the class__reorganize() & supporting code was moved to libdwarves.so, where its is made useful for the other dwarves, which leads us to the next dwarves news:


The class tracer now is past its stone age, getting fast to an exciting iron age: printk was replaced by a not so carefully (as in SMP?!?! I only have a dual pentium 100 machine here, damn!) stolen from blktrace (I hope that the UTT effort is merged sooner than later so that I can use it), namely using debugfs + relay to, ho hum, relay the internal state of the class (a funny name for structs) being traced to userspace, ah, since I’m talking about the internal state: the way that it is collected at the probe points now is much, much improved, looking like blktrace, except for the fact that the data being collected is the subset of the members that are “reducible” to a basic type (signed and unsigned int, long, char, long long), which, for now, its just the basic types, but will be shortly augmented by a set of “reducers” for things like spinlock_t, wait_queue_head_t, etc.

Ok, lets break the paragraph a bit to get away from my problem with writing: very long paragraphs…

Back to how ctracer packs the internal state: have you seen how pahole can reorganize structs to fill holes and reduce the structure size? Yeah, we just look at all the struct members, looking for DW_TAG_basic_type members and leave just those, then call class__reorganize() and the struct gets in the best possible layout to save space at each probe point.

Ctracer also uses the same opportunity to generate a tool called ctracer2ostra.c, that is used to convert the binary traces to a colon separated list of formatted values easily parsable by ostra-cg, the callgraph tool that produces nifty buzzword soups (CSS, javascript, ajax in the future, cross-referencing with LXR, python matplotlib graphs of values that each traced class member got in the course of tracing, and plottings for the times each of the “class” “methods” took).

The new improved process for this is detailed in the README.ctracer file, but to give you a quick glance of what it involves:

rpm -ivh http://oops.ghostprotocols.net:81/acme/dwarves/rpm/libdwarves1-0-14.i386.rpm http://oops.ghostprotocols.net:81/acme/dwarves/rpm/dwarves-0-14.i386.rpm
mkdir foo
cd foo
ln -s /usr/lib/ctracer/* .
make CLASS=sock
insmod ctracer.ko
# do some networking kung foo fighting activity or change CLASS to your preferred struct (but beware, as we use do_gettimeofday (how lame) you can get some crashes if tracing something more fishy than struct sock methods)
cat /sys/kernel/debug/ctracer.o > /tmp/ctracer.log
rmmod ctracer.ko
make callgraphs
# Enjoy things like this, where I was bold enough to try this thing on my main machine while doing all sorts of stuff, believe me, it was just one or two dozen harmless oopses, kprobes trying to justify its paycheck!

And to top it all I’ve been getting help from Davi Arnaut on the global variables and DW_AT_location front, were he contributed basic support for where variables are in the computer memory pyramid (Registers? Stack?), he even wrote the first non-acme dwarf, pglobal, that shows all the global variables in your dear project binary files, now its just a matter of extending this to use libebl and get the register names for each arch to go to the next ctracer level, where we’ll stop using costly Jay-probes and get the elusive parameters directly from their hiding place!

Syndicated 2007-02-02 21:17:28 from Arnaldo's Ramblings

Kill-a-hole: Reorganizing struct layouts

Say a project has this struct:

struct cheese {
char name[52];
char a;
int b;
char c;
int d;
short e;

And we want to see how the layout looks like more precisely:

[acme@filo examples]$ pahole swiss_cheese cheese
/* <11b> /home/acme/git/pahole/examples/swiss_cheese.c:3 */
struct cheese {
char name[52]; /* 0 52 */
char a; /* 52 1 */

/* XXX 3 bytes hole, try to pack */

int b; /* 56 4 */
char c; /* 60 1 */

/* XXX 3 bytes hole, try to pack */

/* — cacheline 1 boundary (64 bytes) — */
int d; /* 64 4 */
short int e; /* 68 2 */
}; /* size: 72, cachelines: 2 */
/* sum members: 64, holes: 2, sum holes: 6 */
/* padding: 2 */
/* last cacheline: 8 bytes */
[acme@filo examples]$

Heck, what a swiss cheese! Surely we can do better, huh? Lets ask pahole for a little help:

[acme@filo examples]$ pahole -kV swiss_cheese cheese
/* moving c(size=1) to after a(offset=52, size=1, hole=3) */
/* moving e(size=2) to after c(offset=53, size=1, hole=2) */

/* <11b> /home/acme/git/pahole/examples/swiss_cheese.c:3 */
struct cheese {
char name[52]; /* 0 52 */
char a; /* 52 1 */
char c; /* 53 1 */
short int e; /* 54 2 */
int b; /* 56 4 */
int d; /* 60 4 */
/* — cacheline 1 boundary (64 bytes) — */
}; /* size: 64, cachelines: 1 */
/* saved 8 bytes and 1 cacheline! */
[acme@filo examples]$

Much better now, no?

Ok, lets try something more interesting, like some Linux kernel structs, the output is big, so here are some links for some structs that spent some jiffies on pahole’s spa:

Type demotion of bigger than needed bitfields will help getting more saved :-)

Syndicated 2007-01-30 15:20:26 from Arnaldo's Ramblings

150 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!