Older blog entries for acme (starting at number 152)

Some hops thru the USA

So I’ll be flying two hours from now, with a little layover at Sao Paulo trying to be cautions about the brazilian hellish flying russian roulette, that should be past news by now, but would you take any chances on a _real_ russian roulette? me neither.

So I’ll be on the east coast for some 4 days and then on another hop, close to the canadian border in the west coast for 4 days more, then back home, so expect no news (or just a few, to pester ya with news about you know what ;) ).

Syndicated 2007-02-03 16:14:11 from Arnaldo's Ramblings

News fom dwarves land

Pahole:

Now we can also move bitfields around to combine it with other fields, moving a list of members with the same offset (the definition of a bitfield) to after other members that have holes bigger than the size of the (possibly type demoted) list of members comprising a bitfield, also there is a new pahole command line option to show all the steps involved in reorganizing a struct, –show_reorg_steps, look here for how it looks like.

Also related to this pahole news is the fact that the class__reorganize() & supporting code was moved to libdwarves.so, where its is made useful for the other dwarves, which leads us to the next dwarves news:

Ctracer:

The class tracer now is past its stone age, getting fast to an exciting iron age: printk was replaced by a not so carefully (as in SMP?!?! I only have a dual pentium 100 machine here, damn!) stolen from blktrace (I hope that the UTT effort is merged sooner than later so that I can use it), namely using debugfs + relay to, ho hum, relay the internal state of the class (a funny name for structs) being traced to userspace, ah, since I’m talking about the internal state: the way that it is collected at the probe points now is much, much improved, looking like blktrace, except for the fact that the data being collected is the subset of the members that are “reducible” to a basic type (signed and unsigned int, long, char, long long), which, for now, its just the basic types, but will be shortly augmented by a set of “reducers” for things like spinlock_t, wait_queue_head_t, etc.

Ok, lets break the paragraph a bit to get away from my problem with writing: very long paragraphs…

Back to how ctracer packs the internal state: have you seen how pahole can reorganize structs to fill holes and reduce the structure size? Yeah, we just look at all the struct members, looking for DW_TAG_basic_type members and leave just those, then call class__reorganize() and the struct gets in the best possible layout to save space at each probe point.

Ctracer also uses the same opportunity to generate a tool called ctracer2ostra.c, that is used to convert the binary traces to a colon separated list of formatted values easily parsable by ostra-cg, the callgraph tool that produces nifty buzzword soups (CSS, javascript, ajax in the future, cross-referencing with LXR, python matplotlib graphs of values that each traced class member got in the course of tracing, and plottings for the times each of the “class” “methods” took).

The new improved process for this is detailed in the README.ctracer file, but to give you a quick glance of what it involves:

rpm -ivh http://oops.ghostprotocols.net:81/acme/dwarves/rpm/libdwarves1-0-14.i386.rpm http://oops.ghostprotocols.net:81/acme/dwarves/rpm/dwarves-0-14.i386.rpm
mkdir foo
cd foo
ln -s /usr/lib/ctracer/* .
make CLASS=sock
insmod ctracer.ko
# do some networking kung foo fighting activity or change CLASS to your preferred struct (but beware, as we use do_gettimeofday (how lame) you can get some crashes if tracing something more fishy than struct sock methods)
cat /sys/kernel/debug/ctracer.o > /tmp/ctracer.log
rmmod ctracer.ko
make callgraphs
# Enjoy things like this, where I was bold enough to try this thing on my main machine while doing all sorts of stuff, believe me, it was just one or two dozen harmless oopses, kprobes trying to justify its paycheck!

And to top it all I’ve been getting help from Davi Arnaut on the global variables and DW_AT_location front, were he contributed basic support for where variables are in the computer memory pyramid (Registers? Stack?), he even wrote the first non-acme dwarf, pglobal, that shows all the global variables in your dear project binary files, now its just a matter of extending this to use libebl and get the register names for each arch to go to the next ctracer level, where we’ll stop using costly Jay-probes and get the elusive parameters directly from their hiding place!

Syndicated 2007-02-02 21:17:28 from Arnaldo's Ramblings

Kill-a-hole: Reorganizing struct layouts

Say a project has this struct:

struct cheese {
char name[52];
char a;
int b;
char c;
int d;
short e;
};

And we want to see how the layout looks like more precisely:

[acme@filo examples]$ pahole swiss_cheese cheese
/* <11b> /home/acme/git/pahole/examples/swiss_cheese.c:3 */
struct cheese {
char name[52]; /* 0 52 */
char a; /* 52 1 */

/* XXX 3 bytes hole, try to pack */

int b; /* 56 4 */
char c; /* 60 1 */

/* XXX 3 bytes hole, try to pack */

/* — cacheline 1 boundary (64 bytes) — */
int d; /* 64 4 */
short int e; /* 68 2 */
}; /* size: 72, cachelines: 2 */
/* sum members: 64, holes: 2, sum holes: 6 */
/* padding: 2 */
/* last cacheline: 8 bytes */
[acme@filo examples]$

Heck, what a swiss cheese! Surely we can do better, huh? Lets ask pahole for a little help:

[acme@filo examples]$ pahole -kV swiss_cheese cheese
/* moving c(size=1) to after a(offset=52, size=1, hole=3) */
/* moving e(size=2) to after c(offset=53, size=1, hole=2) */

/* <11b> /home/acme/git/pahole/examples/swiss_cheese.c:3 */
struct cheese {
char name[52]; /* 0 52 */
char a; /* 52 1 */
char c; /* 53 1 */
short int e; /* 54 2 */
int b; /* 56 4 */
int d; /* 60 4 */
/* — cacheline 1 boundary (64 bytes) — */
}; /* size: 64, cachelines: 1 */
/* saved 8 bytes and 1 cacheline! */
[acme@filo examples]$

Much better now, no?

Ok, lets try something more interesting, like some Linux kernel structs, the output is big, so here are some links for some structs that spent some jiffies on pahole’s spa:

Type demotion of bigger than needed bitfields will help getting more saved :-)

Syndicated 2007-01-30 15:20:26 from Arnaldo's Ramblings

Unfolding structs

Check the git cset comment! New rpms are available at the usual place with this new feature, for the curious, here is struct task_struct unfolded.

Syndicated 2007-01-29 15:17:02 from Arnaldo's Ramblings

ctracer made easy

Make sure you have the kernel-debuginfo package installed!

rpm -ivh http://oops.ghostprotocols.net:81/acme/dwarves/rpm/libdwarves1-0-9.i386.rpm http://oops.ghostprotocols.net:81/acme/dwarves/rpm/dwarves-0-9.i386.rpm
mkdir foo
cd foo
ln -s /usr/lib/ctracer/Makefile .
ln -s /usr/lib/ctracer/ctracer_jprobe.c .
make CLASS=sock
insmod ctracer.ko
# do some networking activity or just move the mouse :-)
dmesg # or tail /var/log/messages
rmmod ctracer

Now to work on kahole, kill a hole, a tool that will reorganize structs to kill holes and that will provide the bits needed for struct “views”, i.e. to specify a list of fields in a struct that are of interest for collection at each probe point and that will also be used to create a userspace utility that will read the relay channel (/sys/kernel/ctracer0) and generate what ostra-cg, the callgraph tool from the OSTRA days needs to generate things such as this session collecting struct task_struct methods calls in the Linux kernel, ah, ostra-cg also produces this methods statistics and from this page you get plottings such as this one for sched_
fork calls
, workload is forgotten right now, but as soon as I get my new test machine we’ll have more interesting graphics :-)

Syndicated 2007-01-27 21:20:21 from Arnaldo's Ramblings

libdwarves (mostly) explained

In response to a comment I’ve added comments to the ctracer tool source code in an attempt to help people interested in helping improving these tools, check it out!

Syndicated 2007-01-20 15:04:30 from Arnaldo's Ramblings

How many functions receive as a parameter a pointer to a struct?

Implemented a new command line option in pahole: –nr_members, look here for the result using a kernel built for qemu.

Syndicated 2007-01-13 16:04:08 from Arnaldo's Ramblings

Support for more C++ tags

Added support for DW_TAG_reference and DW_AT_specification, still have to work on DW_TAG_namespace and rework the list of tags to make classes within classes to properly be shown in the tools, nowadays its just a flat namespace, which has not been a problem for the tools written so far, but for pdwtags, the dwarf I’m working on now, that will do a complete dump of all tags it is a must.

Syndicated 2007-01-13 13:51:47 from Arnaldo's Ramblings

connection_sock work going on nicely, today sent patches for lots of network families, tomorrow, finally, will try to test the ax25/netrom/rose big patch with bpqether and the ax25 utilities, if all goes well, send the patches to Dave and turn the chainsaw to net/bluetooth/, that will be interesting with all the internal proto infrastructure it has, lets see if we can take advantage of its approach in other protos or if getting what the net/ipv4/ has for a similar purpose is better.

Added my TODO list to this page.

143 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!