Systemtap 0.9 - Markers everywhere
We recently released Systemtap 0.9 and one of the nice new features included is the user space markers that Stan Cox has been working on. They were designed so that they should be compatible with dtrace static user space markers, so you can immediately take advantage of them if your program already has those included. It even comes with a little dtrace python script wrapper that automagically does the right thing during the build. A nice example of that is postgresql, which has a set of markers to observe transactions, database locks, etc. All you have to do is recompile postgresql with –enable-dtrace and tada, out roll systemtap enabled markers that you can use for getting some high level events from your database. Like for example how many and how long transactions take:
$ stap -x `pgrep -n postgres` postgresql-transactions.stp
committed transactions:
transaction id: time
34: 1593213ns
36: 2817146ns
37: 1463901ns
38: 1427854ns
aborted transactions: 4
We are trying to get some of these static markers activated in packages compiled for Fedora as SystemtapStaticProbes F11 Feature, so you can use them out of the box.
But you can also add your own markers to existing code. Daniel Tralamazza has been experimenting with a small glibc patch to add markers around various pthread mutexes, for doing userland synchronization primitives analysis. Then you can easily get things like the top 10 most shared locks.