poor man's profiler

rationale

Sampling tools like oprofile or dtrace's profile provider don't really provide methods to see what [multithreaded] programs are blocking on - only where they spend CPU time. Though there exist advanced techniques (such as systemtap and dtrace call level probes), it is overkill to build upon that. Poor man doesn't have time. Poor man needs food.

method

For a poor developer to understand what a program is doing, he needs to see stacks. Once upon a time (back in Linux 2.4) there was a 'pstack' tool for that, Solaris has it too.

Modern Linux systems though do not have such facilities, and one needs to improvise, like.. use debuggers - they can walk threads and provide stacks.

technology

Getting stacks:

gdb -ex "set pagination 0" -ex "thread apply all bt" \ --batch -p $(pidof mysqld)

(echo "set pagination 0"; echo "thread apply all bt"; echo "quit"; cat /dev/zero ) | gdb -p $(pidof mysqld)

BEGIN { s = ""; } /Thread/ { print s; s = ""; } /^\#/ { if ($3 != "in") { $4 = $2 } } /^\#/ { if (s != "" ) { s = s "," $4} else { s = $4 } } END { print s }

#!/bin/bash nsamples=1 sleeptime=0 pid=$(pidof mysqld) for x in $(seq 1 $nsamples) do gdb -ex "set pagination 0" -ex "thread apply all bt" -batch -p $pid sleep $sleeptime done | \ awk ' BEGIN { s = ""; } /^Thread/ { print s; s = ""; } /^\#/ { if (s != "" ) { s = s "," $4} else { s = $4 } } END { print s }' | \ sort | uniq -c | sort -r -n -k 1,1

output

success stories and references

Or for version-impaired (gdb 6.3 and older):Collapsing traces (awk!):Full technology demonstration: