Debugging is a complex task. Most of the work time, developers spend on debugging and it is important to be familiar with many debugging tools In Linux, the native debugger is GDB and it is command line based and looks ugly and primitive. Many developers , especially those who moved from windows and worked with tools like visual studio, don’t give GDB a chance. GDB is command line based and you need to learn the commands and it looks like debugging in the 80’s but there are some tasks you can only do with GDB We will use the following code: #include<stdio.h> #define addnum(a,b) ((a)+(b)) #define max(a,b) (a)>(b)?(addnum(a,100)):(addnum(b,50)) int a1=10,a2=20; void f2() { printf("X"); } void print_dbg() { char buf[15]; printf("a1=%d a2=%d

",a1,a2); strcpy(buf, "hello world"); } int add(int a, int b) { int c; c= max(a+90,b+20); c+=b; return c; } void f1() { int i; int *p=NULL; print_dbg(); i=add(80,90); for(i=0;i<100;i++) { if(i % 20 == 0) a1++; f2(); } *p=100; } int arr[1000]; void main() { arr[0]=1; f1(); printf("hello %d

",arr[0]); } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 #include<stdio.h> #define addnum(a,b) ((a)+(b)) #define max(a,b) (a)>(b)?(addnum(a,100)):(addnum(b,50)) int a1 = 10 , a2 = 20 ; void f2 ( ) { printf ( "X" ) ; } void print_dbg ( ) { char buf [ 15 ] ; printf ( "a1=%d a2=%d

" , a1 , a2 ) ; strcpy ( buf , "hello world" ) ; } int add ( int a , int b ) { int c ; c = max ( a + 90 , b + 20 ) ; c += b ; return c ; } void f1 ( ) { int i ; int * p = NULL ; print_dbg ( ) ; i = add ( 80 , 90 ) ; for ( i = 0 ; i < 100 ; i ++ ) { if ( i % 20 == 0 ) a1 ++ ; f2 ( ) ; } * p = 100 ; } int arr [ 1000 ] ; void main ( ) { arr [ 0 ] = 1 ; f1 ( ) ; printf ( "hello %d

" , arr [ 0 ] ) ; }

1 – Calling functions while debugging

While GDB stops on a breakpoint or stepping the code you can call a function with or without parameters.

(gdb) b add Breakpoint 1 at 0x400654: file ./a.c, line 21. (gdb) r Starting program: myapp a1=10 a2=20 Breakpoint 1, add (a=80, b=90) at ./a.c:21 21 int c= a+90; (gdb) call print_dbg() a1=10 a2=20 (gdb) delete breakpoints Delete all breakpoints? (y or n) y (gdb) b a.c:34 Breakpoint 2 at 0x4006bd: file ./a.c, line 34. (gdb) c Continuing. Breakpoint 2, f1 () at ./a.c:34 34 a1++; (gdb) call add(2,3) $1 = 95 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 ( gdb ) b add Breakpoint 1 at 0x400654 : file . / a . c , line 21. ( gdb ) r Starting program : myapp a1 = 10 a2 = 20 Breakpoint 1 , add ( a = 80 , b = 90 ) at . / a . c : 21 21 int c = a + 90 ; ( gdb ) call print_dbg ( ) a1 = 10 a2 = 20 ( gdb ) delete breakpoints Delete all breakpoints ? ( y or n ) y ( gdb ) b a . c : 34 Breakpoint 2 at 0x4006bd : file . / a . c , line 34. ( gdb ) c Continuing . Breakpoint 2 , f1 ( ) at . / a . c : 34 34 a1 ++ ; ( gdb ) call add ( 2 , 3 ) $ 1 = 95

2 – Watch Points

One of the great features GDB has is the ability to break while something is changed. It can be a variable, an address or a register:

Watch variable change:

(gdb) watch a1 Hardware watchpoint 1: a1 (gdb) r Starting program: /home/developer/examples/debuggingExamples/debugtests/app a1=10 a2=20 Hardware watchpoint 1: a1 Old value = 10 New value = 11 f1 () at ./a.c:35 35 f2(); 1 2 3 4 5 6 7 8 9 10 11 12 ( gdb ) watch a1 Hardware watchpoint 1 : a1 ( gdb ) r Starting program : / home / developer / examples / debuggingExamples / debugtests / app a1 = 10 a2 = 20 Hardware watchpoint 1 : a1 Old value = 10 New value = 11 f1 ( ) at . / a . c : 35 35 f2 ( ) ;

Watch address change

We first print the array address :

(gdb) p &arr $1 = (int (*)[1000]) 0x601080 <arr> 1 2 ( gdb ) p & arr $ 1 = ( int ( * ) [ 1000 ] ) 0x601080 < arr >

then ask GDB to stop while the content in that address changed

(gdb) watch *0x601080 Hardware watchpoint 1: *0x601080 (gdb) r Starting program: /home/developer/examples/debuggingExamples/debugtests/app Hardware watchpoint 1: *0x601080 Old value = 0 New value = 1 main () at ./a.c:43 43 f1(); 1 2 3 4 5 6 7 8 9 10 11 ( gdb ) watch * 0x601080 Hardware watchpoint 1 : * 0x601080 ( gdb ) r Starting program : / home / developer / examples / debuggingExamples / debugtests / app Hardware watchpoint 1 : * 0x601080 Old value = 0 New value = 1 main ( ) at . / a . c : 43 43 f1 ( ) ;

Watch register change

The same can be done with register.

(gdb) watch $rax Watchpoint 1: $rax (gdb) c Continuing. Watchpoint 1: $rax Old value = 4196067 New value = 0 0x00000000004006f6 in main () at ./a.c:43 43 f1(); 1 2 3 4 5 6 7 8 9 10 11 12 ( gdb ) watch $ rax Watchpoint 1 : $ rax ( gdb ) c Continuing . Watchpoint 1 : $ rax Old value = 4196067 New value = 0 0x00000000004006f6 in main ( ) at . / a . c : 43 43 f1 ( ) ;

3 – Debugging Preprocessor Macros

Debugging macros is hard, It is preferred to work with inline function ,

While compiling the code, you can use -E to see only preprocessor output. You can ask GDB to expand a macro to see how it is handled on runtime:

(gdb) b add Breakpoint 1 at 0x400654: file ./a.c, line 28. (gdb) r Starting program: /home/developer/examples/debuggingExamples/debugtests/app a1=10 a2=20 Breakpoint 1, add (a=80, b=90) at ./a.c:28 28 c= max(a+90,b+20); (gdb) macro expand max(a+90,b+20) expands to: (a+90)>(b+20)?(((a+90)+(100))):(((b+20)+(50))) 1 2 3 4 5 6 7 8 9 10 ( gdb ) b add Breakpoint 1 at 0x400654 : file . / a . c , line 28. ( gdb ) r Starting program : / home / developer / examples / debuggingExamples / debugtests / app a1 = 10 a2 = 20 Breakpoint 1 , add ( a = 80 , b = 90 ) at . / a . c : 28 28 c = max ( a + 90 , b + 20 ) ; ( gdb ) macro expand max ( a + 90 , b + 20 ) expands to : ( a + 90 ) > ( b + 20 ) ? ( ( ( a + 90 ) + ( 100 ) ) ) : ( ( ( b + 20 ) + ( 50 ) ) )

4 – Inspecting the code without running

If you write fault handlers in your code and the program crashed, the faulty address is printed to the log:

signal 11 received EIP address = 0x400702 Segmentation fault (core dumped) 1 2 3 signal 11 received EIP address = 0x400702 Segmentation fault ( core dumped )

You can load the program to GDB without running it:

# gdb ./app 1 # gdb ./app

And ask the debugger to list the faulty address:

(gdb) l *0x400702 0x400702 is in f1 (./a.c:45). 40 { 41 if(i % 20 == 0) 42 a1++; 43 f2(); 44 } 45 *p=100; 46 } 47 int arr[1000]; 48 49 void main() 1 2 3 4 5 6 7 8 9 10 11 12 ( gdb ) l * 0x400702 0x400702 is in f1 ( . / a . c : 45 ) . 40 { 41 if ( i % 20 == 0 ) 42 a1 ++ ; 43 f2 ( ) ; 44 } 45 * p = 100 ; 46 } 47 int arr [ 1000 ] ; 48 49 void main ( )

This is very useful if the program is running on a different platform (i.e embedded system) , you can use the cross debugger (arm-linux-gdb)

5 – Inspecting core dumps

While the program crashed , the system can dump its state to a core dump file if you run:

# ulimit -c unlimited 1 # ulimit -c unlimited

Run the program, it will crash and create a core file

# ./app a1=10 a2=20 Segmentation fault (core dumped) 1 2 3 # ./app a1 = 10 a2 = 20 Segmentation fault ( core dumped )

Load the program with the core dump to GDB:

# gdb ./app ./core GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from ./app...done. [New LWP 34577] Core was generated by `./app'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x0000000000400706 in f1 () at ./a.c:45 45 *p=100; (gdb) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 # gdb ./app ./core GNU gdb ( Ubuntu 7.11.1 - 0ubuntu1 ~ 16.5 ) 7.11.1 Copyright ( C ) 2016 Free Software Foundation , Inc . License GPLv3 + : GNU GPL version 3 or later < http : //gnu.org/licenses/gpl.html> This is free software : you are free to change and redistribute it . There is NO WARRANTY , to the extent permitted by law . Type "show copying" and "show warranty" for details . This GDB was configured as "x86_64-linux-gnu" . Type "show configuration" for configuration details . For bug reporting instructions , please see : < http : //www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at : < http : //www.gnu.org/software/gdb/documentation/>. For help , type "help" . Type "apropos word" to search for commands related to "word" . . . Reading symbols from . / app . . . done . [ New LWP 34577 ] Core was generated by ` . / app ' . Program terminated with signal SIGSEGV , Segmentation fault . #0 0x0000000000400706 in f1 () at ./a.c:45 45 * p = 100 ; ( gdb )

You can see the problem and now you can inspect the state at the crashing moment:

backtrace:

(gdb) bt #0 0x0000000000400706 in f1 () at ./a.c:45 #1 0x0000000000400727 in main () at ./a.c:52 1 2 3 ( gdb ) bt #0 0x0000000000400706 in f1 () at ./a.c:45 #1 0x0000000000400727 in main () at ./a.c:52

registers:

(gdb) info registers rax 0x0 0 rbx 0x0 0 rcx 0xfbad2a84 4222429828 rdx 0x58 88 rsi 0x7fce21600780 140523299932032 rdi 0x7fce215ff620 140523299927584 rbp 0x7fff73ecdd90 0x7fff73ecdd90 rsp 0x7fff73ecdd80 0x7fff73ecdd80 r8 0x7fce21600780 140523299932032 r9 0x7fce21807700 140523302057728 r10 0x1d6 470 r11 0x7fce212ab290 140523296436880 r12 0x4004e0 4195552 r13 0x7fff73ecde80 140735138291328 r14 0x0 0 r15 0x0 0 rip 0x400706 0x400706 <f1+132> eflags 0x10202 [ IF RF ] cs 0x33 51 ss 0x2b 43 ds 0x0 0 es 0x0 0 fs 0x0 0 gs 0x0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 ( gdb ) info registers rax 0x0 0 rbx 0x0 0 rcx 0xfbad2a84 4222429828 rdx 0x58 88 rsi 0x7fce21600780 140523299932032 rdi 0x7fce215ff620 140523299927584 rbp 0x7fff73ecdd90 0x7fff73ecdd90 rsp 0x7fff73ecdd80 0x7fff73ecdd80 r8 0x7fce21600780 140523299932032 r9 0x7fce21807700 140523302057728 r10 0x1d6 470 r11 0x7fce212ab290 140523296436880 r12 0x4004e0 4195552 r13 0x7fff73ecde80 140735138291328 r14 0x0 0 r15 0x0 0 rip 0x400706 0x400706 < f1 + 132 > eflags 0x10202 [ IF RF ] cs 0x33 51 ss 0x2b 43 ds 0x0 0 es 0x0 0 fs 0x0 0 gs 0x0 0

stack (display 20 words from the stack pointer address)

(gdb) x/20w $rsp 0x7fff73ecdd80: 4196176 100 0 0 0x7fff73ecdd90: 1944903072 32767 4196135 0 0x7fff73ecdda0: 4196176 0 556116016 32718 0x7fff73ecddb0: 0 0 1944903304 32767 0x7fff73ecddc0: 562207904 1 4196111 0 1 2 3 4 5 6 ( gdb ) x / 20w $ rsp 0x7fff73ecdd80 : 4196176 100 0 0 0x7fff73ecdd90 : 1944903072 32767 4196135 0 0x7fff73ecdda0 : 4196176 0 556116016 32718 0x7fff73ecddb0 : 0 0 1944903304 32767 0x7fff73ecddc0 : 562207904 1 4196111 0

and more

6 – Automation

GDB support automation. For example , you can ask GDB to do something on a breakpoint hit:

(gdb) b add Breakpoint 1 at 0x400654: file ./a.c, line 28. (gdb) commands Type commands for breakpoint(s) 1, one per line. End with a line saying just "end". >print a1 >call print_dbg() >end (gdb) r Starting program: /home/developer/examples/debuggingExamples/debugtests/app a1=10 a2=20 Breakpoint 1, add (a=80, b=90) at ./a.c:28 28 c= max(a+90,b+20); $1 = 10 a1=10 a2=20 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 ( gdb ) b add Breakpoint 1 at 0x400654 : file . / a . c , line 28. ( gdb ) commands Type commands for breakpoint ( s ) 1 , one per line . End with a line saying just "end" . > print a1 > call print_dbg ( ) > end ( gdb ) r Starting program : / home / developer / examples / debuggingExamples / debugtests / app a1 = 10 a2 = 20 Breakpoint 1 , add ( a = 80 , b = 90 ) at . / a . c : 28 28 c = max ( a + 90 , b + 20 ) ; $ 1 = 10 a1 = 10 a2 = 20

You can define it in .gdbinit file (in your debugging folder) and GDB loads and execute it automatically

.gdbinit file:

echo "hi" b add commands print a1 call print_dbg() 1 2 3 4 5 6 echo "hi" b add commands print a1 call print_dbg ( )

While running the debugger the file is executed

# gdb ./app GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from ./app...done. "hi"Breakpoint 1 at 0x400654: file ./a.c, line 28. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 # gdb ./app GNU gdb ( Ubuntu 7.11.1 - 0ubuntu1 ~ 16.5 ) 7.11.1 Copyright ( C ) 2016 Free Software Foundation , Inc . License GPLv3 + : GNU GPL version 3 or later < http : //gnu.org/licenses/gpl.html> This is free software : you are free to change and redistribute it . There is NO WARRANTY , to the extent permitted by law . Type "show copying" and "show warranty" for details . This GDB was configured as "x86_64-linux-gnu" . Type "show configuration" for configuration details . For bug reporting instructions , please see : < http : //www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at : < http : //www.gnu.org/software/gdb/documentation/>. For help , type "help" . Type "apropos word" to search for commands related to "word" . . . Reading symbols from . / app . . . done . "hi" Breakpoint 1 at 0x400654 : file . / a . c , line 28.

7 – GDB Macros

With .gdbinit file you can also define a new commands (macros) for your debugging sessions. For example if you write the following to .gdbinit:

define irg info registers info threads end define pdb print a1 print a2 call print_dbg() end 1 2 3 4 5 6 7 8 9 10 define irg info registers info threads end define pdb print a1 print a2 call print_dbg ( ) end

It defines 2 new macros you can use – pdb , irg

(gdb) pdb $2 = 10 $3 = 20 a1=10 a2=20 (gdb) irg rax 0x0 0 rbx 0x0 0 rcx 0x6f77206f6c6c6568 8031924123371070824 rdx 0x7ffff7dd3780 140737351858048 rsi 0x5a 90 rdi 0x50 80 rbp 0x7fffffffdb50 0x7fffffffdb50 rsp 0x7fffffffdb50 0x7fffffffdb50 r8 0x0 0 r9 0xd 13 r10 0x0 0 r11 0x246 582 r12 0x4004e0 4195552 r13 0x7fffffffdc60 140737488346208 r14 0x0 0 r15 0x0 0 rip 0x400654 0x400654 <add+10> eflags 0x246 [ PF ZF IF ] cs 0x33 51 ss 0x2b 43 ds 0x0 0 es 0x0 0 fs 0x0 0 gs 0x0 0 Id Target Id Frame * 1 process 35330 "app" add (a=80, b=90) at ./a.c:28 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 ( gdb ) pdb $ 2 = 10 $ 3 = 20 a1 = 10 a2 = 20 ( gdb ) irg rax 0x0 0 rbx 0x0 0 rcx 0x6f77206f6c6c6568 8031924123371070824 rdx 0x7ffff7dd3780 140737351858048 rsi 0x5a 90 rdi 0x50 80 rbp 0x7fffffffdb50 0x7fffffffdb50 rsp 0x7fffffffdb50 0x7fffffffdb50 r8 0x0 0 r9 0xd 13 r10 0x0 0 r11 0x246 582 r12 0x4004e0 4195552 r13 0x7fffffffdc60 140737488346208 r14 0x0 0 r15 0x0 0 rip 0x400654 0x400654 < add + 10 > eflags 0x246 [ PF ZF IF ] cs 0x33 51 ss 0x2b 43 ds 0x0 0 es 0x0 0 fs 0x0 0 gs 0x0 0 Id Target Id Frame * 1 process 35330 "app" add ( a = 80 , b = 90 ) at . / a . c : 28

8 – Define GDB variables

You can define variables on your debugging session and use it on other commands:

(gdb) set $myvar = 100 (gdb) call add($myvar,20) $4 = 310 1 2 3 ( gdb ) set $ myvar = 100 ( gdb ) call add ( $ myvar , 20 ) $ 4 = 310

9 – Tips For Debugging Multiple Processes and Threads

While debugging a process that call fork(2) system call, you can choose if you want to continue with the child or the parent

You can also tell the debugger to detach the other process.

(gdb) set follow-fork-mode child (gdb) set detach-on-fork on 1 2 ( gdb ) set follow - fork - mode child ( gdb ) set detach - on - fork on

If you are debugging a multithreaded application, while a breakpoint hits, the debugger will freeze all the threads. You can tell the debugger to let the other threads continue run using:

(gdb) set non-stop on 1 ( gdb ) set non - stop on

10 – Integrating with IDE

The Last reason to use GDB is the ability to integrate it with some front ends. First , you can run gdb with -tui option to see the sources while running and also some other windows in text mode:

# gdb -tui ./app 1 # gdb -tui ./app

You can also find some front ends that give you the GDB command line so you can use both options (command line and GUI). You can find a list here

Last note

We are developers, we are using the keyboard all the time, learning GDB commands takes 30 minutes and after using a cheat sheet for a while, you won’t need the reference anymore