Debugging Go Code with LLDB

Updated Jan 4, 2016

This applies to the gc toolchain. Besides this overview you might want to consult the LLDB manual.

Introduction

When you compile and link your Go programs with the gc toolchain on Linux, Mac OS X, FreeBSD or NetBSD, the resulting binaries contain DWARFv3 debugging information that recent versions (>3.7) of the LLDB debugger can use to inspect a live process or a core dump. You will probably need to build LLDB from trunk to get the go support.

Pass the '-w' flag to the linker to omit the debug information (for example, go build -ldflags "-w" prog.go ).

The code generated by the gc compiler includes inlining of function invocations and registerization of variables. These optimizations make debugging with lldb harder. To disable them when debugging, pass the flags -gcflags "-N -l" to the go command used to build the code being debugged.

Getting lldb

The latest release lldb (3.7) doesn’t contain the go extensions, so you will need to build it from trunk.

Common Operations

Show file and line number for code, set breakpoints and disassemble (lldb) l (lldb) l line (lldb) l file.go:line (lldb) b line (lldb) b file.go:line (lldb) disas

Show backtraces and unwind stack frames: (lldb) bt (lldb) frame n

Show the name, type and location on the stack frame of local variables, arguments and return values: (lldb) frame variable (lldb) p varname (lldb) expr -T -- varname

Go Extensions

Expression Parsing

LLDB includes a go expression parser.

(lldb) p x (lldb) expr *(*int32)(t) (lldb) help expr

Interfaces

By default, LLDB shows the dynamic type of interface values. This is usually a pointer. Consider func foo(a interface{}) { ... } . If you call foo(1.0) , lldb will treat a as *float64 inside foo. You can disable this behavior for a single expression or globally:

(lldb) expr -d no-dynamic-values -- a (lldb) settings set target.prefer-dynamic-values no-dynamic-values

Data Formatters

LLDB includes data formatters for go strings and slices. See the LLDB docs for custom variable formatting. If you want to extend the builtin formatters, see GoLanguageRuntime.cpp.

Channels and maps are 'reference' types, which lldb treats as pointers to C++-like types hash<int,string>* . Dereferencing will show the internal representation.

Goroutines

LLDB treats Goroutines as threads.

(lldb) thread list (lldb) bt all (lldb) thread select 2

Known Issues

Debug info is wrong when code is built with optimizations (the default). Make sure to add -gcflags "-N -l" to your go build or go install command.

to your or command. You cannot change the values of variables or call go functions

Need better support for chan and map types

The debug info doesn’t include imported packages, so you need to use the full package path in expressions. When the package path includes non-identifier chars you need to quote it: x.(*foo/bar.BarType) or (*“v.io/x/foo”.FooType)(x)

or Debug info doesn’t include scopes, so variables are visible before they’re initialized. If there are local vars with the same name (e.g. shadowed vars), you don’t know which is which.

The debug info only describes the location of variables in memory. You may see stale data while variables are in registers.

Can’t print function types

Tutorial

In this tutorial we will inspect the binary of the regexp package's unit tests. To build the binary, change to $GOROOT/src/regexp and run go test -gcflags "-N -l" -c . This should produce an executable file named regexp.test.

Getting Started

Launch lldb, debugging regexp.test:

$ lldb regexp.test (lldb) target create "regexp.test" Current executable set to 'regexp.test' (x86_64). (lldb)

Setting breakpoints

Set a breakpoint at the TestFind function:

(lldb) b regexp.TestFind

Sometimes the go compiler prefixes function names with the full path. If you can’t find the simple name, you can try using a function name regex:

(lldb) break set -r regexp.TestFind$ Breakpoint 5: where = regexp.test`_/code/go/src/regexp.TestFind + 37 at find_test.go:149, address = 0x00000000000863a5

Run the program:

(lldb) run --test.run=TestFind Process 8496 launched: '/code/go/src/regexp/regexp.test' (x86_64) Process 8496 stopped * thread #9: tid = 0x0017, 0x00000000000863a5 regexp.test`_/code/go/src/regexp.TestFind(t=0x000000020834a000) + 37 at find_test.go:149, stop reason = breakpoint 2.1 3.1 5.1 frame #0: 0x00000000000863a5 regexp.test`_/code/go/src/regexp.TestFind(t=0x000000020834a000) + 37 at find_test.go:149 146 // First the simple cases. 147 148 func TestFind(t *testing.T) { -> 149 for _, test := range findTests { 150 re := MustCompile(test.pat) 151 if re.String() != test.pat { 152 t.Errorf("String() = `%s`; should be `%s`", re.String(), test.pat)

Execution has paused at the breakpoint. See which goroutines are running, and what they're doing:

(lldb) thread list Process 8496 stopped thread #1: tid = 0x12201, 0x000000000003c0ab regexp.test`runtime.mach_semaphore_wait + 11 at sys_darwin_amd64.s:412 thread #2: tid = 0x122fa, 0x000000000003bf7c regexp.test`runtime.usleep + 44 at sys_darwin_amd64.s:290 thread #4: tid = 0x0001, 0x0000000000015865 regexp.test`runtime.gopark(unlockf=0x00000000000315a0, lock=0x00000002083220b8, reason="chan receive") + 261 at proc.go:131 thread #5: tid = 0x0002, 0x0000000000015865 regexp.test`runtime.gopark(unlockf=0x00000000000315a0, lock=0x00000000002990d0, reason="force gc (idle)") + 261 at proc.go:131 thread #6: tid = 0x0003, 0x0000000000015754 regexp.test`runtime.Gosched + 20 at proc.go:114 thread #7: tid = 0x0004, 0x0000000000015865 regexp.test`runtime.gopark(unlockf=0x00000000000315a0, lock=0x00000000002a07d8, reason="finalizer wait") + 261 at proc.go:131 * thread #9: tid = 0x0017, 0x00000000000863a5 regexp.test`_/code/go/src/regexp.TestFind(t=0x000000020834a000) + 37 at find_test.go:149, stop reason = breakpoint 2.1 3.1 5.1

the one marked with the * is the current goroutine.

Inspecting the Source

Use the "l" or "list" command to inspect source code.

(lldb) l (lldb) # Hit enter to repeat last command. Here, list the next few lines

Naming

Variable and function names must be qualified with the name of the packages they belong to. The Compile function from the regexp package is known to lldb as 'regexp.Compile'.

Methods must be qualified with the name of their receiver types. For example, the *Regexp type’s String method is known as 'regexp.(*Regexp).String'.

Variables referenced by closures will appear as pointers magically prefixed with '&'.

Inspecting the stack

Look at the stack trace for where we’ve paused the program:

(lldb) bt * thread #9: tid = 0x0017, 0x00000000000863a5 regexp.test`_/code/go/src/regexp.TestFind(t=0x000000020834a000) + 37 at find_test.go:149, stop reason = breakpoint 2.1 3.1 5.1 * frame #0: 0x00000000000863a5 regexp.test`_/code/go/src/regexp.TestFind(t=0x000000020834a000) + 37 at find_test.go:149 frame #1: 0x0000000000056e3f regexp.test`testing.tRunner(t=0x000000000003b671, test=0x000000020834a000) + 191 at testing.go:447 frame #2: 0x00000000002995a0 regexp.test`/code/go/src/regexp.statictmp_3759 + 96 frame #3: 0x000000000003b671 regexp.test`runtime.goexit + 1 at asm_amd64.s:2232

The stack frame shows we’re currently executing the regexp.TestFind function, as expected.

The command frame variable lists all variables local to the function and their values, but is a bit dangerous to use, since it will also try to print uninitialized variables. Uninitialized slices may cause lldb to try to print arbitrary large arrays.

The function’s arguments:

(lldb) frame var -l (*testing.T) t = 0x000000020834a000

When printing the argument, notice that it’s a pointer to a Regexp value.

(lldb) p re (*_/code/go/src/regexp.Regexp) $3 = 0x000000020834a090 (lldb) p t (*testing.T) $4 = 0x000000020834a000 (lldb) p *t (testing.T) $5 = { testing.common = { mu = { w = (state = 0, sema = 0) writerSem = 0 readerSem = 0 readerCount = 0 readerWait = 0 } output = (len 0, cap 0) {} failed = false skipped = false finished = false start = { sec = 63579066045 nsec = 777400918 loc = 0x00000000002995a0 } duration = 0 self = 0x000000020834a000 signal = 0x0000000208322060 } name = "TestFind" startParallel = 0x0000000208322240 } (lldb) p *t.startParallel (hchan<bool>) $3 = { qcount = 0 dataqsiz = 0 buf = 0x0000000208322240 elemsize = 1 closed = 0 elemtype = 0x000000000014eda0 sendx = 0 recvx = 0 recvq = { first = 0x0000000000000000 last = 0x0000000000000000 } sendq = { first = 0x0000000000000000 last = 0x0000000000000000 } lock = (key = 0x0000000000000000) }

That hchan<bool> is the runtime-internal representation of a channel.

Stepping forward:

(lldb) n # execute next line (lldb) # enter is repeat (lldb) # enter is repeat Process 17917 stopped * thread #8: tid = 0x0017, 0x000000000008648f regexp.test`_/code/go/src/regexp.TestFind(t=0x000000020834a000) + 271 at find_test.go:151, stop reason = step over frame #0: 0x000000000008648f regexp.test`_/code/go/src/regexp.TestFind(t=0x000000020834a000) + 271 at find_test.go:151 148 func TestFind(t *testing.T) { 149 for _, test := range findTests { 150 re := MustCompile(test.pat) -> 151 if re.String() != test.pat { 152 t.Errorf("String() = `%s`; should be `%s`", re.String(), test.pat) 153 } 154 result := re.Find([]byte(test.text)) (lldb) p test.pat (string) $4 = "" (lldb) p re (*_/code/go/src/regexp.Regexp) $5 = 0x0000000208354320 (lldb) p *re (_/code/go/src/regexp.Regexp) $6 = { expr = "" prog = 0x0000000208ac6090 onepass = 0x0000000000000000 prefix = "" prefixBytes = (len 0, cap 0) {} prefixComplete = true prefixRune = 0 prefixEnd = 0 cond = 0 numSubexp = 0 subexpNames = (len 1, cap 1) { [0] = "" } longest = false mu = (state = 0, sema = 0) machine = (len 0, cap 0) {} } (lldb) p *re.prog (regexp/syntax.Prog) $7 = { Inst = (len 3, cap 4) { [0] = { Op = 5 Out = 0 Arg = 0 Rune = (len 0, cap 0) {} } [1] = { Op = 6 Out = 2 Arg = 0 Rune = (len 0, cap 0) {} } [2] = { Op = 4 Out = 0 Arg = 0 Rune = (len 0, cap 0) {} } } Start = 1 NumCap = 2 }

We can step into the Stringfunction call with "s":

(lldb) s Process 17917 stopped * thread #8: tid = 0x0017, 0x0000000000067332 regexp.test`_/code/go/src/regexp.(re=0x0000000208354320, ~r0="").String + 18 at regexp.go:104, stop reason = step in frame #0: 0x0000000000067332 regexp.test`_/code/go/src/regexp.(re=0x0000000208354320, ~r0="").String + 18 at regexp.go:104 101 102 // String returns the source text used to compile the regular expression. 103 func (re *Regexp) String() string { -> 104 return re.expr 105 } 106 107 // Compile parses a regular expression and returns, if successful,

Get a stack trace to see where we are:

(lldb) bt * thread #8: tid = 0x0017, 0x0000000000067332 regexp.test`_/code/go/src/regexp.(re=0x0000000208354320, ~r0="").String + 18 at regexp.go:104, stop reason = step in * frame #0: 0x0000000000067332 regexp.test`_/code/go/src/regexp.(re=0x0000000208354320, ~r0="").String + 18 at regexp.go:104 frame #1: 0x00000000000864a0 regexp.test`_/code/go/src/regexp.TestFind(t=0x000000020834a000) + 288 at find_test.go:151 frame #2: 0x0000000000056e3f regexp.test`testing.tRunner(t=0x000000000003b671, test=0x000000020834a000) + 191 at testing.go:447 frame #3: 0x00000000002995a0 regexp.test`/code/go/src/regexp.statictmp_3759 + 96 frame #4: 0x000000000003b671 regexp.test`runtime.goexit + 1 at asm_amd64.s:2232