tl;dr

If you enjoy this article, subscribe (via RSS or e-mail) and follow me on twitter.

Debugging boot loaders and other low level pieces of a computer system can be pretty tricky, especially because you may not have multiprocess support or access to a hard drive or other devices.

This blog post examines one way of debugging these sorts of systems by examining an insanely clever GNU assembler macro and some GDB stub code in the GRUB boot loader.

This piece of stub code allows a programmer to debug GRUB with GDB over a serial cable to help diagnose a broken boot loader.

why?

Firstly, the macro that will be examined is truly a thing of beauty . The macro generates assembly code stubs for a range of interrupts via recursion and, with very clever use of labels, automatically writes the addresses of the generated assembly to an array so that those addresses can later be used as interrupt handler offsets.

Secondly, I think debugging is actually much more interesting than programming in most cases. In particular, debugging low level things like GRUB are particularly interesting to me because of the weird situations that arise. Imagine you are trying to debug something, but you have no keyboard, maybe video is only sort of working, you don’t have multiprocess support, and you aren’t able to communicate with your hard drive.

How do you debug a page fault in a situation like this?

This blog post will attempt to explain how GRUB overcomes this by using some really clever code coupled with GDB’s remote debug functionality.

overview of what you are about to see

GRUB’s GDB module is loaded.

The module calls a function named grub_gdb_idtinit .

. grub_gdb_idtinit loads the interrupt descriptor table with addresses of functions to be executed when various interrupts are raised on the system.

loads the interrupt descriptor table with addresses of functions to be executed when various interrupts are raised on the system. The addresses of the interrupt handlers are from an array called grub_gdb_trapvec .

. The code for two different types of interrupt handlers is generated with a series of insanely clever macros, explained in detail below. The main macro named ent uses recursion and clever placement of labels to automatically generate the assembly code stubs and write their addresses to grub_gdb_trapvec .

of interrupt handlers is generated with a series of insanely clever macros, explained in below. The main macro named uses recursion and clever placement of labels to automatically generate the assembly code stubs and write their addresses to . The addresses of the interrupt handler code is filled in the grub_gdb_trapvec array by using labels.

array by using labels. The generated code of the interrupt handlers themselves call grub_gdb_trap .

. grub_gdb_trap reads and writes packets according to GDB’s remote serial protocol.

reads and writes packets according to GDB’s remote serial protocol. The remove debugger is now able to set breakpoints, dump register contents, or step through instructions via serial cable.

Prepare yourself.

GRUB’s GDB module initialization

The GRUB 2.0 boot loader supports dynamically loaded modules to extend the functionality of GRUB. I’m going to dive right into the GDB module, but you can read more about writing your own modules here.

The GRUB’s GDB module has an init function that looks like this:

GRUB_MOD_INIT (gdb) { grub_gdb_idtinit (); cmd = grub_register_command ("gdbstub", grub_cmd_gdbstub, N_("PORT"), N_("Start GDB stub on given port")); cmd_break = grub_register_command ("gdbstub_break", grub_cmd_gdb_break, 0, N_("Break into GDB")); /* other code */

This module init function starts by calling a function named grub_gdb_idtinit which has a lot interesting code that we will examine shortly. As we will see, this function creates a set of interrupt handlers and installs them so that any exceptions (divide by 0, page fault, etc) that are generated will trigger GDB on the remote computer.

After that, two commands named gdbstub and gdbstub_break are registered with GRUB. If the GRUB user issues one of these commands, the corresponding functions are executed.

The first command, gdbstub attaches a specified serial port to the GDB module so that the remote GDB session can communicate with this computer.

The second command, gdbstub_break simply raises a debug interrupt on the system by calling the function grub_gdb_breakpoint after some error checking:

void grub_gdb_breakpoint (void) { asm volatile ("int $3"); }

This works just fine because the grub_gdb_idtinit has registered a handler for the debug interrupt.

entering the rabbit hole: grub_gdb_idtinit

The grub_gdb_idtinit function which is called during initialization is pretty straightforward. It simply creates interrupt descriptor table (IDT) entries which point at interrupt handlers for interrupt numbers 0 through 31. The basic idea here is that something bad happens (page fault, general protection fault, divide by zero, …) and the CPU calls a handler function to report the exception or error condition.

You can read more about interrupt and exception handling on the Intel 64 and and IA-32 CPUs by reading the Intel® 64 and IA-32 Architectures: Software Developer’s Manual volume 3A, chapter 6 available from Intel here.

Take a look at the C code for grub_gdb_idtinit , paying close attention to the for loop:

/* Set up interrupt and trap handler descriptors in IDT. */ void grub_gdb_idtinit (void) { int i; grub_uint16_t seg; asm volatile ("xorl %%eax, %%eax

" "mov %%cs, %%ax

" :"=a" (seg)); for (i = 0; i <= GRUB_GDB_LAST_TRAP; i++) { grub_idt_gate (&grub_gdb_idt[i], grub_gdb_trapvec[i], seg, GRUB_CPU_TRAP_GATE, 0); } grub_gdb_idt_desc.base = (grub_addr_t) grub_gdb_idt; grub_gdb_idt_desc.limit = sizeof (grub_gdb_idt) - 1; asm volatile ("sidt %0" : : "m" (grub_gdb_orig_idt_desc)); asm volatile ("lidt %0" : : "m" (grub_gdb_idt_desc)); }

You'll notice that this function maps interrupt numbers to handler function addresses in a for -loop. The function addresses come from an array named grub_gdb_trapvec .

The grub_idt_gate function called above simply constructs the interrupt descriptor table entry, given:

a memory location for the entry to live (above: grub_gdb_idt[i] )

) the address of the handler function from the grub_gdb_trapvec array (above: grub_gdb_trapvec[i] )

array (above: ) the segment selector (above: seg )

) and finally the gate type (above: GRUB_CPU_TRAP_GATE ) and privilege bits (above: 0 )

Note that the last two inline assembly statements store existing IDT descriptor and set a new IDT descriptor, respectively.

Naturally, the next question is: where do the function addresses in grub_gdb_trapvec come from and what, exactly, do those handler functions do when executed?

grub_gdb_trapvec: a series of clever macros

It turns out that grub_gdb_trapvec is an array which is constructed through a series of really fucking sexy macros in an assembly.

Let's first examine grub_gdb_trapvec :

/* some things removed for brevity */ .data VECTOR VARIABLE(grub_gdb_trapvec) ent EC_ABSENT, 0, 7 ent EC_PRESENT, 8 ent EC_ABSENT, 9 ent EC_PRESENT, 10, 14 ent EC_ABSENT, 15, GRUB_GDB_LAST_TRAP

This code creates a global symbol named grub_gdb_trapvec in the data section of the compiled object. The contents of grub_gdb_trapvec are constructed by a series of invocations of the ent macro.

Let's take a look at the ent macro (I removed some Apple specific code for brevity) and go through it piece by piece:

.macro ent ec beg end=0 #define EC \ec #define BEG \beg #define END \end .text 1: .if EC add $4, %esp .endif

This is the start of the ent macro. This code creates a macro named ent and gives names to the arguments handed over to the macro. It assigns a default value of 0 to end , the third argument.

After that, it uses C preprocessor macros named EC , BEG , and END . This is done to assist with cross-platform builds of this source (specifically for dealing with OSX weirdness).

Next, some code is added to the text section of the object. The start of the code is going to be given the label 1 , so that it can be easily referred to later. This label is what will be used to automatically fill in the addresses of the assembly code stubs a little later.

Finally, the add $4, %esp code is included in the assembled object only if EC is non-zero.

EC is the first argument to the ent macro which could either be EC_ABSENT (0) or EC_PRESENT (1) as you saw above. EC stands for "error code." Some interrupts/exceptions that can be generated put an error code on the stack when they occur. If this interrupt/exception places an error code on the stack, this line of code is adding to the stack pointer (remember: on x86 the stack grows down, from higher addresses to lower addresses) to position the stack pointer above the error code. This is done to ensure the stack is at the same position regardless of whether or not an error code is inserted on the stack. In the case where an error code does exist, it is ignored by the code below.

save_context mov $EXT_C(grub_gdb_stack), %esp mov $(BEG), %eax /* trap number */ call EXT_C(grub_gdb_trap) load_context iret

This next piece of code begins by using another macro called save_context which writes out the current register values to memory. Next, the address of a piece of memory called grub_gdb_stack is written to %esp . After this instruction, all future code that runs will be using stack space backed by a section of memory named grub_gdb_stack . The interrupt number is written to the %eax register and then the C function grub_gdb_trap is called. We'll take a look at what this function does in a bit. The load_context macro does the opposite of save_context and restores all register values from memory.

Finally, an iret instruction is used to continue execution. In most cases, this instruction restores the system to a broken state where it will hang, trigger another exception, or just reboot itself depending on how many levels deep you have gotten yourself in exceptions.

/* * Address entry in trapvec array. */ .data VECTOR .long 1b /* * Next... (recursion). */ .if END-BEG > 0 ent \ec "(\beg+1)" \end .endif .endm

This is the last piece of the amazing ent macro. It refers to a data section created earlier when the grub_gdb_trapvec symbol was being created and in this section the address where label 1 exists is written.

Thus, the address of the code which saves the CPU context, switches out the stack, and invokes grub_gdb_trap is written out.

The ent macro ends by re-invoking itself to generate more code in the .text and fill in more addresses in the .data section for each interrupt/exception in the range passed in to ent as BEG and ENG .

Wow. An macro.

grub_gdb_trap

The code generated by the ent macro calls grub_gdb_trap . In other words, this function is called whenever an interrupt/execption is raised while GRUB is running and the GDB module is loaded.

This function pulls data off the serial port (which you set up when you ran gdbstub in GRUB as seen earlier). The data coming in on the serial port are packets as per GDB's remote serial protocol. These packets contain commands from the remote GDB session. grub_gdb_trap parses these packets, executes the commands, and replies. So, packets are parsed and registers are updated, memory is written or read, and data is passed back over the serial port. This is what allows a remote GDB session on another computer connected via serial cable to set breakpoints, examine registers, or single step code.

Conclusion

GDB's remote serial protocol is very powerful.

Likewise, knowing how to use GNU as can help you construct really clever macros to generate repetitive assembly code easily.

can help you construct really clever macros to generate repetitive assembly code easily. Writing a C-stub to parse GDB's remote serial protocol and carry out the commands can allow you to debug weird things, even if the target system lacks multiprocess support, system calls, or a hard drive.

Go read the GRUB source code. It's pretty interesting.

If you enjoyed this article, subscribe (via RSS or e-mail) and follow me on twitter.

References