Super User

How to Debug the Linux Kernel with QEMU and Libvirt

How can you verify that your Linux kernel image will boot on real hardware? Booting your image with virtualization technologies can’t provide you with an absolute guarantee that your software will run on metal. However, QEMU and Libvirt give you enough capabilities to conduct a quick sanity check and get rid of most bugs at the development stage.

In this article, we explain how you can debug your Linux kernel and its modules during runtime. This article will be useful for Linux kernel developers who want to speed up the time to market of their software.

Written by: Sergii Kusii, Software Developer, Linux Development Team

Contents:

Environment

Debugging without source code

Debugging with source code

Debugging with third-party modules

Conclusion

We’ll provide you with step-by-step instructions on how you can find errors in your Linux kernel module or kernel image at the development stage. Our method includes three levels:

Debugging without source code

Debugging with source code

Debugging with third-party modules

You can either use one of these levels or dig deep and go through the whole process of kernel debugging.

Environment

To conduct kernel debugging, you should prepare the necessary environment. First of all, you need to set up the guest target.

The guest is a Libvirt or QEMU-style virtual machine that we’ll debug. The host is an operating system where debugging is taking place.

To create an environment, we use Ubuntu 16.04 as the host operating system and Ubuntu 16.04 as the guest operating system.

However, you can also use other guest Linux distributions. Just make sure to install the corresponding packages beforehand.

Here’s what you need to install for debugging:

QEMU as a hypervisor and emulator for hardware virtualization Libvirt as a daemon and a toolkit for managing platform virtualization GDB as a project debugger A QEMU virtual machine (in our case, it’s Ubuntu 16.04 under the name 04)

Related services Kernel and Driver Development

Debugging without source code

Debugging without the source code is the simplest way to quickly debug the kernel if you need to look at the call stack and variable values. You can verify the kernel image in this way:

Configure Libvirt on the host operating system with the following command:

virsh edit ubuntu16.04

In your editor of choice, you can open the file with virtual machine settings. Find the domain tag and add the following option:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0' > <qemu:commandline> <qemu:arg value='-s'/> </qemu:commandline>

Save the file and exit the editor. Now you need to restart the virtual machine (if it’s already running).

The -s parameter means that QEMU will open port 1234 for debugging by default. If you need to open other ports, you can set them in the domain XML file:

<qemu:commandline> <qemu:arg value='-gdb'/> <qemu:arg value='tcp::1235'/> </qemu:commandline>

The specified port should open after the virtual machine starts.

Disable Kernel Address Space Layout Randomization (KASLR) on the guest. To do this, you need to apply changes directly to the GRand Unified Bootloader (GRUB) configuration. Open /etc/default/grub and add nokaslr to the GRUB_CMDLINE_LINUX_DEFAULT variable:

GRUB_CMDLINE_LINUX_DEFAULT="splash quiet nokaslr"

Update GRUB with this command:

sudo update-grub

and reboot the guest operating system.

The Linux kernel packages do not include debug symbols, as they’re stripped from binary files at build time. However, you need to install debug symbols to let GDB assist you during debugging. To install debug symbols for the existing kernel on the guest, see the detailed description here.

Here how we’ll install debug symbols on Ubuntu:

#GPG key import sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C8CAB6595FDFF622 #Add repository config codename=$(lsb_release -c | awk '{print $2}') sudo tee /etc/apt/sources.list.d/ddebs.list << EOF deb http://ddebs.ubuntu.com/ ${codename} main restricted universe multiverse deb http://ddebs.ubuntu.com/ ${codename}-security main restricted universe multiverse deb http://ddebs.ubuntu.com/ ${codename}-updates main restricted universe multiverse deb http://ddebs.ubuntu.com/ ${codename}-proposed main restricted universe multiverse EOF sudo apt-get update sudo apt-get install linux-image-$(uname -r)-dbgsym

You can manually download and install the ddeb package from here as an example. However, be careful to install a package with the compatible kernel and architecture.

Here’s how to install debug symbols on Fedora Linux:

sudo dnf debuginfo-instal kernel-$(uname -r)

You can find a complete description of how to install debug symbols on Fedora Linux here.

Copy the /usr/lib/debug/boot/vmlinux-$(uname -r) file from the guest to any folder on the host so GDB can use it. In our case, we have the following file: /usr/lib/debug/boot/vmlinux-4.13.0-36-generic Run the QEMU virtual machine on the host and connect it to the debugger:

gdb ./vmlinux-4.13.0-36-generic (gdb) target remote :1234

Here’s an example of Linux kernel debugging for Ubuntu 16.04 (4.13.0-36):

kernel_new# gdb ./vmlinux-4.13.0-36-generic GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from ./vmlinux-4.13.0-36-generic...done. (gdb) target remote :1234 Remote debugging using :1234 native_safe_halt () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/include/asm/irqflags.h:54 54 /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/include/asm/irqflags.h: No such file or directory. (gdb) bt #0 native_safe_halt () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/include/asm/irqflags.h:54 #1 0xffffffff8191e20e in arch_safe_halt () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/include/asm/paravirt.h:92 #2 default_idle () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/kernel/process.c:354 #3 0xffffffff81036b45 in arch_cpu_idle () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/kernel/process.c:345 #4 0xffffffff8191e743 in default_idle_call () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/kernel/sched/idle.c:98 #5 0xffffffff810cd5f2 in cpuidle_idle_call () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/kernel/sched/idle.c:156 #6 do_idle () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/kernel/sched/idle.c:246 #7 0xffffffff810cd853 in cpu_startup_entry (state=CPUHP_ONLINE) at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/kernel/sched/idle.c:351 #8 0xffffffff81910e78 in rest_init () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/init/main.c:437 #9 0xffffffff824a514d in start_kernel () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/init/main.c:702 #10 0xffffffff824a42d5 in x86_64_start_reservations (real_mode_data= <optimized out>) at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/kernel/head64.c:318 #11 0xffffffff824a4417 in x86_64_start_kernel (real_mode_data=0x8a000 <error: Cannot access memory at address 0x8a000>) at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/kernel/head64.c:299 #12 0xffffffff810000cf in secondary_startup_64 () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/kernel/head_64.S:220 #13 0x0000000000000000 in ?? () (gdb) break do_dentry_open Breakpoint 1 at 0xffffffff812528e0: file /build/linux-hwe-4GXcua/linux-hwe-4.13.0/fs/open.c, line 704. (gdb) c Continuing. Thread 1 hit Breakpoint 1, do_dentry_open (f=0xffff88007b1a4a00, inode=0xffff88007a0a1008, open=0x0 <irq_stack_union>, cred=0xffff8800775eea80) at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/fs/open.c:704 704 /build/linux-hwe-4GXcua/linux-hwe-4.13.0/fs/open.c: No such file or directory. (gdb) print inode->i_ino $1 = 4026532024 (gdb)

Debugging with source code

Source code allows the debugger to associate events and circumstances in program execution with their corresponding location. So using the source code, you can run the (gdb) list command without errors like “No such file or directory.” If you have a source code file, make it available to GDB for kernel debugging in the following way:

Load the source code to the host (see detailed description here). You can read how to download the source code from the git repository here.

git clone git://kernel.ubuntu.com/ubuntu/xenial-maverick.git # find for kernel 4.13.0-36 git tag -l | grep 4.13.0-36 # Ubuntu-hwe-4.13.0-36.40_16.04.1 git checkout -b ubuntu-hwe-4.13.0-36 Ubuntu-hwe-4.13.0-36.40_16.04.1

Connect the source code to GDB with the (gdb)set substitute-path command:

gdb ./vmlinux-4.13.0-36-generic (gdb) set substitute-path /build/linux-hwe-4GXcua/linux-hwe-4.13.0 /media/veracrypt2/linux_kernel_new/ubuntu-xenial

If you look at the previous logs, you can see the path to the source code in the debug symbols that is /build/linux-hwe-4GXcua/linux-hwe-4.13.0

/build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/include/asm/irqflags.h: No such file or directory.

Substitute the original source code path with the path to our source code using the set substitute-path command. In our case, the new path is /media/veracrypt2/linux_kernel_new/ubuntu-xenial .

Debugging with third-party modules

This method is for those who want to debug a module that’s not included in the official Linux kernel version. For instance, we’ll try to debug our own module.

Build the module with debug symbols on the guest and install it. To build the module with debug symbols, add the -g flag (EXTRA_CFLAGS += -g) to the compiler options. Copy the built module to the host folder together with the debug symbols (or to the folder with the source code). In our case, we get this:

ls /media/veracrypt2/linux_kernel_new myspecificmodule.ko ubuntu-xenial vmlinux

Build the GDB scripts on the host so we can debug modules that are loaded. Unfortunately, scripts are provided only with the whole kernel, so we need to stop the building process after we get the debian/build/build-generic/scripts/gdb/linux/constants.py file (you can find more information about kernel building here).

sudo apt build-dep --yes linux-image-4.13.0-36-generic sudo apt install --yes fakeroot libncurses5-dev fakeroot debian/rules clean fakeroot debian/rules binary-headers binary-generic binary-perarch

If you want to continue building after interruption, make an additional link to the script:

ln -s /media/veracrypt2/linux_kernel_new/ubuntu-xenial/scripts/gdb/vmlinux-gdb.py /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/build/build-generic/vmlinux-gdb.py

Permit the GDB to load. Create the ~/.gdbinit file as a user on the host:

$ cat ~/.gdbinit add-auto-load-safe-path /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/build/build-generic/vmlinux-gdb.py

The file with debug symbols should be named vmlinux to make our scripts runnable, so rename this file on the host:

mv vmlinux-4.13.0-36-generic vmlinux

Run GDB and make sure that everything is okay with the scripts. We can also check if it’s possible to set breakpoints in our module and debug it (see comments in the script). The breakpoint makes the debugger stop execution after a specified function or line number in the source file.

gdb ./vmlinux GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from ./vmlinux...done. (gdb) set substitute-path /build/linux-hwe-4GXcua/linux-hwe-4.13.0 /media/veracrypt2/linux_kernel_new/ubuntu-xenial (gdb) source /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/build/build-generic/vmlinux-gdb.py # load and check if scripts work correctly (gdb) target remote :1234 Remote debugging using :1234 native_safe_halt () at /build/linux-hwe-4GXcua/linux-hwe-4.13.0/arch/x86/include/asm/irqflags.h:54 warning: Source file is more recent than executable. 54 } (gdb) lx-symbols # load all available modules loading vmlinux scanning for modules in /media/veracrypt2/linux_kernel_new no module object found for 'snd_hda_codec_generic' no module object found for 'snd_hda_intel' no module object found for 'snd_hda_codec' no module object found for 'snd_hda_core' no module object found for 'snd_hwdep' no module object found for 'snd_pcm' no module object found for 'crct10dif_pclmul' no module object found for 'crc32_pclmul' no module object found for 'ghash_clmulni_intel' no module object found for 'pcbc' no module object found for 'snd_seq_midi' no module object found for 'snd_seq_midi_event' no module object found for 'aesni_intel' no module object found for 'aes_x86_64' no module object found for 'snd_rawmidi' no module object found for 'crypto_simd' no module object found for 'glue_helper' no module object found for 'cryptd' no module object found for 'snd_seq' no module object found for 'snd_seq_device' no module object found for 'snd_timer' no module object found for 'input_leds' no module object found for 'snd' no module object found for 'joydev' no module object found for 'serio_raw' no module object found for 'i2c_piix4' no module object found for 'soundcore' no module object found for 'mac_hid' no module object found for 'parport_pc' no module object found for 'ppdev' loading @0xffffffffc0151000: /media/veracrypt2/linux_kernel_new/myspecificmodule.ko # our module is loaded no module object found for 'lp' no module object found for 'parport' no module object found for 'autofs4' no module object found for 'qxl' no module object found for 'ttm' no module object found for 'drm_kms_helper' no module object found for 'syscopyarea' no module object found for 'sysfillrect' no module object found for 'sysimgblt' no module object found for 'fb_sys_fops' no module object found for 'psmouse' no module object found for 'virtio_blk' no module object found for 'virtio_net' no module object found for 'floppy' no module object found for 'drm' no module object found for 'pata_acpi' (gdb) break do_dentry_open_operation_handler # breakpoint on the function of our module Breakpoint 1 at 0xffffffffc0154c50: file /home/quest/Desktop/mymodule/myspecificmodule/myFunctions.c, line 118. (gdb) c Continuing. [Switching to Thread 2] Thread 2 hit Breakpoint 1, do_dentry_open_operation_handler (file=0xffff880073d2c300, inode=0xffff88007af766e8, open=0x0 <irq_stack_union>, cred=0xffff8800737ff900) at /home/quest/Desktop/mymodule/myspecificmodule/myFunctions.c:118 118 /home/quest/Desktop/mymodule/myspecificmodule/myFunctions.c: No such file or directory. (gdb)

The line “no module object found for” means that there are no other modules that can be loaded. If the debugger can’t find source code files, you can manually add links to the source code for the modules. If any modules are available on the machine, you’ll see the following:

(gdb) lx-symbols # load all available modules loading vmlinux scanning for modules in /media/veracrypt2/linux_kernel_new loading @0xffffffffc02bb000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/pci/hda/snd-hda-codec-generic.ko loading @0xffffffffc0240000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/pci/hda/snd-hda-intel.ko loading @0xffffffffc0288000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/pci/hda/snd-hda-codec.ko loading @0xffffffffc0273000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/hda/snd-hda-core.ko loading @0xffffffffc023a000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/snd-hwdep.ko loading @0xffffffffc0250000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/snd-pcm.ko loading @0xffffffffc0235000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/arch/x86/crypto/crct10dif-pclmul.ko loading @0xffffffffc022b000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/arch/x86/crypto/crc32-pclmul.ko loading @0xffffffffc026e000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/arch/x86/crypto/ghash-clmulni-intel.ko loading @0xffffffffc01f6000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/crypto/pcbc.ko loading @0xffffffffc01f1000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/seq/snd-seq-midi.ko loading @0xffffffffc01bc000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/seq/snd-seq-midi-event.ko loading @0xffffffffc01fc000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/arch/x86/crypto/aesni-intel.ko loading @0xffffffffc01c2000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/arch/x86/crypto/aes-x86_64.ko loading @0xffffffffc01e8000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/snd-rawmidi.ko loading @0xffffffffc01e3000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/crypto/crypto_simd.ko loading @0xffffffffc01b7000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/arch/x86/crypto/glue_helper.ko loading @0xffffffffc01dc000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/crypto/cryptd.ko loading @0xffffffffc01cb000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/seq/snd-seq.ko loading @0xffffffffc01a6000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/snd-seq-device.ko loading @0xffffffffc01ae000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/snd-timer.ko loading @0xffffffffc0182000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/input/input-leds.ko loading @0xffffffffc0191000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/core/snd.ko loading @0xffffffffc0187000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/input/joydev.ko loading @0xffffffffc017d000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/input/serio/serio_raw.ko loading @0xffffffffc00d3000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/i2c/busses/i2c-piix4.ko loading @0xffffffffc009f000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/sound/soundcore.ko loading @0xffffffffc0071000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/macintosh/mac_hid.ko loading @0xffffffffc0174000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/parport/parport_pc.ko loading @0xffffffffc016a000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/char/ppdev.ko loading @0xffffffffc0151000: /media/veracrypt2/linux_kernel_new/myspecificmodule.ko loading @0xffffffffc008d000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/char/lp.ko loading @0xffffffffc0144000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/parport/parport.ko loading @0xffffffffc0066000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/fs/autofs4/autofs4.ko loading @0xffffffffc0133000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/gpu/drm/qxl/qxl.ko loading @0xffffffffc011b000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/gpu/drm/ttm/ttm.ko loading @0xffffffffc00f1000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/gpu/drm/drm_kms_helper.ko loading @0xffffffffc0061000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/video/fbdev/core/syscopyarea.ko loading @0xffffffffc00ea000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/video/fbdev/core/sysfillrect.ko loading @0xffffffffc00e3000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/video/fbdev/core/sysimgblt.ko loading @0xffffffffc00de000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/video/fbdev/core/fb_sys_fops.ko loading @0xffffffffc00ae000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/input/mouse/psmouse.ko loading @0xffffffffc00a4000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/block/virtio_blk.ko loading @0xffffffffc0094000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/net/virtio_net.ko loading @0xffffffffc0079000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/block/floppy.ko loading @0xffffffffc0008000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/gpu/drm/drm.ko loading @0xffffffffc0000000: /media/veracrypt2/linux_kernel_new/ubuntu-xenial/debian/linux-image-extra-4.13.0-36-generic/lib/modules/4.13.0-36-generic/kernel/drivers/ata/pata_acpi.ko

If you want to check other modules, you can add links to the source code for them in the same way as we did in the first step:

(gdb) set substitute-path /home/quest/Desktop/mymodule /media/veracrypt2/linux_kernel_new/mymodule

Conclusion

Debugging a Linux kernel image or module is a complex task that many developers face. Using such virtualization technologies as QEMU and Libvirt allow them to detect and fix the majority of defects at the development stage.

In this article, we showed you three levels of debugging: debugging with and without source code and debugging with the help of third-party modules. Depending on the task at hand, you can either debug your kernel image or module on one of these levels or go through the whole process.

Here at Apriorit, we have a team of passionate kernel and driver developers who will be glad to assist you in creating your own software solution for the Linux platform.