or how to mess with your kernel.

Introduction

This is a description on how to cross compile[1] a custom Linux kernel for the Raspberry Pi 3B including kernel debugging and memory leak investigation configurations.

[1] Using a host Fedora 28 to a guest Raspberry Pi system; instructions may differ depending on your OS, for example for Debian-based OS’s it can be something similar to gcc-arm-linux-gnueabihf

Setting Up

Getting the kernel

First thing to do while reading this, is to start downloading the kernel repo; It can take some time.

git clone --depth=1 https://github.com/raspberrypi/linux/

Current branch at the time of writing (July 2018) is the rpi-4.14.y .

Using an old . config

If there’s already a known working kernel configuration file on hand from the OS then it’s advised to use it. It usually is on the location /proc/config.gz . For the file to exist, the option Kernel .config support should be enabled in the kernel. This is the case on the current Raspberry Pi 3 and in case you don’t see the file there then do a sudo modprobe configs and the file will magically appear in the right place.

# On the host

KERNEL=kernel7

# Transfer your old file from your Pi via scp

zcat /proc/config.gz > .config

# Clean any old config files

make mrproper

# make the kernel aware of using an old configuration

make ARCH=arm CROSS_COMPILE=arm-linux-gnu- oldconfig

# and setup with new options; look on the next section

make ARCH=arm CROSS_COMPILE=arm-linux-gnu- menuconfig

Detailed debug info configuration

As a reference here’s the (missing) required configuration for a debug and memory leak detection kernel. For more information consider reading the Linux Kernel Documentation. This is for reference only, the .config file shouldn’t be manually edited as there are dependencies between the options; that may result in a non/malfunctioning kernel.

# Note: The commented options are important but they were already enabled

# = CONFIG_PRINTK_TIME=y

# = CONFIG_DEBUG_KERNEL=y

# = CONFIG_KALLSYMS=y

# = CONFIG_KGDB=y

# = CONFIG_KGDB_SERIAL_CONSOLE=y

# = CONFIG_KGDB_KDB=y

# = CONFIG_PROC_FS=y

# = CONFIG_MAGIC_SYSRQ = y

# = CONFIG_SYSFS=y

CONFIG_KALLSYMS_ALL=y

CONFIG_PREEMPT=y # CLASH WITH VOLUNTAR?Y

CONFIG_KDB_KEYBOARD=y

CONFIG_PROC_VMCORE=y

CONFIG_PROC_KCORE=y

CONFIG_CRASH_DUMP=y

CONFIG_DEBUG_INFO=y

CONFIG_KEXEC=y

CONFIG_SLUB_DEBUG_ON=y

CONFIG_SLUB_STATS=y

CONFIG_DEBUG_SLAB=y

CONFIG_DEBUG_SPINLOCK_SLEEP=y

CONFIG_DEBUG_SPINLOCK=y

CONFIG_DEBUG_BUGVERBOSE=y

CONFIG_DEBUG_KMEMLEAK=y

CONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE=4096

CONFIG_DEBUG_KMEMLEAK_TEST=m

Making the kernel

Fire it and grab a cup of tea while waiting for the compilation!

make -j6 ARCH=arm CROSS_COMPILE=arm-linux-gnu- zImage modules dtbs

Debug Notes

Add to the /boot/cmdline.txt the following, where user_debug=N is one of the following:

1 — undefine instruction events

2 — system calls

4 — invalid data aborts

8 — SIGSEGV faults

16 — SIGBUS faults

Installing onto the SD Card

To identify the SD run lsblk before and after the SD insertion. You ‘re going to get two partitions: p1 should be the boot partition, while p2 the rootfs . Now create two folders in oder to mount the fs , for example:

mkdir -p mnt/bootfs

mkdir mnt/rootfs

Mount and install the modules:

sudo mount /dev/xxxp1 mnt/bootfs

sudo mount /dev/xxxp2 mnt/rootfs sudo make ARCH=arm CROSS_COMPILE=arm-linux-gnu- INSTALL_MOD_PATH=mnt/rootfs modules_install

After that you need to copy the kernel and device tree blobs onto the SD card:

# Run all that as root

cp mnt/bootfs/$KERNEL.img mnt/bootfs/$KERNEL-backup.img

cp arch/arm/boot/zImage mnt/bootfs/$KERNEL.img

cp arch/arm/boot/dts/*.dtb mnt/bootfs/

cp arch/arm/boot/dts/overlays/*.dtb* mnt/bootfs/overlays/

cp arch/arm/boot/dts/overlays/README mnt/bootfs/overlays/

umount mnt/bootfs

umount mnt/rootfs

Debugging

This is just a reference and some things to try using the compiled kernel to get more debugging information. I am writing a new article that is addressing the issue in more detail. Stay tuned!

Memory leaks

# Using kernel memory leak debugging

# sudo mount -t debugfs nodev /sys/kernel/debug # superfluous — already mounted

sudo bash -c “echo scan > /sys/kernel/debug/kmemleak”

sudo cat /sys/kernel/debug/kmemleak # Repeat to see updated info # IF the `echo scan…` part above *fails*, check /var/log/kern.log

# What I saw: “Kernel memory leak detector disabled”,

# “Early log buffer exceeded (####), please increase DEBUG_KMEMLEAK_EARLY_LOG_SIZE” # I increased CONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE and rebuilt/reinstalled the kernel # Another handy command:

sudo slabtop -sc

Oops and bugs

If the kernel is compiled with the option CONFIG_KALLSYMS then the dump is usable as it is. The most important information to keep is the EIP line ignoring the 0010: . By looking at the uncompressed kernel ( vmlinux and not zImage or vmlinuz ) with nm vmlinux | sort | less one can extract the name list, match it against the EIP from the kernel crash and look up the kernel function that contains the offending address.

Note: The address can’t be matched exactly (so it cant’ be grepped) but it’s a good starting point as to discover the location of the call.

Alternatively, you can use gdb on a running kernel. (read-only; i.e. you cannot change values or set break points.) To do this, first compile the kernel with -g ; edit arch/arm/Makefile appropriately, then do a make clean. You’ll also need to enable CONFIG_PROC_FS (via make config ).

After you’ve rebooted with the new kernel, launch gdb vmlinux /proc/kcore . You can now use all the usual gdb commands. The command to look up the point where your system crashed is l *0xXXXXXXXX . (Replace the XXXes with the EIP value.)

gdb’ing a non-running kernel currently fails because gdb (wrongly) disregards the starting offset for which the kernel is compiled.

Troubleshooting

Can’t load kernel modules

Check which modules are loading through the conf file emacs -nw /etc/modules-load.d/*.conf (try commenting a few out) and run sudo systemctl status systemd-modules-load.service which should give more detailed information on what is failing.

Cannot find wifi card / No Internet

It seems a bit weird as sometimes after a kernel installation the internet installation seems to fail. I suggest that before installing the modified kernel do a sudo raspi-update to get the current firmware blobs.

Useful articles

Many of the information above has been compiled through a lot of experimentation, crashed systems, dumped cores and a lot of reburned SD cards. Thanks all the authors for their Free and Open Source approach.

Linux Kernel Admin Guide

Debugging the linux kernel

The kernel newbie debugging

Debug kernel space memory leak

Trace and debug the linux kernel

Newbie corner kernel

Kernel debugging tips