Posts Tagged ‘kernel’

Linux Kernel Debugging with kgdb

April 28, 2010

I’ve gone through a fair amount of pain the in the past couple of weeks while trying to debug in the linux kernel. To debug as you would with gdb you need to use kgdb. As of the linux kernel 2.6.26 kgdb is part of the kernel. The documentation is a little bit confusing as some of the components, kgdb8250 and kgdboe, don’t seem to be present. So the only option is serial debugging.

This was a bit of a problem as neither of my machines had a serial port. Well the desktop has the hardware on the motherboard, but no physical port actually connected to it. So the first step was to add a PCI-serial card. I linked the port to ttyS2 with the following line in /etc/serial.conf

/dev/ttyS2 uart 16550A port 0xe480 irq 16 baud_base 115200 spd_normal skip_test

Using a USB-Serial dongle allowed me to verify a connection with my laptop. Now I was ready to try making the kgdb connection. I verified that kgdb is enabled in the generic ubuntu kernel, then added a boot item in grub with the following options.

kgdbwait kgdboc=ttyS2,115200

I followed this process to connect but unfortunately at boot time ttyS2 is not linked properly.  So I changed the kernel options to


After booting the machine I had to manually enable kgdboc.

$ sudo -s

$ echo ttyS2,115200 > /sys/module/kgdboc/parameters/kgdboc

At this point the machine totally freezes waiting for the debugger to connect. So I started gdb as noted before and the connection was made. I issued continue on gdb and the desktop started responding again.  To break again I press Alt+SysRq g on the desktop.

Now that the connection is there I need debug symbols. To make sure I had symbols for the kernel I built my own kernel image.  The process for building on ubuntu is found here with special information for karmic here.  I ended up making a new sub-flavour as described here. Once the kernel was installed and booting I turned to getting symbols for my kernel module.  I compiled my module with CONFIG_DEBUG_INFO=y to ensure it contained the symbols. Then I had to make gdb load the symbols. That process is detailed on pages 100-101 of Linux Device Drivers chapter 4(PDF).  It’s also available here.

Now I could set a breakpoint in my module and step through it.  However when it came to setting a breakpoint in the kernel code I also got this error:

Cannot insert breakpoint 5.
Error accessing memory address 0xffffffff813a40fb: Unknown error 18446744073709551615.

I disabled CONFIG_DEBUG_RODATA in my kernel compile and was able to set breakpoints. I’m not certain that this was the fix as there were some other changes in the process I was using to debug at the same time as this change. But it’s worth a try if you’re getting this error.