Is there any short A to Z description of how to debug the Linux kernel that has been tested and contains ALL necessary steps ? Esp. for Yocto?

Question

Debugging the Linux Kernel with kgdb over rs-232 needs several preparation steps. I found awesome documentation, but no single-source that is fully self-contained and summarizes all steps needed, does not explain for ages, and has been tested. And also covers Yocto.

Is there any source that covers all that is needed in one single and short description ?

I.e.:

• What files are needed in the directory GDB is started from (e.g. kernel awareness, source, vmlinux) and how to get theese, where to put it ?
• When and where to get a cross-gdb from ?
• ALL kernel config options needed, also the not-obvious ones (like CONFIG_RANDOMIZE_BASE)
• How to configure the serial ports
• Explaining a working back and forth of breaking into debugee and debugger to get started.
• Explaining one rock-solid option of stopping the kernel that runs everywhere.
• Explaining how to get this done not only for PC-PC debugging, but also for Yocto targets.

Answer 1

Debugging the Linux Kernel via a Nullmodem-Cable:

It took me a while to get a kgdb connection with Linux kernel awareness fully running. I share my way of doing this with Ubuntu Eoan (optional: Yocto Warrior) in 2020 here:

Tested with:

• Debugging a linux based Intel PC from an Intel MacBook running MacOS Catalina. Using the gdb from the Homebrew package "i386-elf-gdb“. (wituout „-tui“ option in GDB)
• Debugging a linux based ARM target (i.mx6, Yocto) from a linux based Intel PC.

Prerequisites:

You need two computers and a serial nullmodem cable. Check the cable by firiing up a serial termianl (e.g. screen or putty) on both hosts, connecting to your serial port (e.g. /dev/ttyS0 or /dev/ttyUSB0) and print characters from each station to the other. Remember the /dev/tty ports you confirmed.

Preparation:

You need on the first debuggee computer, we call it „target":

• Special kernel installed that contains symbols, kgdb support etc.

• Learn how to compile and install a kernel and use in make menuconfig belows configuration. You can search for Sybmbols with F8 or the / key in menuconfig. (E.g. wiki.ubuntu.com. There take care in the first paragraph to execute deb-src before apt-get :)

  # CONFIG_SERIAL_KGDB_NMI is not set

  CONFIG_CONSOLE_POLL=y

  # CONFIG_DEBUG_INFO is not set

  CONFIG_KGDB=y

  CONFIG_KGDB_SERIAL_CONSOLE=y

  # CONFIG_KGDB_TESTS is not set

  # CONFIG_KGDB_KDB is not set

  CONFIG_FRAME_POINTER=y

  CONFIG_DEBUG_INFO=y

  # CONFIG_DEBUG_INFO_REDUCED is not set

  # CONFIG_DEBUG_INFO_SPLIT is not set

  CONFIG_DEBUG_INFO_DWARF4=y

  CONFIG_GDB_SCRIPTS=y

  CONFIG_STRIP_ASM_SYMS=y

  # CONFIG_RANDOMIZE_BASE is not set

(Note for advanced Yocto use, skip if you're debugging a PC: In yocto I created in my layer a file: recipes-kernel/linux/linux-mainline_%.bbappend with the content: FILESEXTRAPATHS_prepend := "${THISDIR}/files:" SRC_URI += "file://kgdb.cfg“ And in files/kgdb.cfg I added the config fragment shown above (without the on ARM unavailable options CONFIG_RANDOMIZE_BASE and CONFIG_FRAME_POINTER) )

You need on the second debugger computer, we call it „debugger pc":

• Full kernel source code, same code you used to compile the kernel above. (If you compiled the .o and .ko objects in place and not in a build-folder you better not copy the directory from the other pc, where you called make etc. in, but then better grab fresh sources again.)
• vmlinux file containing the symbols (lies in the kernel source root, or build folder on the highest level after kernel make).
• vmlinux-gdb.py file that was made during the kernel build (also lies at the same position on the highest level.).
• All scripts in the folder scripts/gdb (Folder scripts in the same toplevel-position. If you use a dedicated build folder use the script folder from there, not from the source folder.)
• (Advanced: If both computers don’t match in CPU, like Intel and Arm, a cross-gdb build. Ignore if you're on Intel/AMD.)

Note for advanced Yocto use, I did something like (ignore if you debug a PC):

 bitbake -c patch virtual/kernel     #(apply the changed kernel config from above)
 bitbake -f -c compile virtual/kernel   #(unpack is not sufficient because of vmlinux-gdb.py)
 mkdir ~/gdbenv
 cp -a tmp/work-shared/phyboard-mira-imx6-14/kernel-source/. ~/gdbenv
 cp tmp/work/phyboard_mira_imx6_14-phytec-linux-gnueabi/linux-mainline/4.19.100-phy1-r0.0/build/vmlinux ~/gdbenv
 cp tmp/work/phyboard_mira_imx6_14-phytec-linux-gnueabi/linux-mainline/4.19.100-phy1-r0.0/build/vmlinux-gdb.py ~/gdbenv
 mkdir ~/gdbenv/scripts
 cp -r tmp/work/phyboard_mira_imx6_14-phytec-linux-gnueabi/linux-mainline/4.19.100-phy1-r0.0/build/scripts/gdb ~/gdbenv/scripts

Then (ignore if you're on a PC) yocto bitbake -c populate_sdk [my-image]

Then (still ignore on PC) install the sdk .sh-installation file from your deploy directory on the debugger pc and start the environment as guided by the output of the install script (remember that command), then use "$GDB" for starting the cross-gdb instead of „gdb".

Debug execution

Launch on the debugger two console screens:

Console 1, ssh: +++++++++++++++++++++++++++++++++++++++

 ssh [email protected]
 sudo -s
 echo ttyS0,9600n8 > /sys/module/kgdboc/parameters/kgdboc
 echo 1 > /proc/sys/kernel/sysrq
 echo g > /proc/sysrq-trigger

Console 2, local: ++++++++++++++++++++++++++++++++++++++++

 cd ~/gdbenv
 gdb -tui ./vmlinux

 add-auto-load-safe-path ~/gdbenv
 source ~/gdbenv/vmlinux-gdb.py
 set serial baud 9600
 target remote /dev/ttyS0                        (use the tty port you confirmed in the beginning)
 b [name of the c funtion you want to debug]
 cont

Back to console 1, ssh: +++++++++++++++++++++++++++++++++++++++

 [Now trigger the function, e.g. sudo modprobe yourFancyKernelModule]

Back to console 2, local: ++++++++++++++++++++++++++++++++++++++++

Now use gdb functions, like bt, step, next, finish ...

You can also use linux-aware commands. Call "apropos lx“ in gdb for a list of commands.