I am debugging a Linux kernel crash. I am looking at the last words found in a kernel log available after the crash and reboot. I am trying to interpret the callstack of a failed task.
I am not sure I understand all the components of the output. Let's say I see something similar to this output:
Call Trace:
? show_cpuinfo+0x1/0x3f1
? seq_read+0x157/0x435
proc_reg_read+0x3e/0x60
__vfs_read+0x1b/0x34
vfs_read+0x99/0x152
ksys_read+0x61/0xd2
__x64_sys_read+0x1a/0x1c
do_syscall_64+0x60/0x1cb
entry_SYSCALL_64_after_hwframe+0x170/0x0
The names of functions are clear to me, but what about other bits?
What does the
+0x61/0xd2notation mean? One of the numbers seems to be an offset for the instruction pointer relative to the beginning of a symbol in each frame, but what does the second number indicate?Some frames are prepended with the
?character, like? seq_read, while others are not, like__vfs_read. What is the meaning of the question sign? Are there other symbols used in this position?SOLVED The panic message is printed between two
------------[ cut here ]------------lines. I believe they separate it from other stuff that might get printed to the same log. But in my case, there are two pairs of the "cut here" sections following each other, with different call stacks and different reasons specified for a crash. What can be an explanation for such output? It turned out the first callstack is printed usingWARN_ON_ONCE(), i.e. it is a warning. The second callstack is when the kernel has finally given up. In my case the first warning seems to be serious enough to be upgraded to a bug and to cause panic without waiting for the second event to happen. I can do it withsudo /sbin/sysctl kernel.panic_on_warn=1