Analyze Linux core dump on different machine: threads and shared libs

79 Views Asked by At

Here is the sad story:

  1. I built an executable thedarkmod.x64 on my machine, with debug symbols moved off to a separate thedarkmod.x64.debug file.
  2. A user experienced a crash on this executable, while running it under gdb. She saved core dump core.1600 using generate-core-file gdb command.
  3. I downloaded this core file and opened it by starting gdb ./thedarkmod.x64 core.1600.
  4. I switch between threads and run bt command, but I see some trash instead of proper stack traces.

Note: the user has my thedarkmod.x64.debug file, and when she runs bt just before saving the core dump, she sees meaningful stack trace.


When I start gdb on core dump, I see many warning messages like:

  • Unable to find libthread_db matching inferior's thread library, thread debugging will not be available
  • warning: .dynamic section for "libXXX.so" is not at the expected address (wrong library or version mismatch?)

According to this question, the first warning seems to imply that unless I have the same version of libthread_db.so.1 as the one on the machine where core dump was saved, I cannot see anything useful in multithreaded program. I asked the user to find this file and give it to me, but it did not help. Then I asked to provide libpthread.so.0 file too, and after some struggling with set solib-search-path, set sysroot, set auto-load safe-path, and set libthread-db-search-path I managed to get this warning replaced with "Thread debugging using libthread_db enabled", but stack traces were still wrong.

So the questions are:

  1. Is there a way to properly inspect core dumps generated on Linux machine with vastly different environment? I mean different kernel, pthreads, glibc, etc. Is there anywhere detailed instructions on how to achieve that?
  2. Is there any gdb command like generate-core-file includecode, which would embed all the code including the necessary .so libs into the core file, so that I could open it on my machine without additional hassle?

At the current moment, I have to admit that Linux core dumps are pretty useless to me, since I am not ready to create a new Linux VM for every core dump submitted.


UPDATE: I managed to obtain proper stack trace.

  1. The solution provided in the "duplicate question" did not work for me. No solib-related setting was enough, only set sysroot helped.
  2. In my particular case, the stack trace ended inside free function of libc. Gdb could not walk through its call stack, supposedly because it was compiled with -fomit-frame-pointer just like most libraries. Making sure gdb loads libc.so.6 obtained from the user's machine helped.

Here is the full list of gdb commands which I use (each of them is necessary to make things work, except for bt of course):

# note: all the .so files obtained from user machine must be put into local directory.
#
# most importantly, the following files are necessary:
#   1. libthread_db.so.1 and libpthread.so.0: required for thread debugging.
#   2. other .so files are required if they occur in call stack.
#
# these files must also be renamed exactly as the symlinks
# i.e. libpthread-2.28.so should be renamed to libpthread.so.0

# load executable file
file ./thedarkmod.x64

# force gdb to forget about local system!
# load all .so files using local directory as root
set sysroot .

# drop dump-recorded paths to .so files
# i.e. load ./libpthread.so.0 instead of ./lib/x86_64-linux-gnu/libpthread.so.0
set solib-search-path .
# disable damn security protection
set auto-load safe-path /

# load core dump file
core core.6487

# print stacktrace
bt
0

There are 0 best solutions below