How to resolve systemd-coredump error in journal: "kernel: Core dump to |/usr/lib/systemd/systemd-coredump pipe failed"

Question

I'm trying to set up systemd-coredump on Ubuntu 18.04, so that I can catch and log crashes of my C++ application for debugging.

So far, I've installed systemd-coredump version 237-3ubuntu10.47 from apt, and I'm able to trigger a crash by sending my application a segmentation fault signal:

sudo kill -s SEGV <application-pid>

However, I don't see a dump in /var/crash/ as I expected. Running sudo coredumpctl list does not show any crashes, either; it only replies No coredumps found.

I read the systemd-coredump manual that logs are stored in the journal, so I opened it with sudo journalctl and search for my kill command. After it, I found this error message:

Jun 30 21:53:41 ip-100-90-52-170 kernel: Core dump to |/usr/lib/systemd/systemd-coredump pipe failed

I examined the directory /usr/lib/systemd/, and I found that systemd-coredump did not exist. However, I'm not sure if this ...file? ..directory? is supposed to be created on the fly. Is there perhaps a permission issue during file/directory creation, because /usr/lib/systemd/ is owned by root, while my application runs as an unprivileged user?

Here is my kernel.core_pattern configuration, /usr/lib/sysctl.d/50-coredump.conf. (It's the default.)

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

# See sysctl.d(5) for the description of the files in this directory,
# and systemd-coredump(8) and core(5) for the explanation of the
# setting below.

kernel.core_pattern=|/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %e

And my coredump configuration, /etc/systemd/coredump.conf (also the default).

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See coredump.conf(5) for details.

[Coredump]
#Storage=external
#Compress=yes
#ProcessSizeMax=2G
#ExternalSizeMax=2G
#JournalSizeMax=767M
#MaxUse=
#KeepFree=

I also confirmed that I have no config snippets in /etc/systemd/coredump.conf.d/ (In fact, there is no such directory.)

Answer 1

TL;DR: My core_pattern was being overridden by /etc/sysctl.d/core.conf.

From re-reading the systemd-coredump manual, I eventually realized that /usr/lib/systemd/systemd-coredump was not just a file or directory where dumps were logged, but it was supposed to be the systemd-coredump binary, itself. So obviously, the fact that it was not present was a problem.

I also noticed that the error in the journal showed that the kernel was looking for the systemd-coredump binary in /usr/lib/systemd/systemd-coredump, not /lib/systemd/systemd-coredump, as my configuration showed. And in fact, a binary did exist at /lib/systemd/systemd-coredump.

Therefore, my next step was to figure out why the kernel was trying to use /usr/lib/systemd/systemd-coredump. For that, I performed a recursive file search with grep. The only config file I found containing the misconfigured binary path was /etc/sysctl.d/core.conf:

kernel.core_pattern = |/usr/lib/systemd/systemd-coredump --backtrace %p %u %g %s %t %e
kernel.core_uses_pid = 0
fs.suid_dumpable = 2
suid_dumpable = 2

Although the file /etc/sysctl.d/core.conf is not mentioned in the systemd-coredump manual, it is apparently another way to override the core_pattern, because after I commented out the kernel.core_pattern line in /etc/sysctl.d/core.conf and rebooted my VM, I was able to crash my application and see the dumps (and no error in the journal)! :)

$ sudo coredumpctl list
TIME                            PID   UID   GID SIG COREFILE  EXE
Wed 2021-06-30 22:56:23 UTC   23796   888   888  11 present   <my-application>

Answer 2

you conclude from the fact that the executable systemd-coredump is not in /usr/lib/systemd, it is not a problem. Well it is, your system is looking for that executable on that location there, does not find it, and that causes the errormessage. There is another file where this location can be set: /usr/lib/sysctl/50-coredump.conf. I think you'll find the right location there:

/lib/systemd/systemd-coredump.conf

Stef