cmake: how to set rpath in a shared library with only target_link_directories (without target_link_libraries)?

Question

The goal of my project is as follows:
From my main executable, I want to load a library (libfoo.so) that loads a second library (libbar.so).

I want to not specify either relative or absolute paths in any filename arguments that I pass to dlopen: i.e. I'd like my code to read "dlopen("libfoo.so", RTLD_LAZY)" and not "/path/to/libfoo.so" or "../to/libfoo.so".
My understanding is that the way shared libraries are looked up (by libdl) is either 1) the value of the environment variable LD_LIBRARY_PATH, 2) "embedding" RPATH in binaries, 3) certain standard directories known to libdl.

My project's directory structure is like so:

.
├── CMakeLists.txt
├── build # this directory exists to perform an "out-of-source" build with "cmake .."
├── libs
│   ├── CMakeLists.txt
│   ├── bar.c
│   └── foo.c
└── main.c

main.c can successfully do dlopen("libfoo.so", RTLD_LAZY)
I did this by adding a target_link_directories(main PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/libs) statement in the CMakeLists.txt that compiles main.c.
This seems to have the effect of adding RPATH in the main executable, as desired:

$ objdump -x ./main | grep PATH
  RUNPATH              /home/user/code/scratch/3/build/libs

But foo.c is unsuccessful doing dlopen("libbar.so", RTLD_LAZY) even though I've added a target_link_directories(foo PRIVATE ${CMAKE_CURRENT_BINARY_DIR}) statement in its CMakeLists.txt.
I notice that in spite of having added the target_link_directories statement, libfoo.so does not have a RPATH:

$ objdump -x ./libs/libfoo.so | grep PATH
$

---

Things I've tried
Seemingly, RPATH is not added to shared libraries unless there is at least one target_link_libraries statement -- even if it's an "unnecessary" library.
I.e. if I link libfoo.so with libbar.so, then libfoo.so has the desired RPATH:

# Linking libbar works, but I'd prefer not to do this:
target_link_libraries(foo bar)

...results in:

$ objdump -x ./libs/libfoo.so | grep PATH
  RUNPATH              /home/user/code/scratch/3/build/libs

...also if I link an "unnecessary" shared lib along with the target_link_directories statement, then also, libfoo.so has the desired RPATH:

# Linking libbar works, but I'd prefer not to do this:
# target_link_libraries(foo bar)

# Linking an unnecessary library, then doing target_link_directories also works:
target_link_libraries(foo dl)
target_link_directories(foo PRIVATE ${CMAKE_CURRENT_BINARY_DIR})

$ objdump -x ./libs/libfoo.so | grep PATH
  RUNPATH              /home/user/code/scratch/3/build/libs

---

Questions
Do I understand CMake's behavior correctly: target_link_directories statements only result in a corresponding RPATH entry in the shared library if there is at least one target_link_library statement (even for an "unnecessary" lib) for the shared lib?
If that is correct, can someone please explain the rationale?

Is there another, "cleaner" way to add a RPATH to a shared lib (preferably with target_link_directories) without any "unnecessary" statements (like target_link_library to unnecessary libraries)?

---

The code/files:

// main.c
#include <dlfcn.h>
#include <stdlib.h>
#include <stdio.h>

int main(int argc, char* argv[]) {
  void* handle = dlopen("libfoo.so", RTLD_LAZY);
  if (!handle) {
    printf("dlopen error: %s\n", dlerror());
    return EXIT_FAILURE;
  }
  {
    void (*fptr)() = dlsym(handle, "func");

    if (fptr) { fptr(); }
  }
  dlclose(handle);
  return EXIT_SUCCESS;
}

// libs/foo.c
#include <dlfcn.h>
#include <stdio.h>

void func() {
  void* handle = dlopen("libbar.so", RTLD_LAZY);
  printf("here in libfoo!\n");
  if (!handle) {
    printf("dlopen error: %s\n", dlerror());
    return;
  }
  {
    void (*fptr)() = dlsym(handle, "func");
    if (fptr) { fptr(); }
  }
  dlclose(handle);
}

// libs/bar.c
#include <stdio.h>

void func() {
  printf("here in libbar!\n");
}

# CMakeLists.txt
cmake_minimum_required(VERSION 3.16)
project(my_prj)

add_subdirectory(libs)

add_executable(main main.c)
target_link_libraries(main dl)
target_link_directories(main PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/libs)

# libs/CMakeLists.txt
add_library(bar SHARED bar.c)

add_library(foo SHARED foo.c)

# This is what I want, but it doesn't work:
target_link_directories(foo PRIVATE ${CMAKE_CURRENT_BINARY_DIR})

# Linking libbar works, but I'd prefer not to do this:
# target_link_libraries(foo bar)

# Linking an unnecessary library, then doing target_link_directories also works:
# target_link_libraries(foo dl)
# target_link_directories(foo PRIVATE ${CMAKE_CURRENT_BINARY_DIR})

Answer 1

Here's a working build. You just need to adjust the BUILD_RPATH property appropriately and, if you write install() rules, adjust your INSTALL_RPATH to be similar to what I wrote in this answer. The following build is robust and adjusts the BUILD_RPATH:

cmake_minimum_required(VERSION 3.23)
project(test)

add_library(bar SHARED libs/bar.c)

add_library(foo MODULE libs/foo.c)
target_link_libraries(foo PRIVATE bar)

add_executable(main main.c)
target_link_libraries(main PRIVATE ${CMAKE_DL_LIBS})
set_property(TARGET main APPEND PROPERTY BUILD_RPATH "$<TARGET_FILE_DIR:foo>")

The last two lines are important. You must link to CMAKE_DL_LIBS to portably call dlopen and friends. The second line makes sure the directory containing libfoo, which you know main will load, is in the RPATH.

Here's the console output:

$ cmake -G Ninja -S . -B build -DCMAKE_BUILD_TYPE=RelWithDebInfo
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/alex/test/build

$ cmake --build build/ --verbose
[1/6] /usr/bin/cc -Dbar_EXPORTS  -O2 -g -DNDEBUG -fPIC -MD -MT CMakeFiles/bar.dir/libs/bar.c.o -MF CMakeFiles/bar.dir/libs/bar.c.o.d -o CMakeFiles/bar.dir/libs/bar.c.o -c /home/alex/test/libs/bar.c
[2/6] /usr/bin/cc -Dfoo_EXPORTS  -O2 -g -DNDEBUG -fPIC -MD -MT CMakeFiles/foo.dir/libs/foo.c.o -MF CMakeFiles/foo.dir/libs/foo.c.o.d -o CMakeFiles/foo.dir/libs/foo.c.o -c /home/alex/test/libs/foo.c
[3/6] /usr/bin/cc   -O2 -g -DNDEBUG -MD -MT CMakeFiles/main.dir/main.c.o -MF CMakeFiles/main.dir/main.c.o.d -o CMakeFiles/main.dir/main.c.o -c /home/alex/test/main.c
[4/6] : && /usr/bin/cc -fPIC -O2 -g -DNDEBUG   -shared -Wl,-soname,libbar.so -o libbar.so CMakeFiles/bar.dir/libs/bar.c.o   && :
[5/6] : && /usr/bin/cc -O2 -g -DNDEBUG  CMakeFiles/main.dir/main.c.o -o main  -Wl,-rpath,/home/alex/test/build  -ldl && :
[6/6] : && /usr/bin/cc -fPIC -O2 -g -DNDEBUG   -shared  -o libfoo.so CMakeFiles/foo.dir/libs/foo.c.o  -Wl,-rpath,/home/alex/test/build  libbar.so && :

$ ./build/main
here in libfoo!
here in libbar!

---

Answering some follow-up questions from the comments:

[W]hat is $<TARGET_FILE_DIR:foo>? Is TARGET_FILE_DIR a CMake variable (visible in CMakeLists.txts)?

It is not a variable, it is a generator expression. The values of these expressions are determined after the whole configuration step has been executed. In this way, we can be sure that this expression will expand to the actual directory containing libfoo.so, not merely the one we expect will contain it.

In general, I prefer to use generator expressions rather than variables whenever I can. They tend to lend CMake programming a more declarative rather than imperative feel and have fewer edge cases. For example, a user might set the value of CMAKE_RUNTIME_OUTPUT_DIRECTORY to something unexpected. This breaks your build if you compute the RPATH from CMAKE_CURRENT_BINARY_DIR or something.

[C]an you speak to the difference between target_link_libraries(main dl) (my version) and target_link_libraries(main PRIVATE ${CMAKE_DL_LIBS}) (your version)?

There are two differences here, both important:

1. Using target_link_libraries without a visibility specifier puts it into a weird limbo state that is sort of like PRIVATE, maybe, depending on the policy settings and whether or not any other calls to target_link_libraries have a visibility specifier. Both for clarity and to avoid these pitfalls, you should always specify one of PRIVATE, INTERFACE, or PUBLIC.
2. Using CMAKE_DL_LIBS is the correct way to link to whichever library contains the dlopen family of library functions. Did you know that HP-UX uses -ldld? Or that AIX uses -lld? Or that BSDs (including macOS) don't have a separate library for that? Well, passing -ldl is broken for those platforms.

Using target_link_libraries without a visibility specifier or passing it raw link flags are both serious code smells in the modern (post CMake ~3.5) era. Try to avoid them and ask questions when you don't think you can.