LD_PRELOAD malloc to shm_open

Question

I'm trying to intercept malloc call using LD_PRELOAD. I want to move all memory allocation to shared memory by changing malloc to shm_open followed by mmap. How can I do it?

LD_PRELOAD of malloc works fine. I can intercept every malloc call. However, calling shm_open in intercepted malloc fails because shm_open requires linking of librt which links to libdl that dlsym in LD_PRELOAD requires. There is a recursive interposition. I thought about creating a static library of wrapped shared memory allocation. Then call it from intercepted malloc. But librt cannot be linked dynamically.

Answer 1

How can I do it?

In general, you can't.

If you want to interpose low-level functions like malloc, your best bet is to only use lower-level direct system calls.

Using anything higher-level, such as shm_open, is bound to run into trouble sooner of later. Even if shm_open didn't use librt and dlopen today, there is nothing that prevents it doing so tomorrow (and breaking your carefully constructed house of cards).

Besides the obvious direct recursion problems, there may also be "order of initialization" problems (e.g. shm_open may require that malloc and librt have initialized, which isn't guaranteed when the very first malloc is called).