Weak-linking vs "--as-needed"

Question

I'm having trouble with using a library that contains weak-symbols and the --as-needed linker flag.

Example

(This uses the Jack library)

$ cat <<EOF >myjack.c
#include <jack/weakjack.h>
#include <jack/jack.h>
int main() {
  if (jack_client_opent)
     jack_client_open("foobar", JackNoStartServer, 0, 0);
  else return 1;
  return 0;
}
EOF

$ gcc -o myjack myjack.c  -Wl,--no-as-needed -ljack

$ ./myjack && echo "ok" || echo "K.O."
ok

$ ldd myjack | grep jack
    libjack.so.0 => /usr/lib/x86_64-linux-gnu/libjack.so.0 (0x00007f16f615f000)

$ gcc -o myjack myjack.c  -Wl,--as-needed -ljack

$ ./myjack && echo "ok" || echo "K.O."
K.O.

$ ldd myjack | grep jack

$

(The example code was edited to not segfault any more, as the segfault is not my actually problem)

The problem

It seems that the problem is:

• Jack declares all symbols as weak (if I include <jack/weakjack.h>). this is fine with me; I do want my symbols to stay weak. esp. my program is weakly linking against jack on OSX (-weak_framework Jackmp), which requires to include <jack/weakjack.h>

• When linking with --as-needed, the linker excludes any library, that does not reference at least one non-weak symbol. from the manpage:

--as-needed causes a DT_NEEDED tag to only be emitted for a library that at that point in the link satisfies a non-weak undefined symbol reference from a regular object file

• some OSs (e.g. Ubuntu-16.04LTS) have --as-needed enabled by default.

Now I think that --as-needed is a nice linker feature to get rid of many really unneeded runtime dependencies.

However, I fail to see why a weak dependency is considered as no dependency at all. For me, a weak dependency is to enable optional features. I do want these features to be enabled if possible, and the decision whether this is possible should be a runtime decision. With the current behavior, it becomes a compile-time decision. (If I wanted that, I would simply disable the relevant code via some preprocessor magic).

One solution is obviously to just add --no-as-needed to the linker flags. I don't want this: I do want to get rid of overlinking, if my distribution (or whoever compiles my binary) thinks this is the thing to do.

So I might turn on as-needed after linking in my known-weak library:

  gcc -o myjack myjack.c  -Wl,--no-as-needed -ljack -Wl,--as-needed ...

but this feels wrong as well, as then all libraries after my forced-needed library are suddenly forced to --as-needed (which might not be what my distribution or whoever compiles my binary thinks that this is the thing to do). It also seems to be adding a lot of cruft to the build chain, just because some library happens to export weak symbols only. I do not want to manually track all libraries that do this.

I also could of course simply not include <jack/weakjack.h>. The reason why it is included is because the application also works on OSX, where I do want to optionally depend on the JACK framework (so I link with -weak_framework Jackmp), and keep my program runnable in th absence of that framework.

I really don't want to clutter my application code because of the subtle differences between linkers on various platforms. This is probably the main issue I'm having with all this: why should I add platform-specific code to my application to cater for different linker specifics - I'd probably be OK with adding feature-specific code, e.g. not including weakjack.h if the compiler has no equivalent for -weak_library or -weak_framework; but currently it seems that the closest I can get is something like #ifdef __APPLE__ which makes me shudder in this context).

So I'd really love some option to force libraries that only have weak symbols to be dylinked nevertheless.

Is there such a thing?

Answer 1

I'm having trouble with using a library that contains weak-symbols and the --as-needed linker flag.

No, you're not.

Find out where your libjack.so is, e.g.

$ locate libjack
/usr/lib/x86_64-linux-gnu/libjack.so
/usr/lib/x86_64-linux-gnu/libjack.so.0
/usr/lib/x86_64-linux-gnu/libjack.so.0.1.0
...

Then use nm to examine the symbol types of the JACK API in libjack.so:

$ nm -C -D /usr/lib/x86_64-linux-gnu/libjack.so | grep jack_
000000000000e7e0 T jack_acquire_real_time_scheduling
000000000000d530 T jack_activate
000000000002ccf0 T jack_client_close
000000000000e820 T jack_client_create_thread
....
....
000000000000f340 T jack_uuid_empty
000000000000f320 T jack_uuid_parse
000000000000f2e0 T jack_uuid_to_index
000000000000f330 T jack_uuid_unparse

You'll find they are all type T ( = ordinary global symbol in text section: man nm). There are a few weak symbols in the library:

$ nm -C -D /usr/lib/x86_64-linux-gnu/libjack.so | egrep ' (w|W) '
                 w __cxa_finalize
                 w __gmon_start__
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
                 w _Jv_RegisterClasses
0000000000025410 W std::ctype<char>::do_widen(char) const
0000000000014c10 W void std::vector<unsigned short, std::allocator<unsigned short> >::_M_emplace_back_aux<unsigned short const&>(unsigned short const&)
0000000000014b10 W std::pair<std::_Rb_tree_iterator<unsigned short>, bool> std::_Rb_tree<unsigned short, unsigned short, std::_Identity<unsigned short>, std::less<unsigned short>, std::allocator<unsigned short> >::_M_insert_unique<unsigned short>(unsigned short&&)
0000000000014ad0 W std::_Rb_tree<unsigned short, unsigned short, std::_Identity<unsigned short>, std::less<unsigned short>, std::allocator<unsigned short> >::_M_erase(std::_Rb_tree_node<unsigned short>*)

But none of them are in the JACK API. Nothing you can do short of rebuilding your libjack.so is going to change that. The right way to characterise your problem is:

I'm having trouble linking a library with the --as-needed linker flag to a program in which I decided to weaken all my references to that library

The defining symbol references of the JACK API in libjack.so are all strong. You have written a program directing the compiler to emit, in your object code, symbols that are weak undefined references to the JACK API, and you are finding that, with as-needed linkage, these weak references fail to compel linkage of libjack.so to provide their missing definitions.

it seems that the problem is:

jack declares all symbols as weak (if I include ).

when linking with --as-needed, the linker excludes any library, that does not reference at least one non-weak symbol.

some OSs (e.g. Ubuntu-16.04LTS) have --as-needed enabled by default.

The last two points are correct. The schism between distros that link shared libraries as-needed by default and distros that don't goes back to Debian Wheezy, 2013, which went over to as-needed. Since then, the Debian-derived distro-clan has followed suit while the RedHat/Fedora clan has stuck with the status quo ante.

The first point is confused. libjack.so, as we've noted, exports a strongly defined JACK API that you cannot alter by writing and compiling new code. If you include <jack/weakjack.h> in one of your source files, then you are declaring all JACK API symbols weak, in your code, and the compiler will give you an object file that contains only weak references to the JACK API. <jack/weakjack.h> just defines macros that have that effect.

It would be surprising if an old and major linux library like libjack has botched its adaptation to the as-needed schism. I suspect you've overlooked some of the small print about jack/weakjack.h:

Detailed Description

One challenge faced by developers is that of taking advantage of new features introduced in new versions of [ JACK ] while still supporting older versions of the system. Normally, if an application uses a new feature in a library/API, it is unable to run on earlier versions of the library/API that do not support that feature. Such applications would either fail to launch or crash when an attempt to use the feature was made. This problem cane be solved using weakly-linked symbols.

...

A concrete example will help. Suppose that someone uses a version of a JACK client we'll call "Jill". Jill was linked against a version of JACK that contains a newer part of the API (say, jack_set_latency_callback()) and would like to use it if it is available.

When Jill is run on a system that has a suitably "new" version of JACK, this function will be available entirely normally. But if Jill is run on a system with an old version of JACK, the function isn't available.

With normal symbol linkage, this would create a startup error whenever someone tries to run Jill with the "old" version of JACK. However, functions added to JACK after version 0.116.2 are all declared to have "weak" linkage which means that their abscence doesn't cause an error during program startup. Instead, Jill can test whether or not the symbol jack_set_latency_callback is null or not. If its null, it means that the JACK installed on this machine is too old to support this function. If its not null, then Jill can use it just like any other function in the API. For example:

if (jack_set_latency_callback) {
    jack_set_latency_callback (jill_client, jill_latency_callback, arg);
}

However, there are clients that may want to use this approach to parts of the JACK API that predate 0.116.2. For example, they might want to see if even really old basic parts of the API like jack_client_open() exist at runtime.

Such clients should include <jack/weakjack.h> before any other JACK header. This will make the entire JACK API be subject to weak linkage, so that any and all functions can be checked for existence at runtime. It is important to understand that very few clients need to do this - if you use this feature you should have a clear reason to do so.

[emphasis added]

This makes clear that a program, like yours, that takes the exceptional step of of including jack/weakjack.h for the purpose of weakening its references to the entire JACK API can be expected to run successfully only if it tests the definedness of every JACK API symbol before referencing it and handles the case in which it is not defined. Your program does not conform. This one does:

myjack1.c

#include <jack/weakjack.h>
#include <jack/jack.h>
#include <stdio.h>

int main() {
    if (jack_client_open) { 
        jack_client_open("foobar", JackNoStartServer, 0, 0);
    } else {
        puts("`jack_client_open` is not available");
    }
    return 0;
}

and do does this one:

myjack2.c

#include <jack/weakjack.h>
#include <jack/jack.h>
#include <dlfcn.h>
#include <stdlib.h>
#include <stdio.h>

int main() {
    jack_client_t * (*jack_client_open_fp)
        (const char *, jack_options_t,jack_status_t *,...) = jack_client_open;

    if (!jack_client_open_fp) {
        void * dsoh = dlopen("libjack.so",RTLD_LAZY);
        if (!dsoh) {
            fputs("`libjack` is not available\n",stderr);
            exit(EXIT_FAILURE);
        }
        *(void**)(&jack_client_open_fp) = dlsym(dsoh,"jack_client_open");
        if (!jack_client_open_fp) {
            fputs("`jack_client_open` is not available\n",stderr);
            exit(EXIT_FAILURE);
        }
    }
    jack_client_open_fp("foobar", JackNoStartServer, 0, 0);
    exit(EXIT_SUCCESS);
}

which sketches the usual approach to a discoverable API - apt for a program meant to install and run on a system that might not provide libjack at all. So you'd build it without reference to libjack like:

gcc -o myjack2 myjack2.c -ldl

and on Ubuntu 17.04 - which does provide libjack - it might run like:

$ ./myjack2 
Cannot connect to server socket err = No such file or directory
Cannot connect to server request channel
jack server is not running or cannot be started
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for 4294967295, skipping unlock
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for 4294967295, skipping unlock

So the library's T&Cs are in good order with respect to as-needed linkage. That seems to leave you in a position of being independently dissatisfied that as-needed linkage works the way it does, rather than in a different way that would allow you to weaken all your references to the JACK API and still get libjack to be needed by your weak references to its API symbols:-

I fail to see why a weak dependency is considered as no dependency at all. For me, a weak dependency is to enable optional features. I do want these features to be enabled if possible, and the decision whether this is possible should be a runtime decision. With the current behavior, it becomes a compile-time decision.

Your view that a weak symbol reference gives rise to a weak linkage dependency on a library that defines the symbol does not have a footing for the GNU linker. A program depends on a library if its linkage needs a symbol definition that the libary provides; otherwise it doesn't depend on that libary: there aren't weak and strong degrees of dependency. (The Darwin Mach-O linker does support a cognate distinction)

There are weak symbols, as opposed to the default and usual kind, which is strong. {weak|strong} symbol is shorthand for {weakly|strongly} referenced symbol, since the same symbol may be referenced in multiple linker input files, sometimes or always weakly and sometimes or always strongly.

A strong symbol must have exactly one defining reference in the linkage.

A weak symbol is such that:

• The linker is not obliged to find a definition for it: it may remain undefined in the output file

• The linker is not obliged to fault multiple weak definitions of the same symbol in different input files. If exactly one defining reference within the linkage is strong, then that strong definition is picked and all weak ones ignored. If all defining references in the linkage are weak then the linker will pick one at random.

From the first part of that it follows that an undefined weak reference to a symbol does not give rise to a linkage dependency at all. A definition is not needed and the fact that a definition is not needed is the result of a decision by the programmer (e.g. #include <jack/weak_jack.h>) or perhaps by the compiler. It is not reasonable to expect that the linker, if directed to link only shared libraries that are needed, should then link libraries to furnish definitions of symbols for which you or the compiler have told it that definitions are not needed.

If the linker were to behave like that in your case, that would constitute a linktime decision to freeze and enable an API which, by including jack/weak_jack.h, you have indicated you wish to reserve entirely for runtime discovery.

Linking your problem program with -no-as-needed is successful as a way of smothering the bug in the program. The bug is that by including jack/weak_jack.h you commit yourself to runtime discovery of the whole API, but don't fulfil that committment and instead take the availability of the API for granted. Hence the segfault with as-needed linkage. Linking with -no-as-needed just cancels the the effect of including jack/weak_jack.h. Including it says your program doesn't need any of the API definitions: -no-as-needed says, whatever they are, you're getting them all anyway.

In the light of the fact that all JACK APIs post version 0.116.2 are weakly defined without resort to jack/weak_jack.h, I think that you simply don't have any use for this header unless you are indeed planning a program that will do something worthwhile on a host from which libjack is missing. If you are planning that, then you have no alternative to runtime discovery of all the JACK APIs you use, regardless of linkage conventions, because you can't link libjack anyway.

If not, then just link libjack and, if you merely call jack_client_open, your program, on any host, will dynamically link all the API definitions, whatever they are on that host, because your reference to jack_client_open (in the absence of <jack/weak_jack.h>) will make libjack needed, whether that matters to the linker that did the linking or not. If you want to be compatible accross API versions, then you need to implement runtime detection as documented of any API that is documented with the attribute JACK_WEAK_EXPORT - as opposed to JACK_OPTIONAL_WEAK_EXPORT, or JACK_OPTIONAL_WEAK_DEPRECATED_EXPORT: the latter denote fundamental APIs that can only be forcibly weakened via <jack/weak_jack.h>.