Dynamic linking with undefined symbol giving runtime error

[aun...@gmail.com]

I am trying to use a lib(mongodb C driver) in my application but stuck badly in a linking problem. The libs are 4 objects (libbson.a + libbson.so + libmongoc.a + libmongoc.so). My Makefile has libraries added like this. The relevant ones are -lmongoc and -lbson

srv_la_LIBADD = -lrt -lcre2 -lre2 -lcurl -lpthread -lmongoc \
-lbson @MODULES_LIBADD@

The driver libs are compiled and installed successfully at /usr/local/lib. I have even made links to these in /lib and /lib64 directories

The problem is that my application gets compiled and linked successfully but the final srv.so contains UNDEFINED symbols from the mongo C driver(libmongoc.a) which give runtime errors.

Error loading module srv.so:/usr/local/c_icap/lib/c_icap/srv.so: undefined symbol: mongo_cursor_set_query

The linker line from Make process is:

libtool: link: gcc -shared -fPIC -DPIC .libs/srv_la-srv.o
.libs/srv_la-erd.o
.libs/srv_la-erd_list.o
.libs/srv_la-str_util.o
.libs/srv_la-url_util.o
.libs/srv_la-easyzlib.o
.libs/srv_la-state_db.o
.libs/srv_la-data_access_api.o
.libs/srv_la-cJSON.o
.libs/srv_la-clib_es.o
-lrt -lcre2 -lre2 /usr/lib/x86_64-linux-gnu/libcurl.so
-lpthread -lmongoc -lbson -O2 -O2 -Wl,-soname -Wl
,srv.so -o .libs/srv.so


Specifically the objdump of the srv.so says:

root@talha:/webproxy# objdump -t services/.libs/srv.so|grep "mongo"
0000000000000000 *UND* 0000000000000000 mongo_cursor_set_query
0000000000000000 *UND* 0000000000000000 mongo_destroy
0000000000000000 *UND* 0000000000000000 mongo_cmd_authenticate
0000000000000000 *UND* 0000000000000000 mongo_create_index
0000000000000000 *UND* 0000000000000000 mongo_cursor_destroy
0000000000000000 *UND* 0000000000000000 mongo_update
0000000000000000 *UND* 0000000000000000 mongo_cursor_next
0000000000000000 *UND* 0000000000000000 mongo_cursor_init
0000000000000000 *UND* 0000000000000000 mongo_remove
0000000000000000 *UND* 0000000000000000 mongo_connect
0000000000000000 *UND* 0000000000000000 mongo_cursor_bson


Whereas the bson symbols are added correctly. This is strange as both the libs are installed at the same location and have been added in the Makefile.

I tried to link libmongoc.a(static) statically but that results in the linker giving a portability warning of srv.so linking against libmongoc.a and produces final srv.a instead of src.so(but this kind of makes sense). Even then, the symbols are not present in srv.a. Which is even more strange to me.

The the libs were built with -fPIC flag so that part is catered. Any helpful hints? Why is it not linking properly and why is the linker not giving any errors?

[Andrew Haley]

aun...@gmail.com wrote:

> I am trying to use a lib(mongodb C driver) in my application but
> stuck badly in a linking problem. The libs are 4 objects (libbson.a
> + libbson.so + libmongoc.a + libmongoc.so). My Makefile has
> libraries added like this. The relevant ones are -lmongoc and -lbson
>
> srv_la_LIBADD = -lrt -lcre2 -lre2 -lcurl -lpthread -lmongoc \
> -lbson @MODULES_LIBADD@
>
> The driver libs are compiled and installed successfully at
> /usr/local/lib. I have even made links to these in /lib and /lib64
> directories
>
> The problem is that my application gets compiled and linked
> successfully but the final srv.so contains UNDEFINED symbols from
> the mongo C driver(libmongoc.a) which give runtime errors.

Right, so this isn't a linking issue but a runtime paths issue. Use
LD_DEBUG=files to see what the shared library loader is doing. You
need to find out what shared library exports mongo_cursor_set_query
and make sure it's being scanned.

Andrew.

[aun...@gmail.com]

Thanks for the direction.So using LD_DEBUG=files shows that the lib(libmongo.a) which contains the symbol is not being scanned at all. Now what i did was to specify the "-z def" as the linker flag to be sure that all symbols in making my srv.so are resolved. I expected that the linker would give me an error this time since -z def would make sure that nothing unresolved goes into the final lib. But nothing happened. The link went successful. What can be done now.

>
>

[Andrew Haley]

aun...@gmail.com wrote:
>

>> Right, so this isn't a linking issue but a runtime paths issue. Use
>> LD_DEBUG=files to see what the shared library loader is doing. You
>> need to find out what shared library exports mongo_cursor_set_query
>> and make sure it's being scanned.
>
> Thanks for the direction.So using LD_DEBUG=files shows that the
> lib(libmongo.a) which contains the symbol is not being scanned at
> all. Now what i did was to specify the "-z def" as the linker flag
> to be sure that all symbols in making my srv.so are resolved. I
> expected that the linker would give me an error this time since -z
> def would make sure that nothing unresolved goes into the final
> lib. But nothing happened. The link went successful. What can be
> done now.

OK, so a few more things. Use ldd on your lib to see what libraries
it's going to link against. Then use "readelf -d" to see what NEEDED
items there are. You should see mongodb in the NEEDED list.

If mongodb does not appear, the library was not linked correctly, and
you need to look very carefully at the output of make during the
linking stage to see why.

Andrew.

[aun...@gmail.com]

>
> OK, so a few more things. Use ldd on your lib to see what libraries
>
> it's going to link against. Then use "readelf -d" to see what NEEDED
>
> items there are. You should see mongodb in the NEEDED list.
>
>
>
> If mongodb does not appear, the library was not linked correctly, and
>
> you need to look very carefully at the output of make during the
>
> linking stage to see why.
>
>
>
> Andrew.

I figured out that the symbol that i am missing is not located in a shared object(libmongoc.so), rather it is located inside libmongo.a. Now i compared the status of this system with another build machine of mine on which everything works fine. Turns out(result of readelf -d and ldd) that the final lib srv.so is not dependent on libmongoc.so at all.
Below is a comparison of a system on which linking and loading is fine compared to the problematic one:

System working file: readelf -a -s srv.so|grep

0000002a00c0 016b00000007 R_X86_64_JUMP_SLO 000000000006f350 mongo_cursor_set_query + 0
363: 000000000006f350 8 FUNC GLOBAL DEFAULT 11 mongo_cursor_set_query
552: 000000000006f350 8 FUNC GLOBAL DEFAULT 11 mongo_cursor_set_query


System causing problem: output of readelf -a -s srv.so

000000282098 000900000007 R_X86_64_JUMP_SLO 0000000000000000 mongo_cursor_set_query + 0
9: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND mongo_cursor_set_query
477: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND mongo_cursor_set_query


Now what bugs me is that in the makefile i have specified -lmongoc which directs the compiler to link dynamically. The symbol is present in the .a file. How can it work on the system on which it is working correctly? Does the linker fall back to static linking when it cant find a symbol in dynamic lib? In this case how did the linker find the symbol in libmongoc.a whereas on the link line i specified -lmongoc?

[Andrew Haley]

aun...@gmail.com wrote:
>
>
>>
>> OK, so a few more things. Use ldd on your lib to see what libraries
>> it's going to link against. Then use "readelf -d" to see what NEEDED
>> items there are. You should see mongodb in the NEEDED list.
>>
>>
>> If mongodb does not appear, the library was not linked correctly, and
>> you need to look very carefully at the output of make during the
>> linking stage to see why.
>>
>>

> I figured out that the symbol that i am missing is not located in a
> shared object(libmongoc.so), rather it is located inside
> libmongo.a. Now i compared the status of this system with another
> build machine of mine on which everything works fine. Turns
> out(result of readelf -d and ldd) that the final lib srv.so is not
> dependent on libmongoc.so at all.

> Below is a comparison of a system on which linking and loading is fine compared to the problematic one:
>
> System working file: readelf -a -s srv.so|grep
>
> 0000002a00c0 016b00000007 R_X86_64_JUMP_SLO 000000000006f350 mongo_cursor_set_query + 0
> 363: 000000000006f350 8 FUNC GLOBAL DEFAULT 11 mongo_cursor_set_query
> 552: 000000000006f350 8 FUNC GLOBAL DEFAULT 11 mongo_cursor_set_query
>
>
> System causing problem: output of readelf -a -s srv.so
>
> 000000282098 000900000007 R_X86_64_JUMP_SLO 0000000000000000 mongo_cursor_set_query + 0
> 9: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND mongo_cursor_set_query
> 477: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND mongo_cursor_set_query
>
>
> Now what bugs me is that in the makefile i have specified -lmongoc
> which directs the compiler to link dynamically.

No, it doesn't.

> The symbol is present in the .a file. How can it work on the system
> on which it is working correctly? Does the linker fall back to
> static linking when it cant find a symbol in dynamic lib?

I'm guessing that you'll find a libmongo.so that is a linker script
which links against both static and dynamic libs.

Andrew.

[aun...@gmail.com]

What exactly do you mean when you say No,it doesn't?

>
>
>
> > The symbol is present in the .a file. How can it work on the system
>
> > on which it is working correctly? Does the linker fall back to
>
> > static linking when it cant find a symbol in dynamic lib?
>
>
>
> I'm guessing that you'll find a libmongo.so that is a linker script
>
> which links against both static and dynamic libs.
>

You mean to say that libmongoc.so would be a linker script?

>
>
> Andrew.