toolchain static link (libdl.a) issue

[JB Data31]

Hi NDK users,

I'm struggling with toolchain to get a static, arm64 binary.

- Some environment variables to point NDK, SYSROOT, PREFIX...

- Pass the arm64 additional libs creation.

To not bother readers with the whole config, test, libtool issues... I present the problem in a raw way. 

The final command to get binary :

$ aarch64-linux-android-g++ --sysroot /home/jbd/dev/25-android/ndk-toolchain/sysroot -fopenmp -g -O2 -std=c++14 -static-libstdc++ -static -L/home/jbd/dev/25-android/ndk-toolchain/sysroot/usr/lib/aarch64-linux-android -L/home/jbd/dev/25-android/leptonica-arm64/lib -L/home/jbd/dev/25-android/libjpeg-arm64/lib -L/home/jbd/dev/25-android/libtiff-arm64/lib -L/home/jbd/dev/25-android/libpng-arm64/lib -L/home/jbd/dev/25-android/libzlib-arm64/lib -ldl tesseract-tesseractmain.o ./.libs/libtesseract.a /home/jbd/dev/25-android/ndk-toolchain/sysroot/home/jbd/dev/25-android/leptonica-arm64/lib/liblept.a /home/jbd/dev/25-android/ndk-toolchain/sysroot/home/jbd/dev/25-android/libpng-arm64/lib/libpng16.a /home/jbd/dev/25-android/ndk-toolchain/sysroot/home/jbd/dev/25-android/libjpeg-arm64/lib/libjpeg.a /home/jbd/dev/25-android/ndk-toolchain/sysroot/home/jbd/dev/25-android/libtiff-arm64/lib/libtiff.a -lm -lz -o tesseract

The issue :

/home/jbd/dev/25-android/ndk-toolchain/lib64/clang/8.0.2/lib/linux/aarch64/libomp.a(ompt-general.cpp.o): In function `ompt_start_tool':
ompt-general.cpp:(.text.ompt_start_tool+0x20): undefined reference to `dlsym'
/home/jbd/dev/25-android/ndk-toolchain/lib64/clang/8.0.2/lib/linux/aarch64/libomp.a(ompt-general.cpp.o): In function `ompt_pre_init':
ompt-general.cpp:(.text.ompt_pre_init+0xf0): undefined reference to `dlopen'
ompt-general.cpp:(.text.ompt_pre_init+0xfc): undefined reference to `dlsym'
clang80++: error: linker command failed with exit code 1 (use -v to see invocation)

Checks :

$ nm /home/jbd/dev/25-android/ndk-toolchain/sysroot/usr/lib/aarch64-linux-android/libdl.a
libdl_static.o:
...

0000000000000000 T dladdr
0000000000000000 T dlclose
0000000000000000 T dlerror
0000000000000000 T dlopen
0000000000000000 T dlsym
0000000000000000 T dlvsym
...

Days of tests:

Place -ldl every where in the command line...

Test a "magic"  -Wl,--no-as-needed -ldl extra params...

Try locate where the initial lib (libomp.a) that needs libdl.a "in" aarch64-linux-android-g++, clang80++...

LD_LIBRARY_PATH...

...

My feeling is because libcomp needs libdl and libcomp comes with clang80++ (no -lcomp in command line) the tuning is around NDK toolchain.

Help appreciate.

JB.

[Alex Cohn]

Please see https://github.com/android-ndk/ndk/issues/965. TL;NR: libdl.a in NDK is only a placeholder to help you link a binary that will actually never try to dlopen() something.

BR,

Alex

[JB Data31]

Thanks for your answer.

My first struggle is I can't link, I'm not yet executing my static binary.

Reading your posts, I swap to cmake, my fail link was with classic autogen, configure, make : I don't understand why aarch64-linux-android-g++ doesn't evaluate -L... -ldl, no matter it's a stub, it's a valid lib (nm proof).

Using cmake, I regress :-(.

My now error is in compile when executing make in build directory after passing cmake:

/home/jbd/soft/android/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/c++/v1/cmath:314:9: error: no member named 'signbit' in the global namespace; did you mean 'sigwait'?

As suggest by google, I try cmake option -DCMAKE_CXX_FLAGS="-std=c++11 -I/usr/include", because a similar issue on MacOS.

But I'm on Ubuntu and my cmake build show -- Used standard: C++17.

Help appreciate x2.

[Dan Albert]

My first struggle is I can't link, I'm not yet executing my static binary.

I think Alex's point was more that even when you do get it to build, that might not help. Depends on how (if) dlsym is being used in the app.

To get it to build, you probably just need to update your NDK. Start there. If you still have issues that's an easier place to work from.

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk...@googlegroups.com.
To post to this group, send email to andro...@googlegroups.com.
Visit this group at https://groups.google.com/group/android-ndk.
To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/5aebbaad-b2a4-47de-a914-2fdfa6ba56c7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[JB Data31]

Thanks for your answer.

I upgrade NDK : Installed Version: 19.2.5345600 to Available Version: 20.0.5594570.

Same issue:

The link command:

libtool: link: aarch64-linux-android-g++ --sysroot /home/jbd/dev/25-android/ndk-toolchain/sysroot -g -O2 -std=c++17 -static-libstdc++ -static -o tesseract tesseract-tesseractmain.o -fopenmp  -L/home/jbd/dev/25-android/ndk-toolchain/sysroot/usr/lib/aarch64-linux-android -L/home/jbd/dev/25-android/leptonica-arm64/lib -L/home/jbd/dev/25-android/libjpeg-arm64/lib -L/home/jbd/dev/25-android/libtiff-arm64/lib -L/home/jbd/dev/25-android/libpng-arm64/lib -L/home/jbd/dev/25-android/libzlib-arm64/lib -ldl ./.libs/libtesseract.a /home/jbd/dev/25-android/ndk-toolchain/sysroot/home/jbd/dev/25-android/leptonica-arm64/lib/liblept.a /home/jbd/dev/25-android/ndk-toolchain/sysroot/home/jbd/dev/25-android/libpng-arm64/lib/libpng16.a /home/jbd/dev/25-android/ndk-toolchain/sysroot/home/jbd/dev/25-android/libjpeg-arm64/lib/libjpeg.a /home/jbd/dev/25-android/ndk-toolchain/sysroot/home/jbd/dev/25-android/libtiff-arm64/lib/libtiff.a -lm -lz -fopenmp

The error:

/home/jbd/dev/25-android/ndk-toolchain/lib64/clang/8.0.2/lib/linux/aarch64/libomp.a(ompt-general.cpp.o): In function `ompt_start_tool':
ompt-general.cpp:(.text.ompt_start_tool+0x20): undefined reference to `dlsym'
/home/jbd/dev/25-android/ndk-toolchain/lib64/clang/8.0.2/lib/linux/aarch64/libomp.a(ompt-general.cpp.o): In function `ompt_pre_init':
ompt-general.cpp:(.text.ompt_pre_init+0xf0): undefined reference to `dlopen'
ompt-general.cpp:(.text.ompt_pre_init+0xfc): undefined reference to `dlsym'

An analysis:

1/ libdl.a

The libdl.a "stud" lib is OK, -L and -l libtool parameters too.

$ nm /home/jbd/dev/25-android/ndk-toolchain/sysroot/usr/lib/aarch64-linux-android/libdl.a | grep dlsym
0000000000000000 T dlsym

2/ libcomp.a

There's no -lomp parameter in libtool command line so the lib libomp.a comes with aarch64-linux-android-g++.

why -ldl not ? Furthermore -ldl parameter in libtool (LDFLAGS) has no effect...

I'm stuck.

To unsubscribe from this group and stop receiving emails from it, send an email to andro...@googlegroups.com.

[Dan Albert]

That's not r19. r19 doesn't have g++.

Could you also share how you're building? Configure flags, etc?

To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk...@googlegroups.com.

To post to this group, send email to andro...@googlegroups.com.
Visit this group at https://groups.google.com/group/android-ndk.

To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/fe3b7f09-aef4-4bf9-90ba-0fd435691548%40googlegroups.com.

[Ryan Prichard]

I think they're using the standalone toolchain's aarch64-linux-android-g++ wrapper script. The lib64/clang/8.0.2 directory only exists in r19.

On Friday, June 7, 2019 at 9:59:30 AM UTC-7, Dan Albert wrote:

That's not r19. r19 doesn't have g++.

Could you also share how you're building? Configure flags, etc?

[Dan Albert]

Ah, right. I forgot we still installed those aliases.

Still need to know configure flags and whatnot.

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk...@googlegroups.com.
To post to this group, send email to andro...@googlegroups.com.
Visit this group at https://groups.google.com/group/android-ndk.

To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/e920d7c0-0ae1-40dd-8a36-a44d158b863a%40googlegroups.com.

[JB Data31]

I took a little time cause I want to avoid side effects due to days of trys and tests.

Now I am NDK 20.0.5594570, according to python toolchain script output I use NDK prebuilts, no aarch64-linux-android-g++ anymore in build process.

I want to static build products, which need 3rd-part libs.

Now 6 of 9 build process are OK : compile, lib. But not yet static binary.

Here are the issues with env and analysis, it can help:

Global envs

CC = aarch64-linux-android28-clang
CXX = aarch64-linux-android28-clang++
AR = ...

TARGET_HOST = aarch64-linux-android

$ $CC --version
Android (5220042 based on r346389c) clang version 8.0.7 (https://android.googlesource.com/toolchain/clang b55f2d4ebfd35bf643d27dbca1bb22895700817) (https://android.googlesource.com/toolchain/llvm 3c393fe7a7e13b0fba4ac75a01aa683d7a5b11cd) (based on LLVM 8.0.7svn)
Target: aarch64-unknown-linux-android28

$ which $CC
.../ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android28-clang

Issues

1/ configure, system include path

CFLAGS = -static -I... (all 3th-part includes)

LDFLAGS = -all-static -L...(all 3th-part libs)

$ ./configure --host=$TARGET_HOST

...

checking for sysroot... no

...

Configuration is done.

$ make
make  all-recursive
make[1]: Entering directory '/home/jbd/dev/25-android/leptonica-src'
Making all in src
make[2]: Entering directory '/home/jbd/dev/25-android/leptonica-src/src'
  CC       adaptmap.lo
In file included from adaptmap.c:134:
In file included from ./allheaders.h:35:
In file included from ./alltypes.h:31:
In file included from /home/jbd/soft/android/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/stdio.h:41:
/usr/include/x86_64-linux-gnu/sys/cdefs.h:41:6: error: function-like macro '__GNUC_PREREQ' is not defined
# if __GNUC_PREREQ (4, 6) && !defined _LIBC
     ^
/usr/include/x86_64-linux-gnu/sys/cdefs.h:54:30: error: function-like macro '__GNUC_PREREQ' is not defined
# if !defined __cplusplus && __GNUC_PREREQ (3, 3)

Why use of sysroot/usr/include with configure sysroot test saying no.

Anyway, if stdio.h comes from NDK sysroot/usr/include, cdefs.h have to come from NDK sysroot/usr/include/sys too. that's a freaky issue, output says sys/cdefs.h is the native one (/usr/include/x86_64-linux-gnu).

For information, I pass this build, with NDK19 and g++, using gnu native system params... But with NDK20, back using g++ from tool-chain python script, I can't anymore, good because it was not a proper way...

2/ cmake struggles

  2.1/ native cmake, a warmup way.

$ /usr/local/bin/cmake --version
cmake version 3.14.5

CFLAGS = -static -I... (all 3th-part includes)

LDFLAGS = -all-static -L...(all 3th-part libs)

$ cmake .. -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_C_COMPILER="$CC" -DCMAKE_CXX_COMPILER="$CXX" -DCMAKE_LINKER="$LD" -DCMAKE_AR="$AR" -DCMAKE_C_ARCHE_CREATE="<CMAKE_AR> -qa <TARGET> <OBJECTS>" -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -DCMAKE_E_LINKER_FLAGS="-static" -DCMAKE_C_FLAGS="$CFLAGS" -DCMAKE_STATIC_LINKER_FLAGS="$LDFLAGS"

OK

$ make

...

[ 44%] Linking C static library ../../../bin/libopenjp2.a
cd /home/jbd/dev/25-android/openjpeg-src/build/src/lib/openjp2 && /usr/local/bin/cmake -P CMakeFiles/openjp2.dir/cmake_clean_target.cmake
cd /home/jbd/dev/25-android/openjpeg-src/build/src/lib/openjp2 && /usr/local/bin/cmake -E cmake_link_script CMakeFiles/openjp2.dir/link.txt --verbose=1
/home/jbd/dev/25-android/openjpeg-src/build/aarch64-linux-android-ar -qa ../../../bin/libopenjp2.a CMakeFiles/openjp2.dir/thread.c.o CMakeFiles/openjp2.dir/bio.c.o CMakeFiles/openjp2.dir/cio.c.o CMakeFiles/openjp2.dir/dwt.c.o CMakeFiles/openjp2.dir/event.c.o CMakeFiles/openjp2.dir/image.c.o CMakeFiles/openjp2.dir/invert.c.o CMakeFiles/openjp2.dir/j2k.c.o CMakeFiles/openjp2.dir/jp2.c.o CMakeFiles/openjp2.dir/mct.c.o CMakeFiles/openjp2.dir/mqc.c.o CMakeFiles/openjp2.dir/openjpeg.c.o CMakeFiles/openjp2.dir/opj_clock.c.o CMakeFiles/openjp2.dir/pi.c.o CMakeFiles/openjp2.dir/t1.c.o CMakeFiles/openjp2.dir/t2.c.o CMakeFiles/openjp2.dir/tcd.c.o CMakeFiles/openjp2.dir/tgt.c.o CMakeFiles/openjp2.dir/function_list.c.o CMakeFiles/openjp2.dir/opj_malloc.c.o CMakeFiles/openjp2.dir/sparse_array.c.o
/home/jbd/dev/25-android/openjpeg-src/build/aarch64-linux-android-ar: CMakeFiles/openjp2.dir/thread.c.o: File format not recognized

however, the files seem to be correct.

$ file ./CMakeFiles/openjp2.dir/thread.c.o
./CMakeFiles/openjp2.dir/thread.c.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), not stripped

For information, I link aarch64-linux-android-ar to local path because of astonishing CMAKE_AR behavior.

Stuck, so B-plan.

  2.2/ SDK cmake, the proper way.

$ $SDK_PATH/cmake/3.10.2.4988404/bin/cmake --version
cmake version 3.10.2

$ $SDK_PATH/cmake/3.10.2.4988404/bin/cmake .. -DCMAKE_VERBOSE_MAKEFILE=ON -DANDROID_NDK=$NDK_PATH -DCMAKE_TOOLCHAIN_FILE=$NDK_PATH/build/cmake/android.toolchain.cmake -DANDROID_TOOLCHAIN=clang -DCMAKE_MAKE_PROGRAM=$SDK_PATH/cmake/3.10.2.4988404/bin/ninja -DANDROID_NATIVE_API_LEVEL=24 -DCMAKE_LIBRARY_OUTPUT_DIRECTORY=$OPENJPEG_PREFIX -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -DCMAKE_EXE_LINKER_FLAGS="-static" -DCMAKE_C_FLAGS="$CFLAGS" -DCMAKE_STATIC_LINKER_FLAGS="$LDFLAGS"
-- Check for working C compiler: /home/jbd/soft/android/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/clang
-- Check for working C compiler: /home/jbd/soft/android/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -- broken
CMake Error at /home/jbd/soft/android/cmake/3.10.2.4988404/share/cmake-3.10/Modules/CMakeTestCCompiler.cmake:52 (message):
  The C compiler

    "/home/jbd/soft/android/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/clang"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /home/jbd/dev/25-android/openjpeg-src/build/CMakeFiles/CMakeTmp

    Run Build Command:"/home/jbd/soft/android/cmake/3.10.2.4988404/bin/ninja" "cmTC_856eb/fast"
    ninja: error: loading 'build.ninja': No such file or directory

No build.ninja file in SDK, NDK... 

Final round

CFLAGS = -static -I... (all 3th-part includes)

LDFLAGS = -static-libstdc++ -all-static -L (all 3th-part libs)

$ ./configure --host=$TARGET_HOST

...

Configuration is done.

$ make

...

10 minutes of compile OK, local lib OK, but can't build binary because of former 3th-part fails (1/ , 2/).

To unsubscribe from this group and stop receiving emails from it, send an email to andro...@googlegroups.com.

[Dan Albert]

output says sys/cdefs.h is the native one (/usr/include/x86_64-linux-gnu).

That's probably the core of your issues. Can you find the actual compile command used? That would give a pretty good idea of what went wrong (even better would be the compile command plus the -v compile output; that'll include a bunch of information about what include paths it chose and why).

/home/jbd/dev/25-android/openjpeg-src/build/aarch64-linux-android-ar: CMakeFiles/openjp2.dir/thread.c.o: File format not recognized

Any chance you're using LTO? If you build with -flto then you need to either use llvm-ar (which I don't think we even ship currently) or have the archiver plug in available. We install the archiver plugin to the NDK as needed, but it looks like this build maybe copied the archiver into a different directory? It finds the LTO plugin based on a path relative to the archiver executable, so if that wasn't also copied, that could be the issue.

-- Check for working C compiler: /home/jbd/soft/android/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -- broken
CMake Error at /home/jbd/soft/android/cmake/3.10.2.4988404/share/cmake-3.10/Modules/CMakeTestCCompiler.cmake:52 (message):
  The C compiler

    "/home/jbd/soft/android/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/clang"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /home/jbd/dev/25-android/openjpeg-src/build/CMakeFiles/CMakeTmp

    Run Build Command:"/home/jbd/soft/android/cmake/3.10.2.4988404/bin/ninja" "cmTC_856eb/fast"
    ninja: error: loading 'build.ninja': No such file or directory

That's a weird one. tbh that looks like bad error handling and the actual problem was something else. Would need a more detailed log to know though. 

To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk...@googlegroups.com.

To post to this group, send email to andro...@googlegroups.com.
Visit this group at https://groups.google.com/group/android-ndk.

To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/a93bd6ea-d598-4cfb-8b77-ac1b469c6ec1%40googlegroups.com.

[JB Data31]

7 3th-part libs builds OK, 1 static binary build and test on my Androïd device OK.

Back to former issue building the final static binary.

/home/jbd/soft/android/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/8.0.7/lib/linux/aarch64/libomp.a(ompt-general.cpp.o): In function `ompt_start_tool':

ompt-general.cpp:(.text.ompt_start_tool+0x20): undefined reference to `dlsym'

/home/jbd/soft/android/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/8.0.7/lib/linux/aarch64/libomp.a(ompt-general.cpp.o): In function `ompt_pre_init':

ompt-general.cpp:(.text.ompt_pre_init+0xf0): undefined reference to `dlopen'
ompt-general.cpp:(.text.ompt_pre_init+0xfc): undefined reference to `dlsym'

clang++: error: linker command failed with exit code 1 (use -v to see invocation)

Using debug, verbose, -v and so one, here is my work around.

In the .../aarch64-linux-android/bin/ld command, the part of option -start-group -lomp -lgcc -lc --end-group is incomplete and cause the issue.

The proper option snipett is --start-group -ldl -lomp -lgcc -lc --end-group to add missing symbols of libomp.a.

A way to build the binary is to add a shell got from libtool verbose output, add -ldl and execute after make fails.

Test on device OK.

Looking for -lomp, it doesn't come from configure.

$ grep lomp config.log config.status

$

According to me there's an issue, -lomp advent should be -ldl -lomp because .../lib/linux/aarch64/libomp.a needs symbol of .../arch-arm64/usr/lib/libdl.a. configure is not a way to do such an extra param.

[Alex Cohn]

If you can push -ldl to the end of your libs list, that would be enough. You don't need it inside the -start-group … --end-group.  Automake let you use LDADD to specify this. I would suggest to set

LDADD = -lomp -ldl 

BR,

Alex Cohn

[JB Data31]

Thanks for help.

Build process OK.