Linker failed to find a dependent lib (dlopen failed problem).

[Horatio Chien]

I am developing a 64-bit, API 27, Qt-based vendor app.

This app won't be installed by "adb install" but by copying to /vendor/app/<MyQtApp>.

I am getting a dlopen failed problem as following.

01-01 01:01:36.908  4190  4190 D QtJAVA  : Try to load /vendor/app/MyQtApp/plugins/platforms/android/libqtforandroid.so

01-01 01:01:36.908  4190  4190 D linker  : dlopen(name="/vendor/app/MyQtApp/plugins/platforms/android/libqtforandroid.so", flags=0x2, extinfo=[flags=0x200, reserved_addr=0x356, reserved_size=0x7fcd34dd40, relro_fd=-1739459248, library_fd=125, library_fd_offset=0x7fcd34dd00, library_namespace=classloader-namespace@0x7e1db0d210], caller="/system/lib64/libnativeloader.so", caller_ns=(default)@0x7e1dc443e0) ...

01-01 01:01:36.910  4190  4190 D linker  : ... dlopen failed: library "libQt5Gui.so" not found

01-01 01:01:36.910  4190  4190 D linker  : dlerror set to "dlopen failed: library "libQt5Gui.so" not found"

01-01 01:01:36.911  4190  4190 E QtJAVA  : UnsatisfiedLinkError '/vendor/app/MyQtApp/plugins/platforms/android/libqtforandroid.so'

01-01 01:01:36.911  4190  4190 E QtJAVA  : java.lang.UnsatisfiedLinkError: dlopen failed: library "libQt5Gui.so" not found

Before this exception, I had:

01-01 01:01:36.869  4190  4190 D QtJAVA  : Try to load /system/priv-app/MyQtApp/lib/arm64/libQt5Gui.so

01-01 01:01:36.869  4190  4190 D linker  : dlopen(name="/system/priv-app/MyQtApp/lib/arm64/libQt5Gui.so", flags=0x2, extinfo=[flags=0x200, reserved_addr=0x356, reserved_size=0x7fcd34dd40, relro_fd=-1739459248, library_fd=125, library_fd_offset=0x7fcd34dd00, library_namespace=classloader-namespace@0x7e1db0d210], caller="/system/lib64/libnativeloader.so", caller_ns=(default)@0x7e1dc443e0) ...

01-01 01:01:36.870  4190  4190 D linker  : ... dlopen calling constructors: realpath="/vendor/app/MyQtApp/lib/arm64/libQt5Gui.so", soname="libQt5Gui.so", handle=0x65acb23b20126b7f

01-01 01:01:36.870  4190  4190 D linker  : ... dlopen successful: realpath="/vendor/app/MyQtApp/lib/arm64/libQt5Gui.so", soname="libQt5Gui.so", handle=0x65acb23b20126b7f

01-01 01:01:36.870  4190  4190 D linker  : dlsym(handle=0x65acb23b20126b7f("/vendor/app/MyQtApp/lib/arm64/libQt5Gui.so"), sym_name="JNI_OnLoad", sym_ver="(null)", caller="/system/lib64/libart.so", caller_ns=(default)@0x7e1dc443e0) ...

01-01 01:01:36.870  4190  4190 D linker  : ... dlsym successful: sym_name="JNI_OnLoad", sym_ver="(null)", found in="libQt5Core.so", address=0x7d7c686c78

01-01 01:01:36.870  4190  4190 I QtJAVA  : Successfully loaded /system/priv-app/MyQtApp/lib/arm64/libQt5Gui.so

Note that there is a symlink /system/priv-app/MyQtApp -> /vendor/app/MyQtApp

The files actually resides in /vendor/app/MyQtApp/, i.e. they are:

/vendor/app/MyQtApp/lib/arm64/libQt5Gui.so

/vendor/app/MyQtApp/plugins/platforms/android/libqtforandroid.so

My Java code:

Log.d(QtTAG, "Try to load " + libName); // libName shall be an absolute path

File f = new File(libName);
if (f.exists() && f.isFile() && f.canRead()) {
    System.load(libName);
    Log.i(QtTAG, "Successfully loaded " + libName);
}

The info. of libqtforandroid.so:

$ aarch64-linux-android-readelf.exe -d libqtforandroid.so

Dynamic section at offset 0x102df8 contains 37 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libjnigraphics.so]
 0x0000000000000001 (NEEDED)             Shared library: [libandroid.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc++_shared.so]
 0x0000000000000001 (NEEDED)             Shared library: [liblog.so]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so]
 0x0000000000000001 (NEEDED)             Shared library: [libEGL.so]
 0x0000000000000001 (NEEDED)             Shared library: [libQt5Gui.so]
 0x0000000000000001 (NEEDED)             Shared library: [libQt5Core.so]
 0x0000000000000001 (NEEDED)             Shared library: [libGLESv2.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so]
 0x000000000000000e (SONAME)             Library soname: [libqtforandroid.so]
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/../../../lib]

It has a rpath and it indeed depends on libQt5Gui.so.

P.S.

nativeLibraryDir of ApplicationInfo is: /system/priv-app/MyQtApp/lib/arm64

My questions:

1. Why does linker want to load libQt5Gui.so again when libQt5Gui.so has been loaded (in the same for loop)?

2. Why does linker fail to load libQt5Gui.so when loading libqtforandroid.so?

Thank you.

--

The above code is in Java but debugging dlopen seems very ndk-related. I hope I can put this post here.

[J Decker]

rpath: [$ORIGIN/../../../lib]
/vendor/app/MyQtApp/lib/arm64/libQt5Gui.so

where is the arm64 in rpath?

 

P.S.

nativeLibraryDir of ApplicationInfo is: /system/priv-app/MyQtApp/lib/arm64

My questions:

1. Why does linker want to load libQt5Gui.so again when libQt5Gui.so has been loaded (in the same for loop)?

2. Why does linker fail to load libQt5Gui.so when loading libqtforandroid.so?

Thank you.

--

The above code is in Java but debugging dlopen seems very ndk-related. I hope I can put this post here.

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk...@googlegroups.com.
To post to this group, send email to andro...@googlegroups.com.
Visit this group at https://groups.google.com/group/android-ndk.
To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/68b2936d-5d46-48ac-bdff-e0f95030f108%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Horatio Chien]

There is none but I have been told that Android linker does not support RPATH and RUNPATH at all. Android linker does not use them. Is there new change in recent new Android version?

J Decker於 2018年9月12日星期三 UTC+8下午10時57分02秒寫道：

[Ryan Prichard]

Android's linker doesn't support DT_RPATH, but it does support DT_RUNPATH starting with Nougat.

To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/e2bf2257-da28-400e-9b40-a818fdb4a714%40googlegroups.com.

[Horatio Chien]

Thank you for the information, DT_RUNPATH is not used in this case, though.

Ryan Prichard於 2018年9月14日星期五 UTC+8上午5時54分19秒寫道：

[Andrew Esh]

Back in Android 7.0, new security restrictions began to be put in place to prevent the loading of many libraries. See this for particulars:

https://developer.android.com/about/versions/nougat/android-7.0-changes#ndk

You could look in logcat output for "denied" to see if selinux in enforcing policy against loading the library. The messages would look (sort of) like this:

walleye:/ $ logcat -d | grep denied                                                                                                                                                                            

09-17 12:22:58.733  5319  5319 W unknown : type=1400 audit(0.0:17): avc: denied { execute } for comm=44756F4267 ... 333 name="thermal-engine" dev="dm-1" ino=276 scontext=u:r:untrusted_app_27:s0:c512,c768 tcontext=u:object_r:thermal-engine_exec:s0 tclass=file permissive=0

09-17 12:22:58.736  5321  5321 W unknown : type=1400 audit(0.0:18): avc: denied { execute } for comm=44756F4267 ... 333 name="thermal-engine" dev="dm-1" ino=276 scontext=u:r:untrusted_app_27:s0:c512,c768 tcontext=u:object_r:thermal-engine_exec:s0 tclass=file permissive=0

09-17 12:22:59.816  5417  5417 W service: type=1400 audit(0.0:19): avc: denied { ioctl } for path="socket:[61632]" dev="sockfs" ino=61632 ioctlcmd=8927 scontext=u:r:untrusted_app_27:s0:c512,c768 tcontext=u:r:untrusted_app_27:s0:c512,c768 tclass=udp_socket permissive=0

(These messages are for other denials, not a library load.)

You could test if selinux is denying access by turning it off. In an adb shell, do "setenforce 0". If you can then load the library, then it is an sepolicy problem.

And, I wonder if you are aware that there has been a randomized path component added to the path to your app's packaged libraries. (Although you are trying to load from /system/priv-app, so this does not apply.) It gets re-randomized each time the app is installed, and can be read like this:

String nativeLibraryPath = <yourApplicationClass>.getContext().getApplicationInfo().nativeLibraryDir;

If you were to package your own copy of libQt5Gui.so and get it from the above path String, dlopen would certainly load it. This would work even if the system sepolicy is denying access to the system copy of the library.

Links:

https://www.whitewinterwolf.com/posts/2016/08/15/examine-android-selinux-policy/

https://source.android.com/security/selinux/images/SELinux_Treble.pdf

[Horatio Chien]

Thank you for your reply.

I don't see 'denied' related message.

I am aware of the randomized path on my adb-installed APK, but there is no such path for APK under /vendor/app.

I knew the ApplicationInfo's nativeLibraryDir. It is /system/priv-app/MyQtApp/lib/arm64, which is not ideal. I expect it should be /vendor/app/MyQtApp/lib/arm64.

However, I made a symlink: /system/priv-app/MyQtApp -> /vendor/app/MyQtApp.

And I am 100% sure that /vendor/app/MyQtApp/lib/arm64/libQt5Gui.so exists.

Maybe the symbolic link just does not do the magic? Linker just does not search /vendor/app/MyQtApp/lib/arm64?

MyQtApp can be built as a normal adb-installable apk and it runs perfectly correctly. All libs are loadable.

After doing "adb install ...", I can see my libs are put in 2 different places.

1. /data/app/com.packagename.android-<random hash>/lib/arm64

2. /data/data/com.packagename.android/qt-reserved-files

Linker seems smart enough to find the lib in the 1st path, i.e. Linker is able to locate /data/app/com.packagename.android-<random hash>/lib/arm64/libQt5Gui.so.

Can someone explain how linker is able to do this? Why does linker know it should search libs from that path? Is it about LD_LIBRARY_PATH?

I wish linker (dlopen, in detail) is able to search /vendor/app/MyQtApp/lib/arm64 as well when runing MyQtApp.

Thank you.

Andrew Esh於 2018年9月18日星期二 UTC+8上午1時57分49秒寫道：

[Alex Cohn]

Your app being a "system" app, it does not look into the app-specific lib/arm64 directory to load shared libraries. It expects these libraries to be in system/lib64 or in /vendor/lib64. You can provide "security clearance" to libraries under /vendor by adding them to the /vendor/etc/public.libraries.txt file on te device, so that they will be visible to non-system apps, too.

BR,

Alex

[Horatio Chien]

Alex,

Thank you.

However, for some reasons, I cannot move the libraries.

Do you know how to make linker look into more paths?

Since linker looks into normal app's directory, it implies that there is a way to teach linker about what paths it should look into.

Alex Cohn於 2018年9月26日星期三 UTC+8上午3時49分21秒寫道：

[Alex Cohn]

My bad. I see 

Successfully loaded /system/priv-app/MyQtApp/lib/arm64/libQt5Gui.so

which means that one of two went wrong: either this libQt5Gui.so doesn't have correct SONAME, or this RPATH has caused a problem. Both cases may be treated by elfedit or similar.

BR,

Alex