Load system libraries with dlopen

[jcm]

Hi,

I would really appreciate if anybody can confirm if it's possible to
load an Android's system library through dlopen from a library build
with the ndk and, if it actually is, how should it be done.

Currently I'm trying to load Android's system's library libmedia.so by
calling dlopen("/system/lib/libmedia.so", RTLD_LAZY); (I know that the
official way supported by the ndk is by working with Open SL but,
although it's a very interesting project, I want to do some operations
that are not possible with the current available implementation for
Android). This call is done from a library (libA) that is statically
linked into an upper library (libB) with the ndk. And, this last
library, is called by a java application through a JNI.

The problem is that, this call to dlopen, always crashes dumping the
following lines:
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'google/yakju/maguro:4.0.4/IMM76I/330937:user/
release-keys'
pid: 1642, tid: 1687 >>> net.media.test:my_process <<<
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr deadbaad

r0 deadbaad r1 01250488 r2 80000000 r3 00000000
r4 00000000 r5 00000027 r6 013dc888 r7 013c4700
r8 01303274 r9 00000000 10 5b486b38 fp 00000000
ip ffffffff sp 5b6fe578 lr 4005af69 pc 400576c8 cpsr 60000030
d0 5f616964656d6a66 d1 5f67656e5f706466
d2 775f657461657265 d3 6f5f6c61636f6c72
d4 20646e756f46203a d5 436f502065657266
d6 206e6f6973736553 d7 3578307830207461
d8 0000000000000000 d9 0000000000000000
d10 0000000000000000 d11 0000000000000000
d12 0000000000000000 d13 0000000000000000
d14 0000000000000000 d15 0000000000000000
d16 41680f6010000000 d17 3fe0000000000000
d18 3feff985c46d85eb d19 3f49e8ee49e85400
d20 3c87000000000000 d21 3f1145b88564c411
d22 3e21ee94a0c6d216 d23 bda8fae9be8838d4
d24 0000000000000000 d25 0000000000000000
d26 0100010001000100 d27 0100010001000100
d28 0100010001000100 d29 0100010001000100
d30 0001000000010000 d31 0001000000010000
scr 80000012
#00 pc 000176c8 /system/lib/libc.so
#01 pc 0001e864 /system/lib/libc.so (__assert2)
code around pc:
400576a8 4623b15c 2c006824 e026d1fb b12368db \.#F$h.,..&..h#.
400576b8 21014a17 6011447a 48124798 24002527 .J.!zD.`.G.H'%.$
400576c8 f7f47005 2106ef4c efe8f7f5 460aa901 .p..L..!.......F
400576d8 f04f2006 94015380 94029303 eba4f7f5 . O..S..........
400576e8 4622a905 f7f52002 f7f4ebae 2106ef38 .."F. ......8..!
code around lr:
4005af48 41f0e92d 46804c0c 447c2600 68a56824 -..A.L.F.&|D$h.h
4005af58 e0076867 300cf9b5 dd022b00 47c04628 gh.....0.+..(F.G
4005af68 35544306 37fff117 6824d5f4 d1ee2c00 .CT5...7..$h.,..
4005af78 e8bd4630 bf0081f0 00028592 41f0e92d 0F..........-..A
4005af88 fb01b086 9004f602 461f4815 4615460c .........H.F.F.F
memory map around addr deadbaad:
befcf000-beff0000 [stack]
(no map for address)
ffff0000-ffff1000 [vectors]
stack:
5b6fe538 4008365c /system/lib/libc.so
5b6fe53c 01250488 [heap]
5b6fe540 0000013f
5b6fe544 40059d15 /system/lib/libc.so
5b6fe548 4008370c /system/lib/libc.so
5b6fe54c 4008885c
5b6fe550 00000000
5b6fe554 4005af69 /system/lib/libc.so
5b6fe558 00000000
5b6fe55c 5b6fe58c
5b6fe560 013dc888 [heap]
5b6fe564 013c4700 [heap]
5b6fe568 01303274 [heap]
5b6fe56c 4005a0d5 /system/lib/libc.so
5b6fe570 df0027ad
5b6fe574 00000000
#00 5b6fe578 4007a5b5 /system/lib/libc.so
5b6fe57c 5b490368 /data/data/net.media.test/lib/libB.so
5b6fe580 013dc888 [heap]
5b6fe584 4005a401 /system/lib/libc.so
5b6fe588 400836b0 /system/lib/libc.so
5b6fe58c fffffbdf
5b6fe590 4005e865 /system/lib/libc.so
5b6fe594 4007a5b5 /system/lib/libc.so
5b6fe598 5b490368 /data/data/net.media.test/lib/libB.so
5b6fe59c 4005e869 /system/lib/libc.so
#01 5b6fe5a0 00000061
5b6fe5a4 5b490310 /data/data/net.media.test/lib/libB.so
5b6fe5a8 5b490310 /data/data/net.media.test/lib/libB.so
5b6fe5ac 00035b80
5b6fe5b0 01303274 [heap]
5b6fe5b4 5b3ed394 /data/data/net.media.test/lib/libB.so
5b6fe5b8 013c46a8 [heap]
5b6fe5bc 013dc8ec [heap]
5b6fe5c0 5b6fe634
5b6fe5c4 00000003
5b6fe5c8 01303274 [heap]
5b6fe5cc 5b3b4ce8 /data/data/net.media.test/lib/libB.so
5b6fe5d0 5b4be7c0 /data/data/net.media.test/lib/libB.so
5b6fe5d4 000003bc
5b6fe5d8 00000000
5b6fe5dc 5b4ceea4
5b6fe5e0 5b6fe628
5b6fe5e4 00000000

I've checked the path and the library exists and also added
LOCAL_LDLIBS += -ldl into libB's Android.mk to allow using dynamic
loading. But I got always the same dump so I'm starting to doubt if
this is possible (but, initially, this is what java's MediaPlayer and
MediaRecorder classes achieve by loading a JNI that, through dlopen,
loads this lib).

Thank you in advance for any useful info that you can share.

[David Turner]

The fault address "deadbaad" is a magic code that indicates that malloc()/free() detected memory corruption in the current heap and forcibly aborted the program.

Generally speaking, libmedia.so is not exposed by the NDK, which means its implementation could be *anything* (i.e. customized by the OEM, modified between official platform releases, whatever). Trying to use it directly can very easily lead to the kind of crashes you're experimenting.

Bottom line: use OpenSLES.so instead.

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.
To post to this group, send email to andro...@googlegroups.com.
To unsubscribe from this group, send email to android-ndk...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/android-ndk?hl=en.

[Joel Centelles]

Hi David, 

thank you for the info, actually I was suspecting something like this, but, initially, this is one of the advantages of dynamic loading, that you can load libraries not available at build time. Do you mean that, then, is not possible to open system's libraries by this way as you can do in other Unix-like systems? I understand that there should be a way of doing this :-(.

I would gladly use OpenSLES, actually I think that is a very interesting and promising project but, unfortunately, as I explained, current status is not allowing me to do what I want to do, this is, decoding several incoming streams (in several different formats), working with buffers, without having to also include a software codec library. Then I thought about dynamic loading libmedia, like libmedia_jni.so should be doing.

Once again, thank you for the info, but I really would like to know if it's impossible to load limedia and other system's libraries.

BR,

Joel.

2012/5/29 David Turner <di...@android.com>

[David Turner]

On Tue, May 29, 2012 at 10:03 PM, Joel Centelles <joel.ce...@gmail.com> wrote:

Hi David, 

thank you for the info, actually I was suspecting something like this, but, initially, this is one of the advantages of dynamic loading, that you can load libraries not available at build time. Do you mean that, then, is not possible to open system's libraries by this way as you can do in other Unix-like systems? I understand that there should be a way of doing this :-(.

system libraries that are not exported by the NDK are essentially "black boxes" with magical properties and can vary drastically from one device to the other. There is absolutely no guarantee that these libraries have any kind of stable ABI, or won't even disappear in the future (that actually happens regularly, some libraries get renamed, others are split, and some are deleted when we have a better replacement).

Everytime you try to dlopen() such a library, you're just playing russian roulette. But hey, if that's your thing, who am I to prevent you from doing this :-)

On the other hand of the spectrum, the NDK-provided system libraries are *guaranteed* to have a stable ABI (more precisely, all the symbols exported by the version of the system libraries distributed with the NDK, the real versions running on devices might have additionnal symbols that may be volatile / considered private too).

Actually, if you build something with the NDK that only uses these APIs/ABIs, it would be considered a platform bug if your machine code no longer works on a future Android release.

However, these APIs are much more conservative, less powerful than being able to call anything on the platform, but this is usually for good reasons. Pick your poison.

[David Given]

Joel Centelles wrote:
[...]

> thank you for the info, actually I was suspecting something like this,
> but, initially, this is one of the advantages of dynamic loading, that
> you can load libraries not available at build time. Do you mean that,
> then, is not possible to open system's libraries by this way as you can
> do in other Unix-like systems?

The trouble is that because those libraries are not part of the public
ABI, you can't guarantee anything about them. For example, functions
might have different names from device to device, or they may use a
different calling convention to NDK code, or they may use a different
CPU architecture entirely... or they might not be present, or there
might be a completely unrelated library there with the same name!

Before the NDK days, we were building our app using the platform SDK
directly against all these internal libraries. We basically had to do
separate, individual builds for every combination of device and minor
point release of Android, in a desperate attempt to cover all possible
gotchas. To my total surprise, this actually *worked*... on the devices
we could test against. I was so glad when the NDK came along and we
could give this stuff up.

I *seriously* recommend that you don't do this.

--
┌─── ｄｇ＠ｃｏｗｌａｒｋ．ｃｏｍ ───── http://www.cowlark.com ─────
│ "Parents let children ride bicycles on the street. But parents do not
│ allow children to hear vulgar words. Therefore we can deduce that
│ cursing is more dangerous than being hit by a car." --- Scott Adams

[jcm]

Hi David,

thank you very much for the advice, I really appreciate it and I
really get your point about directly loading libmedia.so but I thought
that it was the only chance to do what I was looking for.

Just curious, I made my mind to try to solve this in a different
way :-), does this means that is impossible to open any system library
with dlopen? I understand the risks but I understand that there should
be a way to modify Android.mk to achive this. Am I wrong?

Thanks again and best regards

On 29 mayo, 22:29, David Turner <di...@android.com> wrote:
> On Tue, May 29, 2012 at 10:03 PM, Joel Centelles

> <joel.centel...@gmail.com>wrote:

[jcm]

HI David,

thank you for the advice regarding calling conventions.

Actually, I was afraid of something like this with different vendors
but I had no way to confirm it so, you're feedback based on your
experience has been very useful, thank you very much.

Seems that I would have to find I different way to do it :-D

>  signature.asc
> < 1 KBVerDescargar

[Eyal Zmora]

Joel,

As far as I know, and perhaps David could correct me if I'm wrong, libmedia.so is automatically loaded by the MediaServer process upon boot, so you don't need to explicitly load it yourself. 

It's already loaded.

Eyal

[David Turner]

On Wed, May 30, 2012 at 10:30 PM, jcm <joel.ce...@gmail.com> wrote:

Just curious, I made my mind to try to solve this in a different
way :-), does this means that is impossible to open any system library
with dlopen?

Impossible? No, you can always try dlopen("/system/lib/libmedia.so"). However, you may encounter some problems:

- the library does not *exist* on some devices

- the library might not export the *symbols* you're looking for

- even if it does export them, they might not work properly at all.

All of this is highly specific to the exact system image you're using.

 

I understand the risks but I understand that there should
be a way to modify Android.mk to achive this. Am I wrong?

This has nothing to do with Android.mk, as far as I understand.

[David Turner]

On Thu, May 31, 2012 at 8:01 AM, Eyal Zmora <eyal...@gmail.com> wrote:

Joel,

As far as I know, and perhaps David could correct me if I'm wrong, libmedia.so is automatically loaded by the MediaServer process upon boot, so you don't need to explicitly load it yourself. 

It's already loaded.

The media server runs in a different process than your application. In other words, it doesn't necessarily have the library loaded.

[Eyal Zmora]

But isn't the library mapped to all processes?

[David Given]

Eyal Zmora wrote:
> But isn't the library mapped to all processes?

Only the ones that ask for it. The library is loaded into memory once,
but applications only see it once they call dlopen(...). (Or if it's
loaded automatically by the ELF loader.) Plus, the library gets its own
set of local variables for each process that it's used by.

For all practical purposes, libraries can be thought of as being only
visible to the process that loads it.

(Incidentally, this is one of the biggest problems about running
portable code with the NDK: because each process gets a *single* set of
global variables, no matter how many Activities are in that process,
you're limited to a single instance of your application per process.
Want to run two copies of your app? You need two processes, but the
Android process model won't let you do that.)

[Eyal Zmora]

/proc/< MY PID>/smaps still shows the library load address. 

Are you saying that my process cannot use the library unless it dlopens it?

If so, and if the library is already loaded, what does dlopen actually do?

[David Given]

On 31/05/12 19:43, Eyal Zmora wrote:
> /proc/< MY PID>/smaps still shows the library load address.

In that case, it's already loaded. Probably by ld.so; dalvik may depend
on it.

> Are you saying that my process cannot use the library unless it dlopens it?

If you don't dlopen the library, you can't look up symbols on it, which
means you don't know where its functions are --- which means you can't
call it. So, basically, yes.

(This does not include libraries which are loaded as a dependency of
your executable, which are all handled for you via ld.so.)

> If so, and if the library is already loaded, what does dlopen actually do?

If it's already open, dlopen() returns you a handle for the library
which you can then call dlsym() on, to look up symbols. Libraries can
only be loaded once into each process, and are reference counted, so
they only get unloaded when everyone who had the library open dlcloses
it. (Libraries which are loaded by ld.so are never unloaded.)

--
┌─── ｄｇ＠ｃｏｗｌａｒｋ．ｃｏｍ ───── http://www.cowlark.com ─────

│ "...thus there might be a great reduction in useless expenditure on
│ Nuclear rockets, reducing inflation and stabilising the price of cat
│ foods." --- UK pat. GB1426698

[jcm]

Hi,

thank you everybody for the info.

Thanks to you advices, I definitely decided to avoid directly calling
libmedia but I'm still curious about ho how to load it. As David Tuner
states, initially, just by calling dlopen("/system/lib/libmedia.so");
should be enough but, then I get the "deadbaad" memory error because
libmedia is not exposed to the NDK, then, how can I avoid this last
error. I assumed that this should be done in Android.mk file but, if
not, how can this be fixed?

Thanks.

BR,
Joel.

On 31 mayo, 13:17, David Turner <di...@android.com> wrote:

> > > >>> this is possible (but, initially, this is what java's MediaPlayer and...
>
> leer más »

[Chris Stratton]

On Jun 3, 8:44 am, jcm <joel.centel...@gmail.com> wrote:

> Thanks to you advices, I definitely decided to avoid directly calling
> libmedia but I'm still curious about ho how to load it. As David Tuner
> states, initially, just by calling dlopen("/system/lib/libmedia.so");
> should be enough but, then I get the "deadbaad" memory error because
> libmedia is not exposed to the NDK, then, how can I avoid this last
> error. I assumed that this should be done in Android.mk file but, if
> not, how can this be fixed?

It looks like the crash is intentional:

#01 pc 0001e864 /system/lib/libc.so (__assert2)

The actually __assert2 implementation is fatal - the wrapper macro for
using it in a program as assert() ensures it should only be reached if
the asserted condition is untrue, which would be considered a fatal
error.

You could try to set up for capturing stderr (it's been discussed in
the past) and see if an assertation failure message is generated which
might be at all informative. Or you could try running in gdb with a
breakpoint there and figure out what the message would be.

As a guess, the library has initializer routines which are objecting
to the situation into which it is being loaded.

[Joel Centelles]

Thank you Chris for the hint, I didn't noticed the assert reference. I will try to debug it, once I'm able to get a free moment, and see what I find.

I was just wondering, can this be related to permissions? This is a typical issue in Linux but I'm not so sure in Android.

BR,
Joel.

2012/6/3 Chris Stratton <cs0...@gmail.com>

[Alex Cohn]

I don't think there has ever been a technical problem calling dlopen("/system/lib/libmedia.so", RTLD_LAZY). In principle, there is no guarantee the file will exist on any future Android platform, but even then the call will simply return NULL, and you can find the reason with dlerror(). Maybe your crash happens after that, when you try to bind some functions from that library?

Alex