shared library too large from /system/lib, way to link in /data ?

[George Nychis]

I have a shared library that is 80MB in size, which is well beyond the
free space that my phone (or some other user's phone) is likely to
have.

I see that when I install the application, the library is installed
to: /data/data/myapp/lib and the entire shared library is installed
there (all 80MB).

However, whenever I try to link to it in my Android application with
System.loadLibrary("blah"), it appears that Android tries to install
the library to /system/lib and then continue execution. I then get a
SIGBUS error and my application crashes since it does not fit in
there.

Is there a way that I can instead load the library directly from /data/
data/myapp/lib instead?

[Doug Kwan (關振德)]

You need to set LD_LIBRARY_PATH

> --
> You received this message because you are subscribed to the Google Groups "android-ndk" group.
> To post to this group, send email to andro...@googlegroups.com.
> To unsubscribe from this group, send email to android-ndk...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/android-ndk?hl=en.
>
>

[Dianne Hackborn]

Your shared library code is extracted and placed in {data}/lib at install time.  If there isn't room for it, then the install will file.  It being too large won't cause a SIGBUS.

Android would never try to install it in /system/lib/.  Everything under /system is read-only.  (Unless you are talking about creating your own system image, in which case you generally want to set up the build system to put the library in /system/lib at build time...  though the vast majority of devices I have worked on haven't had near enough space on /system for an extra 80MB shared library).

On Fri, Apr 22, 2011 at 12:06 AM, George Nychis <gny...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.
To post to this group, send email to andro...@googlegroups.com.
To unsubscribe from this group, send email to android-ndk...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/android-ndk?hl=en.

--
Dianne Hackborn
Android framework engineer
hac...@android.com

Note: please don't send private questions to me, as I don't have time to provide private support, and so won't reply to such e-mails.  All such questions should be posted on public forums, where I and others can see and answer them.

[George Nychis]

Hi Dianne,

Thanks for your response.  "/system is read-only" jogged my memory that some long time ago, I must have written a shell command to remount and copy the libs to /system/lib.  And this was true... so this is my fault.  I was doing it to be able to run some standalone applications in a separate shell which would look for them in /system/lib.  Thanks for your explanation.

- George

[Stephen Williams]

Is there any more or less standard way to affect how this works or to do an alternative?

What about putting a shared library in assets and constructing a path to if for System.loadLibrary()?  Or otherwise loading it into sdcard space, perhaps by app download, and loading it from there?  (I believe at 2.2 or so you can load libraries from the sdcard, right?)

Additionally, I haven't yet tracked down how to allow an app to be moved to the sdcard by the user.  Where is that documented?  Ahh, found it:

http://developer.android.com/guide/appendix/install-location.html

But:

> The .apk file is saved on the external storage, but all private user data, databases, optimized .dex files, and extracted native code are saved on the internal device memory.

That's mostly good, except the native code part.  I notice that assets are not mentioned, so presumably they're only in the .apk in the encrypted container on the sdcard.  That's good.

I don't see anything on that page that indicates that the app can write anything to that "encrypted container".  That would be convenient.  Is it a read-only filesystem image, or is it modifiable by the application in some way?

In any case, I still have the question above: Is it possible for the application to extract a .so from assets, write it to the sdcard, and load it?

Closely related to that would be a question about best practices on unloading the shared library when needed.  It appears that the only way to do that would be to use another shared library to make a dlunload() call on the first, which seems feasible.  Care would have to be taken to get the dvm to forget about the library.  What work would be involved?

Stephen

--
Stephen D. Williams s...@lig.net scie...@gmail.com LinkedIn: http://sdw.st/in
V:650-450-UNIX (8649) V:866.SDW.UNIX V:703.371.9362 F:703.995.0407
AIM:sdw Skype:StephenDWilliams Resume: http://sdw.st/gres
Personal: sdw.st facebook.com/sdwlig twitter.com/scienteer

[Dianne Hackborn]

Starting with 2.3 native shared libraries are also placed on the SD card.

The container is not writeable.  It is a filesystem, but only in the most trivial sense.  It is created, the files are placed in it, and then it is truncated to the final size and becomes read only.  It is not feasible to have it dynamically size, since that means the filesystem inside has to grow and shrink as it is being used.

There should be no need to unload a shared library.  Why would you want to?

[Stephen Williams]

(I have the wrong default on my Gmail identities and Gmail isn't smart enough to automatically associate identities with lists/threads... Feature request!)
Nice, having the shared libraries on the SD card is excellent.  That and putting the app on the SD card to begin with solves major problems.

> There should be no need to unload a shared library.  Why would you want to?

Loaded from the SDcard that was just removed...

There are other potential reasons:

• Swapping large modules to do different functions (although probably a non-issue on Android with Linux demand paging from readonly native code sources)
• To swap to a different version of a library (testing, upgrade, option of some kind)
• Security schemes (exotic maybe, but potentially very secure against various attacks)
• Upgrades on the fly (probably not a preferred Android use case, but interesting if possible; see security schemes)
  

Can 2.2 load a shared library from the sdcard if given an explicit path?  If so, for just 2.2, could have code that simulates the 2.3 library placement.

Thanks,
Stephen

[Dianne Hackborn]

On Fri, Apr 22, 2011 at 3:43 PM, Stephen Williams <stephend...@gmail.com> wrote:

Loaded from the SDcard that was just removed...

If your app is on the SD card, the app will need to be killed.

 

• Swapping large modules to do different functions (although probably a non-issue on Android with Linux demand paging from readonly native code sources)

Yes, demand paging should take care of this. 

• To swap to a different version of a library (testing, upgrade, option of some kind)

You could just let your process be killed in the background and restarted. 

• Security schemes (exotic maybe, but potentially very secure against various attacks)

If security is a concern, you could also run these different schemes in different processes.  

Can 2.2 load a shared library from the sdcard if given an explicit path?  If so, for just 2.2, could have code that simulates the 2.3 library placement.

No, the sdcard is noexec.  You really don't want to be loading code off of it since basically any other app can modify its contents.

[Stephen Williams]

On Fri, Apr 22, 2011 at 4:05 PM, Dianne Hackborn <hac...@android.com> wrote:

On Fri, Apr 22, 2011 at 3:43 PM, Stephen Williams <stephend...@gmail.com> wrote:

Loaded from the SDcard that was just removed...

If your app is on the SD card, the app will need to be killed.

 

• To swap to a different version of a library (testing, upgrade, option of some kind)

You could just let your process be killed in the background and restarted. 

• Security schemes (exotic maybe, but potentially very secure against various attacks)

If security is a concern, you could also run these different schemes in different processes.  

And then have to prove you're talking to the right process, but yea...
 

Can 2.2 load a shared library from the sdcard if given an explicit path?  If so, for just 2.2, could have code that simulates the 2.3 library placement.

No, the sdcard is noexec.  You really don't want to be loading code off of it since basically any other app can modify its contents.

So in 2.3+ the shared libraries in an app installed on the sdcard are in the encrypted read-only apk/apk-like device-specific loopback-mounted application-specific filesystem, and that does not have noexec set in the mount options?  Cool.

Is the secure access to that simply the app userid or more aggressive kernel-level app association?  Are group permissions usable between multiple apps for access to resources, assets, and/or shared libraries?  Presumably multiple processes from the same app would share that access.  Is there a way to pass access between apps?

For something like the Qt library installer vs. applications, you want the big shared libraries in a separate app from the actual apps, shared securely.  I know they support passing the shared library paths from the (large, shared) "library app" to the actual (small) apps, but haven't determined if that scheme works for installing the library app on the sdcard.
 

--
Dianne Hackborn
Android framework engineer
hac...@android.com

Thanks,
Stephen

[Dianne Hackborn]

On Fri, Apr 22, 2011 at 4:23 PM, Stephen Williams <stephend...@gmail.com> wrote:

So in 2.3+ the shared libraries in an app installed on the sdcard are in the encrypted read-only apk/apk-like device-specific loopback-mounted application-specific filesystem, and that does not have noexec set in the mount options?  Cool.

Yep.

 

Is the secure access to that simply the app userid or more aggressive kernel-level app association?  Are group permissions usable between multiple apps for access to resources, assets, and/or shared libraries?  Presumably multiple processes from the same app would share that access.  Is there a way to pass access between apps?

It's the user id, as is all security between applications.  I'm not sure what you mean by more aggressive kernel-level app association?  Generally Linux's security between different users is pretty good.

 

For something like the Qt library installer vs. applications, you want the big shared libraries in a separate app from the actual apps, shared securely.  I know they support passing the shared library paths from the (large, shared) "library app" to the actual (small) apps, but haven't determined if that scheme works for installing the library app on the sdcard.

We don't support third party shared libraries at this point; any attempt to implement something like that will be some degree of a hack.

--
Dianne Hackborn
Android framework engineer
hac...@android.com