suitable location for shared .so libs

[Scott Sibley]

Is there a formal approach to providing an ndk app a selection of dynamically loadable libraries? Permissions won't allow me to load them from the sdcard. I'm putting them under /data/local/ atm, but that's not an option on a device without root permissions right? Any hints?

[Mārtiņš Možeiko]

Afaik files in /data/local are writable and executable without root.

--
Mārtiņš Možeiko

> --
> You received this message because you are subscribed to the Google Groups
> "android-ndk" group.
> To post to this group, send email to andro...@googlegroups.com.
> To unsubscribe from this group, send email to
> android-ndk...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/android-ndk?hl=en.

[David Turner]

On Thu, Jan 26, 2012 at 2:32 AM, Mārtiņš Možeiko <martins...@gmail.com> wrote:

Afaik files in /data/local are writable and executable without root.

They are if you are the 'shell' user. I.e. with "adb shell". I don't think they're accessible from an application process. Please correct me if I'm wrong.

[Carlos A. M. dos Santos]

AFAICT you can put them anywhere under
ContextWrapper.getFilesDir().getCanonicalPath(). This will tipically
lead to

/data/data/org.example.myapp/files

You will have to load them with System.load(fullPathName) instead of
System.loadLibrary(shortName). E.g.

System.load("/data/data/org.example.myapp/files/libfoo.so");

--
"The flames are all long gone, but the pain lingers on"

[Tor Lillqvist]

Also, you will have to load them in correct order of dependency. Can be a pain...

--tml

[Martin Storsjö]

Just wanted to check with android representatives - is this an officially
supported use case (loading libraries from other directories than the ones
installed from the apk - possibly downloaded at runtime), or is it
unsupported accidental feature that might break with tighter platform
security at later updates?

// Martin

[David Turner]

It's purely accidental and there is no guarantee that this will continue to work. Actually, I wouldn't be surprised if this didn't break as a side effect of miscellaneous platform improvements I'm aware of (don't ask for details). I would advise strictly avoiding hard-coding any application data paths (e.g. /data/data/<pkgname>/...) or even "/data/local/..." in your source code to avoid this.

As for loading libraries that are located in your own application's directory, I would say it's likely to keep working for a long time, though I can't guarantee we'll never prevent it. We're generally cautious about not breaking existing apps when modifying the platform, even those that do crazy stuff we strictly tell not to do, but there are cases where security and valuable features trump any backwards compatibility concerns.

 

// Martin

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.
To post to this group, send email to andro...@googlegroups.com.

To unsubscribe from this group, send email to android-ndk+unsubscribe@googlegroups.com.

[Martin Storsjö]

On Tue, 10 Apr 2012, David Turner wrote:

> As for loading libraries that are located in your own application's
> directory, I would say it's likely to keep working for a long time,
> though I can't guarantee we'll never prevent it.�We're
> generally�cautious about not breaking existing apps when modifying the
> platform, even those that do crazy stuff we strictly tell not to do, but
> there are cases where security and valuable features trump any backwards
> compatibility concerns.

Thanks, this was the part I was interested in, and the explanation makes
sense.

// Martin

[Davis Ford]

I realize this is an old thread, but can someone confirm if this would work on Android 4.x :

com.example.nativelib.apk => contains a native .so which goes into /data/data/com.example.nativelib/lib.so

com.foo.myapp.apk => Android application, different package namespace and no sharedUserId tries to execute System.load('/data/data/com.example.nativelib/lib.so')

Without resorting to the sharedUserId solution, will System.load( ) succeed in loading an .so from an absolute path when the process has a different uid?  

If not, what is the recommended way forward for sharing an .so among several apk's without modifying the ROM to put libs under the read only /system/lib partition? 

Thanks in advance,

Davis

// Martin

To unsubscribe from this group, send email to android-ndk...@googlegroups.com.

[Latimerius]

I don't know the answer to your question, I'd just like to point out that access to files located in paid APKs is complicated since the (re)introduction of app encryption (in 4.2 I think).

Hardcoding an absolute path to your .so has been pointed out many times as something that's not guaranteed to work.  Interestingly enough, I haven't seen yet any hint as to how to do it right.  It looks like we're given means to load shared object in runtime (dlopen() & co.) but no supported way to specify the library to be loaded.

To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk...@googlegroups.com.

To post to this group, send email to andro...@googlegroups.com.

Visit this group at http://groups.google.com/group/android-ndk.
For more options, visit https://groups.google.com/groups/opt_out.

[Davis Ford]

I have a specialized use case whereby we are targeting custom hardware -- not the handset masses, app encryption won't be an issue.  I'm only interested in setting up a single shared object that can be loaded by multiple apk's on this system.  I'm trying to figure out if I can do this in a simple manner without modifying the ROM, but I can modify the ROM if necessary.

The only mechanisms I see to achieve this are:

a) Put the .so under /data/local/tmp this is r/w directory accessible to all uid's it seems, but I'm not sure if the AOSP cleans this out on some schedule, and the fact that it is writeable poses a concern since apps could delete or modify the .so

b) Put the app in a known static apk's data folder /data/data/com.static.namespace and other apk's can do System.load("/data/data/com.static.namespace/lib/mylib.so") -- I guess I'll have to build a quick test to see if this works or if the OS permissions deny it.  If it does work, this seems like a reasonable alternative since it will not require custom ROM changes, and in the event that an upstream platform change breaks this, I guess we could go into the source and make it work again.

c) Modify the ROM to put the .so into /system/lib or add a new folder that is on the LD_LIBRARY_PATH; the problem with this is that we want to update the .so via the standard mechanism of updating an .apk, and standard .apk's won't have write permissions to /system/lib so some customization is necessary there as well.

d) Use manifest's sharedUserId concept; this is also problematic, as we don't want to force all apps that want to use a single shared object file to have to use the same uid

I'm not sure if there is another alternative, but in my short research that's all I've found.  I'd love to hear if there are any other alternatives I haven't stumbled across.

--
You received this message because you are subscribed to a topic in the Google Groups "android-ndk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/android-ndk/u_JadDj9uBA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to android-ndk...@googlegroups.com.

[Latimerius]

Have you considered copying the .so over from its package and putting it somewhere inside the directory structure of the app that wants to use it?

Moving files between packages is documented and guaranteed to work - you'd just need a ContentProvider in the APK containing the .so .

Writing a file inside app's private area is also a supported operation.  With the .so in a known location within the app's subtree, loading it probably wouldn't need to rely on an absolute path (this would need verification though, I seem to remember having trouble dlopen()ing a .so within the same APK and IIRC I had use an absolute path in the end anyway :-( ).

Obviously, user apps would need a bit of code to check whether copying is necessary (either if the app doesn't have the .so yet, or if the app's copy is outdated).  This will depend on how much control you have over the user apps.

At any rate, the above mechanism would have the advantage of not relying on any undocumented behaviour in any step.

[David Turner]

On Tue, Sep 17, 2013 at 11:01 PM, Davis Ford <davi...@gmail.com> wrote:

I realize this is an old thread, but can someone confirm if this would work on Android 4.x :

com.example.nativelib.apk => contains a native .so which goes into /data/data/com.example.nativelib/lib.so

Technically, the real installation location of the shared libraries changed a lot between different Android revisions, but the installer currently maintains a symbolic link from /data/data/<packagename>/lib that points to the actual location.

However, whether this link can be accessed or followed by any UID is absolutely not guaranteed at all. This is intentional since this may conflict with other system-level features (i.e. "protected" apps, guest mode, multi-user, whatever comes up in the future).

It's not difficult to imagine that the link itself would disappear in a future version of the system (with some VM / linker tricks to keep things working inside applications who want to access their own shared libraries).

So the fact that this may currently work for other UIDs is _purely_ accidental and should not be relied on.

And this is by design, to prevent DLL, versioning and security issues, each Android application should be its own "bundle of everything" and not rely on something external. The sharedUserId feature is here to allow application-specific plugins to be installed along-side the main one, and that's it.

 

com.foo.myapp.apk => Android application, different package namespace and no sharedUserId tries to execute System.load('/data/data/com.example.nativelib/lib.so')

Without resorting to the sharedUserId solution, will System.load( ) succeed in loading an .so from an absolute path when the process has a different uid?  

If not, what is the recommended way forward for sharing an .so among several apk's without modifying the ROM to put libs under the read only /system/lib partition? 

If you really want to do that, you will have to copy the shared library file from your "main location" to your APKs' files/ directory, where it can be loaded directly from.

Just to clarify: Anything else is a hack that probably goes against the platform's overall design.

 

Of course, the easy way out is to modify the platform image, to do what you need, since it's a custom project :-)

Hope this helps

To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk...@googlegroups.com.

To post to this group, send email to andro...@googlegroups.com.

[Davis Ford]

Hi David,

Thanks for clarifying these points.  What you have said makes sense.  I have enough to move forward with, and understand the risks associated with doing it that way.  

Best regards,

Davis

--
You received this message because you are subscribed to a topic in the Google Groups "android-ndk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/android-ndk/u_JadDj9uBA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to android-ndk...@googlegroups.com.

[Davis Ford]

Just one last point I wanted to make: something for the Android team to consider...I wanted to explain *why* I want this functionality, so you might consider it for the future: 

What I have is an application that is mostly native code delivered as a shared object.  It is a very large code base, and it won't be re-written in Java -- probably ever.  But it has been ported to Android/Arm and it only requires a small amount of Java glue to use it.

The binary .so is large ~20MB.  Packaging in each individual apk that wants to use it has several drawbacks:

a) The size -- distributing over the network, requiring people to package it, etc.

b) The versioning / management of versions -- if we want to update the native code, we have to update every apk that uses it -- this is a bit of a nightmare

The alternative approach is that I can encapsulate the .so in its own apk that can be versioned separately.  It can act as a ContentProvider to apk's that require it -- similar perhaps to how OSGi bundles are resolved.  An apk requests a version of the native library and the ContentProvider resolves it for them provided the version requested matches.  This is far more desirable than having each apk that depends on this package the binary inside itself.

The ability to System.load( ) it from an absolute path is also really nice (across uid's), since the startup latency of copying the large shared object to another folder and then loading then library is also a bummer -- then we also have to deal with cached versions of this file all over the filesystem and make sure when an update to the library occurs, we again copy over -- as opposed to just opened a file path.  Anyway, it makes it significantly easier to manage, less complex, and less prone to error.

Ideally, what I'd like to see is a single place...something like /data/lib/[package]/ where apks can store native libraries, and provide read-only access to other apks 

An analogy might be Google Play Services.  What I'm trying to do is similar to Google Play Services, by providing a component that can be versioned on its own, that other applications can load and use.  It's just that my component happens to be a native shared object.

Hope that makes sense -- just trying to explain my use case a little better.  I'm guessing there are others that have similar needs.

Regards,

Davis

[AppCoder]

On Thursday, September 19, 2013 9:40:36 AM UTC-4, Davis Ford wrote:

The alternative approach is that I can encapsulate the .so in its own apk that can be versioned separately.  It can act as a ContentProvider to apk's that require it -- similar perhaps to how OSGi bundles are resolved.  An apk requests a version of the native library and the ContentProvider resolves it for them provided the version requested matches.  This is far more desirable than having each apk that depends on this package the binary inside itself.

Is there some reason (possibly that it won't support multi-session or something that
you would have to work around with forks or something for each client) that you can't
just let the shared library load in it's own apk and expose it's interface (via a
ContentProvider/Service whatever) to the other applications? 

[are...@farmerstel.com]

Greetings:

I've been prowling around through the posts on some of these groups and since this thread seems to hit pretty close to what I'm concerned about I decided to try my luck here:  A few posts prior to this reply Davis Ford said:

a) Put the .so under /data/local/tmp this is r/w directory accessible to all uid's it seems, but I'm not sure if the AOSP cleans this out on some schedule, and the fact that it is writeable poses a concern since apps could delete or modify the .so

b) Put the app in a known static apk's data folder /data/data/com.static.

namespace and other apk's can do System.load("/data/data/com.static.namespace/lib/mylib.so") -- I guess I'll have to build a quick test to see if this works or if the OS permissions deny it. 

My problem is that I have a native apk which runs just fine on the emulator. it does a System.load("/data/mysolibrary.so") which works.  when i run that very same apk on my usb connected tablet it crashes and in the debugger it says it cannot find the .so.
  To me it looks like on the emulator that my shared library gets put into /data but it does not on the hardware.  Referring now back to Davis Ford's comment: how do I get the build process to put that .so into /data when it's run on an actual tablet/phone? I assume
I can root the device and manually put a copy of the .so on my tablet and viola it won't crash and I could continue development and figure out how to fix this later.  adb won't let me write directly to /data though. but I based on reading the above quote I _could_ push to /data/local or /data/data/static.namespace/lib. In the latter action I think my emulator version would break because that .so files does get put into /data apparently.  on another thread a suggested solution was to set "LOCAL_REQUIRED_MODULES := mysolibrary.so" in Android.mk. tried that. it doesn't help. i also already had "LOCAL_MODULE := mysolibrary.so".  How do you tell the ndk-build where to put .so files and what is a good way of getting your source code that reference to that filesystem location?

rwolf

[are...@farmerstel.com]

More prowling led me to find this comment by Felix Homann while talking about "Building shared library that calls other shared libraries with a standalone toolchain":

"Another approach is to bypass the automatic deployment of .so files. Put them in the assets folder instead. In your app copy the assets to a folder of your choice so you can System.load() them. This way you can even load versioned libraries."

 

Maybe that's the answer or most of it. Still don't understand why running on emulator works and on real hardware doesn't.  I suppose this reduces the problem to a Java problem and using android.io package, java.io.File, isFile to check that the .so

hasn't been copied out of assets already(for efficiency). Use read, write to actually move the data. i'm only a few weeks into working with android.

[Davis Ford]

My goal was to avoid the need to copy the file.  So, let's say I have two apk's: apk1 and apk2.

apk1 contains a large native .so file inside it's /libs/armeabi-v7a/ folder -- when installed, it is copied to the app's /data/data/[package-name]/lib/ folder

apk2 depends on apk1 and wants to load the .so

One solution is to to copy the file from the apk1's data folder over to apk2's folder, but I want to avoid this b/c of the startup latency it may incur.  I also don't want to have multiple copies of this .so all over the file system -- it kind of defeats the purpose of an .so in the first place, is a waste of disk space, and becomes an unnecessary maintenance headache dealing with all these different files.  When I update the .so -- I have to check the local copied version against the latest source and deal with it accordingly.  This is not ideal nor wanted.

What I want is just a central folder like /system/lib, let's call it /usr/lib mounted as rw where apps can write shared objects, and other apps can load them.  The /data/data/[package-name]/lib/ folder happens to work with System.load() across apk's, although -- as stated above, this is an anomaly, and may change in a future release.  For now, it is good enough for me to move forward with, and if it ends up being removed in a future release, I aim to either modify the ROM to restore the capability, or move to another solution.

There are various other alternative solutions -- such as sharing context across apps, using a ContentProvider, sharing process and user id, etc.  These are all less than ideal for my situation though.

--

[Randall Wolf]

Doing the System.load from /data/data/[package-name]/lib/mylib.so did the trick. I can get on with development and work on answering other go-this-way/go-that-way questions. I can defer efficiency

and use of assets directory until later.

 

I figured out why I was getting different results from emulator to hardware.  My fault.  My hardware is a nexus 10 and the emulator instance I was using was a nexus 7. apparently

 a System.load from /data/mylib.so works ok with a nexus 7 instance but fails on nexus 10 instance and on real nexus 10 hardware.  the reason this happened is because during my

early experimentation I never saw any difference from when I tested on a nexus 7  instance as opposed to a nexus 10 instance. I drifted to using the 7 instance because I could see a

bit more of my normal screen stuff (I shifted for the same reason someone might use an actual 7 hardware instead of a 10 hardware). Then...I mostly forgot there was a difference in the

2 'places' I was testing.  this is my 1st encounter with an actual difference in behavior. this is something that would be good to remember. I am temporarily a happy camper.

[Alex Cohn]

On Thursday, September 19, 2013 4:01:05 PM UTC+3, Digit wrote:

On Tue, Sep 17, 2013 at 11:01 PM, Davis Ford <davi...@gmail.com> wrote:

I realize this is an old thread, but can someone confirm if this would work on Android 4.x :

com.example.nativelib.apk => contains a native .so which goes into /data/data/com.example.nativelib/lib.so

Technically, the real installation location of the shared libraries changed a lot between different Android revisions, but the installer currently maintains a symbolic link from /data/data/<packagename>/lib that points to the actual location.

However, whether this link can be accessed or followed by any UID is absolutely not guaranteed at all. This is intentional since this may conflict with other system-level features (i.e. "protected" apps, guest mode, multi-user, whatever comes up in the future).

It's not difficult to imagine that the link itself would disappear in a future version of the system (with some VM / linker tricks to keep things working inside applications who want to access their own shared libraries).

Note that this path can be retrieved with http://developer.android.com/reference/android/content/pm/ApplicationInfo.html#nativeLibraryDir API (added in Gingerbread). This only makes sense if you want to use this path from another application, which cannot rely on System.load(libName). Actually, I have never seen devices that did not allow System.loadLibrary("/sdcard/lib" + libName + ".so"), but after all, I never performed formal research of this feature.

So the fact that this may currently work for other UIDs is _purely_ accidental and should not be relied on.

I believe that there are quite a few respectable initiatives that do rely on this fact to allow separate distribution of the common engine. It is actually THE correct way to distribute applications in compliance with LGPL native components.

BR,

Alex