Implementing native code plugins as apk

[Tero Saarni]

I have an application that supports plugins implemented in native code. I would like to package them into individual apk's so that user can download them from Android Market.

The native part of the main application loads plugins with dlopen().  I can use PackageManager to find out installed plugins but how could I find the full path for the plugin .so's (or the .apk's containing the libraries)?

[Olivier Guilyardi]

Hello Tero,

to directly access the files of another app, you may use sharedUserId:
http://developer.android.com/intl/fr/guide/topics/manifest/manifest-element.html#uid

If you can't use sharedUserId (different apk signatures etc..), you may also
retrieve and make a local copy of the .so by using a content provider and
ContentResolver.openInputStream().

That said, there are several security questions if you load and run an external
executable. Please read the recent "Native add-ons" thread on this list.

Olivier

> --
> You received this message because you are subscribed to the Google
> Groups "android-ndk" group.
> To post to this group, send email to andro...@googlegroups.com.
> To unsubscribe from this group, send email to
> android-ndk...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/android-ndk?hl=en.

[Angus Lees]

On Wed, Dec 29, 2010 at 06:32, Olivier Guilyardi <li...@samalyse.com> wrote:
> to directly access the files of another app, you may use sharedUserId:
> http://developer.android.com/intl/fr/guide/topics/manifest/manifest-element.html#uid

sharedUserId is the best way to go for this (afaics).

Be warned that there are numerous serious bugs migrating to
sharedUserId if you are starting from an existing non-sharedUserId app
- depending on Android version it may be silently impossible, render
your existing files inaccessible, or crash the phone.

For ScummVM, I was unfortunate enough to start from a non-sharedUserId
app so I have to unpack .so's from other apks "manually". That was
necessary to work around several package installer bugs on older
Android releases anyway, so it wasn't all bad.

- Gus

[Dianne Hackborn]

On Tue, Dec 28, 2010 at 5:00 PM, Angus Lees <gus...@gmail.com> wrote:

Be warned that there are numerous serious bugs migrating to

sharedUserId if you are starting from an existing non-sharedUserId app
- depending on Android version it may be silently impossible, render
your existing files inaccessible, or crash the phone.

You actually just can't change between a non-shared to a shared user ID, and never have been able to.  Early on the failure was silent, but it has become less so.  The problem is -- your uid changes, so your complete identity changes, including the access to all your files, and anything else that another app is keeping track of about your app.  There has never been a way to handle this kind of change without completely un-installing the old app and re-installing the new one, and I'm not sure there ever will be.

--
Dianne Hackborn
Android framework engineer
hac...@android.com

Note: please don't send private questions to me, as I don't have time to provide private support, and so won't reply to such e-mails.  All such questions should be posted on public forums, where I and others can see and answer them.

[Olivier Guilyardi]

On 12/29/2010 08:59 PM, Dianne Hackborn wrote:
> On Tue, Dec 28, 2010 at 5:00 PM, Angus Lees <gus...@gmail.com
> <mailto:gus...@gmail.com>> wrote:
>
> Be warned that there are numerous serious bugs migrating to
> sharedUserId if you are starting from an existing non-sharedUserId app
> - depending on Android version it may be silently impossible, render
> your existing files inaccessible, or crash the phone.
>
>
> You actually just can't change between a non-shared to a shared user ID,
> and never have been able to. Early on the failure was silent, but it
> has become less so. The problem is -- your uid changes, so your
> complete identity changes, including the access to all your files, and
> anything else that another app is keeping track of about your app.
> There has never been a way to handle this kind of change without
> completely un-installing the old app and re-installing the new one, and
> I'm not sure there ever will be.

So if:
- app A has been distributed for a long time and never used sharedUserId
- and app B is a brand new app (never deployed) which contains the plugins

Then what value to give to sharedUserId in those apps so that app A doesn't
require a reinstall?

--
Olivier

[Dianne Hackborn]

On Wed, Dec 29, 2010 at 12:28 PM, Olivier Guilyardi <li...@samalyse.com> wrote:

So if:

- app A has been distributed for a long time and never used sharedUserId
- and app B is a brand new app (never deployed) which contains the plugins
Then what value to give to sharedUserId in those apps so that app A doesn't
require a reinstall?

You can't.  You can't change the uid of an app without requiring a full uninstall/re-install.

That said, an installed .apk is world-readable, so just getting stuff out of it doesn't require shared user IDs.

Also as far as I know ever since the platform has extracted .so libraries out of an .apk, this also have been world readable, so if you want to rely on it doing the extraction there still should be no need for shared user IDs.

To make this as secure as using shared user IDs without doing so, your app should get the signature/certificate of the plug-in .apk and make sure it is whatever cert you expect (which can be different than the one of your base app) before loading any code from it.  That is the same limit that shared user IDs imposes.  The only thing you lose is that the additional .apks can't add their own required permissions in to the uid you run under.  (I would honestly consider this a feature, the way permissions need to be handled with shared user IDs makes me wish we never made this facility available.)

[Dianne Hackborn]

Oh also do consider using a service-based approach instead of plug-ins.  This has so many benefits -- isolation of code, clear separation of permissions, protection from crashes, etc.  It does require more up-front work, but you will most likely be much happier longer term.

[Olivier Guilyardi]

On 12/29/2010 11:09 PM, Dianne Hackborn wrote:
> On Wed, Dec 29, 2010 at 12:28 PM, Olivier Guilyardi <li...@samalyse.com
> <mailto:li...@samalyse.com>> wrote:
>
> So if:
> - app A has been distributed for a long time and never used sharedUserId
> - and app B is a brand new app (never deployed) which contains the
> plugins
> Then what value to give to sharedUserId in those apps so that app A
> doesn't
> require a reinstall?
>
>
> You can't. You can't change the uid of an app without requiring a full
> uninstall/re-install.

Let me rephrase my question: what is the default user id of an application? Is
it predictable?

If yes, then a solution could be to set sharedUserId in app B so that it matches
the default user id of app A. Please remember that in my example, app B is a
brand new app which has not been deployed yet.

> That said, an installed .apk is world-readable, so just getting stuff
> out of it doesn't require shared user IDs.
>
> Also as far as I know ever since the platform has extracted .so
> libraries out of an .apk, this also have been world readable, so if you
> want to rely on it doing the extraction there still should be no need
> for shared user IDs.

Ok, world readable... It's quite a vulnerability. You already tried to address
it with (obsolete) forward locks. So, how to be sure we can rely on it in the
future? Won't you change your mind?

--
Olivier

[Dianne Hackborn]

On Wed, Dec 29, 2010 at 2:36 PM, Olivier Guilyardi <li...@samalyse.com> wrote:

Let me rephrase my question: what is the default user id of an application? Is

it predictable?

There isn't a default user ID.  A user ID gets allocated and assigned to the application when it is installed.

 

If yes, then a solution could be to set sharedUserId in app B so that it matches
the default user id of app A. Please remember that in my example, app B is a
brand new app which has not been deployed yet.

If you are talking about user ID as the "shared user ID" *name* that you give, that is just a name for shared user IDs.  Applications that don't specify a shared user ID...  well, don't have a shared user ID.

 

Ok, world readable... It's quite a vulnerability. You already tried to address

it with (obsolete) forward locks. So, how to be sure we can rely on it in the
future? Won't you change your mind?

I'm not sure what you mean.  World readable is not a vulnerability; it is just readable by everyone.  Nothing can modify it.  Are you talking about trying to prevent people from copying an app by not letting them get access to its .apk?  If so, that is clearly a futile effort, which is why forward locking is going away and being replaced with the licensing service.

[Angus Lees]

On Thu, Dec 30, 2010 at 06:59, Dianne Hackborn <hac...@android.com> wrote:
> On Tue, Dec 28, 2010 at 5:00 PM, Angus Lees <gus...@gmail.com> wrote:
>>
>> Be warned that there are numerous serious bugs migrating to
>> sharedUserId if you are starting from an existing non-sharedUserId app
>> - depending on Android version it may be silently impossible, render
>> your existing files inaccessible, or crash the phone.
>
> You actually just can't change between a non-shared to a shared user ID, and
> never have been able to.  Early on the failure was silent, but it has become
> less so.  The problem is -- your uid changes, so your complete identity
> changes, including the access to all your files, and anything else that
> another app is keeping track of about your app.  There has never been a way
> to handle this kind of change without completely un-installing the old app
> and re-installing the new one, and I'm not sure there ever will be.

The fix is absolutely trivial: give the sharedUserId attribute a
well-known default value (I'd suggest the Android package name). This
lets future manifest changes refer to that value explicitly to achieve
the desired "give me the user id that I know was already allocated for
my other package" behaviour.

Fwiw, I tried to describe this solution on Android bug 4381 but there
didn't seem to be any way to reopen a bug that was closed prematurely
and I suspect the comment was never seen by anyone who mattered.
Where would be the best place for me to file such a bug / feature
request?

- Gus

[Olivier Guilyardi]

Ok, thank you, then all is well, and one can simply locate the desired apk using
PackageManager, verify its authenticity, and extract the needed files. That's
fairly simple.

One last question. If the plugins are JNI libs, they get extracted in
<dataDir>/lib. Can another app (with a different UID) access these files
directly? I heard people saying that's possible. Can we rely on that too?

--
Olivier

[Dianne Hackborn]

On Wed, Dec 29, 2010 at 3:55 PM, Angus Lees <gus...@gmail.com> wrote:

The fix is absolutely trivial:  give the sharedUserId attribute a

well-known default value (I'd suggest the Android package name).  This
lets future manifest changes refer to that value explicitly to achieve
the desired "give me the user id that I know was already allocated for
my other package" behaviour.

Actually apps with shared user IDs are quite different than those that aren't.  Their permissions are managed very differently, as well as many other things inside of the package manager.  In fact, as I said, the way that permissions need to be managed for shared user IDs is very unfortunate, and I absolutely am not going to have that taint all applications.

Honestly, I strongly discourage people from using shared user IDs.  They are really an exception to the Android sandbox model, and thus create a number of strange behavior and situations.  There *are* some situations where they are useful, really the only reason they were created: to allow multiple well-defined .apks to run their components in the same process.  Note this is *not* the same as plug-ins; it is only a memory optimization. As I said, you will generally be much happier if you implement plug-ins as services so they can be isolated from the main application.

 

Fwiw, I tried to describe this solution on Android bug 4381 but there
didn't seem to be any way to reopen a bug that was closed prematurely
and I suspect the comment was never seen by anyone who mattered.
Where would be the best place for me to file such a bug / feature
request?

I closed the bug because I have no intention to change this.

[Dianne Hackborn]

On Wed, Dec 29, 2010 at 4:11 PM, Olivier Guilyardi <li...@samalyse.com> wrote:

Ok, thank you, then all is well, and one can simply locate the desired apk using

PackageManager, verify its authenticity, and extract the needed files. That's
fairly simple.

Yes.  You may not even need to extract the files -- if this is for native code, you can find the already extracted libraries in persistent storage and world-readable.  Prior to 2.3 you would need to infer the location through the base path of the application's data + "/lib" appended to it.  As of 2.3 there is a new API and you should always start using it if available to find the libs, since in 2.3 they will be on the SD card if the .apk is installed on the SD card: http://developer.android.com/reference/android/content/pm/ApplicationInfo.html#nativeLibraryDir

Note that the directory itself is I believe not world executable, so you can ls it.  You should have a well-defined name you try to open in it, or have the add-on use a <meta-data> in its application or a component tag that tells you the name to look for.  (See NativeActivity for an example.)

One last question. If the plugins are JNI libs, they get extracted in
<dataDir>/lib. Can another app (with a different UID) access these files
directly? I heard people saying that's possible. Can we rely on that too?

Yes, these are world-readable.  There is no reason not to make them world-readable, since the installed .apk must be world-readable (we'll assume the wretched forward locking is gone) and it contains the .so anyway.

[Angus Lees]

On Thu, Dec 30, 2010 at 11:25, Dianne Hackborn <hac...@android.com> wrote:
> On Wed, Dec 29, 2010 at 3:55 PM, Angus Lees <gus...@gmail.com> wrote:
>>
>> The fix is absolutely trivial:  give the sharedUserId attribute a
>> well-known default value (I'd suggest the Android package name).  This
>> lets future manifest changes refer to that value explicitly to achieve
>> the desired "give me the user id that I know was already allocated for
>> my other package" behaviour.
>
> Actually apps with shared user IDs are quite different than those that
> aren't.  Their permissions are managed very differently, as well as many
> other things inside of the package manager.  In fact, as I said, the way
> that permissions need to be managed for shared user IDs is very unfortunate,
> and I absolutely am not going to have that taint all applications.

Oh, I had assumed the only difference was how the uid was allocated
(and obvious implications on file ownership/access). Yes, differing
permissions makes things murkier indeed.

> Honestly, I strongly discourage people from using shared user IDs.  They are
> really an exception to the Android sandbox model, and thus create a number
> of strange behavior and situations.  There *are* some situations where they
> are useful, really the only reason they were created: to allow multiple
> well-defined .apks to run their components in the same process.  Note this
> is *not* the same as plug-ins; it is only a memory optimization. As I said,
> you will generally be much happier if you implement plug-ins as services so
> they can be isolated from the main application.

Also for anyone wanting easy cross-platform code. Dynamically loaded
modules are a fairly universal feature, whereas services are uniquely
Android.

>> Fwiw, I tried to describe this solution on Android bug 4381 but there
>> didn't seem to be any way to reopen a bug that was closed prematurely
>> and I suspect the comment was never seen by anyone who mattered.
>> Where would be the best place for me to file such a bug / feature
>> request?
>
> I closed the bug because I have no intention to change this.

That's unfortunate, but not unexpected. I will try to avoid
sharedUserIds in future apps.

Olivier: you may be interested in the plugin unpacker I have in
ScummVM. You can get the code here (or via svn):
http://scummvm.svn.sourceforge.net/viewvc/scummvm/scummvm/trunk/backends/platform/android/org/inodes/gus/scummvm/Unpacker.java

The current approach is to use an OrderedBroadcast protected by a
signature-protected permission to find the list of plugin packages.
It turns out that signature-protected permissions are busted if the
packages get installed in the wrong order (Android bug 5521), so I
intend to switch to verifying the signatures myself. The rest
(finding packages, unpacking libraries) should still be re-usable
as-is though.

It's under GPL (same as the rest of ScummVM), so you are encouraged to
use/steal non-trivial pieces of it provided you make your source
available too (paraphrasing).

- Gus

[joebowbeer]

Dianne wrote:

> The only thing you lose [without sharedUserId] is that the additional .apks

> can't add their own required permissions in to the uid you run under.

I noticed in the StorageManager docs that sharedUserId has taken on a
new role that might be beneficial for native code plugins:

http://developer.android.com/reference/android/os/storage/StorageManager.html

"Note: you can only mount OBB files for which the OBB tag on the file
matches a package ID that is owned by the calling program's UID. That
is, shared UID applications can attempt to mount any other
application's OBB that shares its UID."

Agreed?

On Dec 29 2010, 2:09 pm, Dianne Hackborn <hack...@android.com> wrote:

> hack...@android.com

[Olivier Guilyardi]

Hello Dianne and others,

I'm waking up an old thread, which was mainly about loading native plugins.

On 12/30/2010 01:25 AM, Dianne Hackborn wrote:

> As I said, you will generally be much happier if you
> implement plug-ins as services so they can be isolated from the main
> application.

Ok, let's say that instead of plugins intended to be loaded with dlopen(), such
plugins are provided as services.

Now, in my case this is for audio plugins. I am not going to send and receive
all PCM audio data through AIDL calls, this is heavy, right?

So, how can I get a piece of shared memory to exchange audio data with a
service? I mean in the same manner as libmedia communicates with audioflinger.

Also, in user-code, accessing services can only be done in Java, not in native
code, right?

TIA

Olivier

[David Turner]

On Tue, Mar 1, 2011 at 9:07 PM, Olivier Guilyardi <li...@samalyse.com> wrote:

Hello Dianne and others,

I'm waking up an old thread, which was mainly about loading native plugins.

On 12/30/2010 01:25 AM, Dianne Hackborn wrote:

> As I said, you will generally be much happier if you
> implement plug-ins as services so they can be isolated from the main
> application.

Ok, let's say that instead of plugins intended to be loaded with dlopen(), such
plugins are provided as services.

Now, in my case this is for audio plugins. I am not going to send and receive
all PCM audio data through AIDL calls, this is heavy, right?

I can't comment on AIDL, but Unix-domain sockets are *very* fast on Linux, and much easier to setup/manager than a shared memory buffer.

 

So, how can I get a piece of shared memory to exchange audio data with a
service? I mean in the same manner as libmedia communicates with audioflinger.

Also, in user-code, accessing services can only be done in Java, not in native
code, right?

TIA

Olivier

[Olivier Guilyardi]

Hello,

I'm waking up that old thread again... Been thinking.

On 03/02/2011 06:00 AM, David Turner wrote:
>
> On 12/30/2010 01:25 AM, Dianne Hackborn wrote:
>
> > As I said, you will generally be much happier if you
> > implement plug-ins as services so they can be isolated from the main
> > application.
>
> Ok, let's say that instead of plugins intended to be loaded with
> dlopen(), such
> plugins are provided as services.
>
> Now, in my case this is for audio plugins. I am not going to send
> and receive
> all PCM audio data through AIDL calls, this is heavy, right?
>
> I can't comment on AIDL, but Unix-domain sockets are *very* fast on
> Linux, and much easier to setup/manager than a shared memory buffer.

Nice, thanks for the advice. But right now I'm interested in shared memory.

> So, how can I get a piece of shared memory to exchange audio data with a
> service? I mean in the same manner as libmedia communicates with
> audioflinger.

Any answer to this question? I know I could share memory between processes and
supposedly different apps by using mmap(). But there is a security issue,
because the mmap'ed file would need to be readable by all, since this is
intended for inter-apps communication.

So, how can I get a piece of shared memory between two apps (third-party) and
this in a secure manner?

--
Olivier