Apps on SD

[CJ Chitwood]

I don't see the card-removal-while-app-is-running thing to be that big
a problem. Tell the users once, that if they remove the card without
a proper dismount (and that's where you'd need to edit Android to
close programs when a dismount request is made by the user) then the
phone will die and/or brick itself. Nuff Said. Trust me, they'll do
it but once if that.

Secondly, with *all* apps being on SD (apps, not OS), you have the
internal memory freed. This "new" memory could be used as a temp
folder, in which the running apps are copied (at least, the core part
of them to save space) so that if the card IS dismounted wrong, the
dismount notice can be sent to apps that are running, and they can
"prepare" for the lost file descriptors (or whatever else) (keep in
mind, my programming experience is very little more than "Hello World"
and contains no GUI).

As just an end-user (again, being not a programmer), I see no reason
why the filesystem inside the "encrypted" file on the FAT can't be
e.g. Reiser or EXT3 (or 4) or the like. In fact, this would make it
more difficult (though far from impossible) for people on at least
Windows systems (the majority of your userbase) to copy the apps from
one card to a card of another phone. Why does the card have to be FAT
in the first place? As difficult as it is to remove (at least, on the
G1) I don't see it being used in many other systems that much.

Anywhoo, I just got interrupted and I have to run, but these were my
thoughts on the subject. I know it's difficult to do, don't get me
wrong, but with the apps that will be coming out, people aren't going
to want to choose which apps to have just to stay in the confines of
the (at least G1) phone's limited internal memory. Apps-on-SD is a
MUST for this to thrive on limited hardware platforms, at least in my
opinion. I know it's not an iPhone competitor, but you have to accept
that the iPhone is what people are comparing (again, at least the G1)
to. Now, if the platform it's put on has non-removable 16 GB memory
like the iPhone, maybe apps-on-SD isn't a big deal, but for me,
personally, as a G1 user, it is. I don't expect my phone to beat an
iPhone but I do expect it to be at least as functional considering it
has comparable hardware (and even some the iPhone doesn't have).

US$0.02. AFAICT, YMMV. HIH.

[Andrew Gorcester]

Without comment on the arguments here, a technical question:

Is it currently possible for an application to "bootstrap" itself by
downloading a small application from the marketplace that then downloads
additional bytecode to the SD card and executes it?

If that is not possible, is it possible for an application to download
all of the actual code to internal memory from the marketplace but then
download additional media (for instance, graphics and a text script
for a game) to the SD card and access it seamlessly from the
application?

Obviously either of these options would require the application itself
to address the questions being discussed here, like what to do if the
card is removed.

[Jeff Hamilton]

> one card to a card of another phone. Why does the card have to be FAT
> in the first place? As difficult as it is to remove (at least, on the
> G1) I don't see it being used in many other systems that much.

When you plug the phone into a desktop machine the desktop sees the
phone as essentially a card reader and gets direct access to the
memory card itself instead of to the file system, so the desktop needs
to have drivers to support the file system on the card. FAT is the
only file system that is common to Windows, Mac OS, and Linux, as well
as various consumer electronics like TVs and NAS units that you can
plug the phone into to copy or view photos.

-Jeff

[hackbod]

On Nov 15, 10:31 am, "Andrew Gorcester" <andrew.gorces...@gmail.com>
wrote:

> Is it currently possible for an application to "bootstrap" itself by
> downloading a small application from the marketplace that then downloads
> additional bytecode to the SD card and executes it?

Currently you can't do this with bytecode, but that will be fixed in a
future release.

> If that is not possible, is it possible for an application to download
> all of the actual code to internal memory from the marketplace but then
> download additional media (for instance, graphics and a text script
> for a game) to the SD card and access it seamlessly from the
> application?

Yes this is possible.

Note, however, that for both of these situations you are leaving this
data out in the open for any other app to corrupt or infect. Not such
a big deal with assets, but definitely a scary thing for code. Enough
that I would strongly recommend against it, since it opens a big door
to viruses and such.

[Jean-Baptiste Queru]

Lots of ideas in your email, but unfortunately there are also a few issues.

I don't believe that it's acceptable for a plaftorm designed for
consumer devices to be designed around the notion that certain
operations will crash the phone. To me that'd be a major design flaw.

I'm not convinced that unmounting would make a difference in first
approximation: once the card is unmounted, the files aren't any more
accessible than if the card had been pulled. Granted, the act of
unmounting could give the system more opportunities to get ready and
for the apps to save their state, but ultimately the system still has
to be protected against the files disappearing with no notice (because
some users will pull their cards anyway).

Using the internal storage to cache apps is an interesting idea, but
it doesn't come for free: it limits the size of any individual app to
the size of the internal storage, it causes some severely user-visible
delays (copying 10+ MB takes time), and it still requires some serious
system-level logic (albeit a different kind of logic) to manage that
space. Worse, since some apps will still be using internal storage for
their data, you could end up with situations where some apps installed
on the SD card can't be launched any more if there's not enough space
to cache the relevant data. Finally, keeping apps "running" without
letting them access their data is unlikely to make them more useful
than just killing them, and creates some additional complexity if you
leave apps installed on the SD card "running" while the card is gone
if you insert a different SD card while they're still running.

Finally, an aspect that you haven't mentioned is the notion of an
"installed" app, and how that notion gets muddied when installing apps
on removable storage: to take a simple example, if you have 2 apps on
2 different SD cards that each publish the same content provider, how
do apps that are not installed on the SD card distinguish between the
two? One way to resolve that might be to not allow apps to be
installed on the internal storage at all (in which case the problem
mostly goes away on its own), but it still seems to me that the are
some legitimate use cases for apps on internal storage (e.g. to manage
a music library that spans multiple SD cards).

JBQ

[Andrew Gorcester]

Running applications off of discs of various types was very common in
computers until fairly recently. There should be some good solutions
to this problem on different platforms out there that we can mine for
ideas.

I think probably the best way to handle the unexpected removal of an
SD card that hosts a running application is to have a standard feature
of the OS that pauses the application with a (possibly catchable?)
UNIX signal, similar to if you send ctrl-Z to a running process on the
command line, and then displays, "please re-insert SD card". Users
can then choose to switch away from the application with the back
button, which means they will be able to pause the application until
their business with the card is done, and then reinsert it and
continue. They can choose to close the application. Or, they can
choose to reinsert the card and resume the application. Linux
programs are already expected to be able to handle pauses like this,
right?

The problem is of course that if the user does choose to close the
application, there should be some way to handle that in an orderly
fashion. But given all of the other ways that programs can be
terminated rudely on a phone, such as the device having its battery
removed, it's arguable applications should be able to handle this
regardless.

[tomgibara]

I've just read this thread and thought I'd throw out an idea that
occurred to me. I've not given this idea any significant thought - it
was the first solution that came into my head and it has a very high
chance of being hopelessly flawed - feel free to shoot it down.
Disclaimer out of the way, here was my immediate line of thinking:

1) APKs are signed and therefore tamper-proofed, so if you want to
store apps on the SD card, you could do worse than simply store the
APKs

2) If you want to temporarily offload an app, you could ensure that
the APK is cached on the SD card, record that the user has accepted
installation of the app, uninstall it from phone storage, but leave
behind its data directory

3) To restore the application (space permitting) you then simply
reinstall the application from the cached APK (without necessarily
notifying the user)

4) Apps obviously rely on establishing IntentFilters and other things
in their manifest. This could be handled by a system 'stub' which
responded to matches on IntentFilters by restoring the application.

Obviously this idea is fraught with complexities (some of which may
kill it stone dead) - not only that, but the delay in installing the
application may render it useless for all but the smallest apps (which
aren't the target anyway). It clearly isn't useful for an application
which needs to be running fairly consistently in the background.

As I write, so many obvious problems occur to me that I'll just leave
the rest to the reader. Nevertheless, I think its three redeeming
features are that:

1) it works mostly by the install/uninstall mechanism which already
works nicely

2) it ensures that any time that application code is running, its
installed 'as normal' on the device (no surprises in that sense)

3) unavailability of the SD Card (or lack of space for the reinstall)
only becomes manifest when restoring the application - at which point
the application won't be running. This can (perhaps!) be masked by
simply letting the IntentFilter match which triggered it fail.

Tom

[Mark Murphy]

Overall, I like the concept!

tomgibara wrote:
> 1) APKs are signed and therefore tamper-proofed, so if you want to
> store apps on the SD card, you could do worse than simply store the
> APKs

That works for free apps. Paid-for apps, though, are a problem -- now,
the APK can be removed from the card and distributed to hither and yon,
which may not be what the app author intended.

It may be that the off-loaded APK on the SD card isn't the virgin APK,
but one that has been lightly encrypted or something, to prevent casual
copying. That might not stop a serious pirate, since the
encryption/decryption logic would be open source.

> 1) it works mostly by the install/uninstall mechanism which already
> works nicely

Maybe the answer is that, rather than making this seamless, we add a
seam, tied in with install/uninstall.

Perhaps the solution is for users to be able to mark apps as "parked" or
something -- they bought it, they want to be able to use it sometime,
but right now they could use the room. Through the Manage Applications
screen, they'd "park" their app, which does what you describe -- leave
the data directory behind but remove the APK. The APK could go to the SD
card or could go to the cloud (e.g., back to App Market or other market
supporting the necessary hooks).

The app would then only be available via the launcher icon(s), which
would lead to a "you parked this app -- do you want it back now?"
screen. Or, you could see it in Manage Applications and restore it from
there.

Apps could be flagged as "parkable locally" (meaning they can be parked
to the SD card) or "only parkable remotely" (meaning they'd have to be
re-downloaded but not re-purchased). That way, developers who don't want
to run the risk of their APKs getting out can bypass the SD card parking
lot. And, some apps that are too dependent on background processing or
other OS hooks are flagged as "not parkable", meaning parking won't be
presented as an available option.

--
Mark Murphy (a Commons Guy)
http://commonsware.com

Android Training on the Ranch! -- Mar 16-20, 2009
http://www.bignerdranch.com/schedule.shtml

[Tom Gibara]

I always assumed that the market must be dealing with your first concern (people grabbing the apk) already - probably by publishing a modified apk that is keyed to the user account and signed by Google. I hadn't bothered to ask for clarification on that because you can't sell apps yet. Now I stop to think about it, I doubt that can work as I envisioned it.

I don't like the idea of the application going back into the cloud, the usefulness of downloading an app is that you 'have it'. That is, it's there quickly and doesn't need anything external to the device to get at it (eg. a network connection). If apks really are unprotected, there may already be alternative ways of accessing them.

Tom

2008/11/17 Mark Murphy <mmu...@commonsware.com>

[hackbod]

I should say, we definitely -do- want to allow users to store
applications on an SD card, but as we can see this is clearly not an
easy thing to do. :)

We have actually had a lot of discussion about this, and our current
plan for supporting it is:

(1) We will have a single file on the SD card holding an encrypted
Linux filesystem. This provides an area on the SD card where we can
place data with all of the full Linux permissions in force: the
ability to restrict who can read and write while files, etc.

(2) The user can install applications onto an SD card, which the
system does by placing them in the encrypted filesystem (the system
would need to grow or shrink that file and its contained filesystem as
its contents change, so any operations that add to it could require a
fair amount of work).

(3) Applications that are installed on the SD card can be run just as
normal; no need to copy them into internal flash or anything like
that, because the filesystem they are on can be trusted.

(4) If the user pulls out the SD card, all applications currently
running on it will be immediately killed and marked as "unavailable"
so that any attempt to use them will behave as if they aren't there
(activity not found exception thrown, broadcast not received, etc).

(5) If the user takes an SD card out of one device and places it into
another, any applications installed on that SD card are NOT available
on the second device, because they have not been installed there.

Of course, this all entails significant work across the entire stack,
from the kernel up to the various app installer and management UIs, so
it's not going to happen over night. :}

[Tom Gibara]

This is clearly a more complete solution than my ad-hoc suggestion.

Does (2) extend to the SD card being mounted on a host computer via USB?

What isn't clear to me is how (or even whether) users will be involved in deciding where an application is installed. A user will probably regard some applications as critical, (eg. a replacement home application or replacement dialer) and will want these to operate with or without an SD card. But involving the user in these decisions might be troublesome (should users be expected to understand the technical implications of installing an application to the SD card?). Conversely, it doesn't seem natural for an application developer to control where the application may be deployed.

I suppose the advice to users could be as straightforward as "If you want the app available at all times, install it to the phone, otherwise you can install it to the SD card)." However, users could still be surprised that an application installed on the phone fails on removal of the SD card, because they are unaware that it depends on an application that resides on the card.

I can't immediately see a practical solution to this other than leaving it to the application developer to educate the user about how their app (and its dependencies) need to be installed.

Tom.

2008/11/17 hackbod <hac...@gmail.com>

[Andrew Gorcester]

Are there technical reasons why the application must be killed and not
paused on removal of the SD card, or is that a design decision?

[Jean-Baptiste Queru]

Now we're talking :)

Within this design, I have a few questions around the notion of
"unavailable" apps:

-what if an app is installed on the SD card, the SD card is removed
and lost, and the user wants to install the same card on internal
storage?

-will it be possible to migrate apps from internal storage to the SD
card (and vice versa) (this would be another way to achieve Tom's
goals, if I've read correctly).

-what will happen if a user wants to install an app on multiple SD cards?

-how will this scale if Android evolves to systems that support
multiple external storage devices?

Thanks,
JBQ

[Mike Lockwood]

Unfortunately the kernel does not handle the SD card removal very
well. It cannot cleanly unmount the SD card until all open files are
closed. So if an application leaves a file open indefinitely, it
prevents the SD card from being unmounted, which means it cannot be
remounted at the /sdcard mount point.

This is something that could be fixed within the Linux file system
implementation, but it would be a lot of work.

Mike

--
Mike Lockwood
Google android team

[Jean-Baptiste Queru]

Besides the kernel issues related to the way linux isn't happy when a
device disappears while there are files opened on it, I'd be worried
about all the IPCs that happen between apps. Any situation where
paused app exposes a content provider, a service, or even an activity
could cause many other apps to get transitively paused.

Let me take an example: suppose that the download manager downloads
data into another app's content provider (that's currently the
long-term design to make the download manager download files into
other apps' private storage locations). Suppose that such a target app
gets paused. The download manager now gets paused. This can in turn
cause the browser, market, gmail, and even system updates to get
paused (if they happen to start a download). Things might ripple much
further. I'd prefer that the download manager get a clear message that
the app is gone and handle that situation in the most appropriate way
(probably mark the download as paused for now, waiting for the app to
resume it when it returns, and flushing it away after a while if it
doesn't hear back from the app).

JBQ

[Dave Johnson]

The way I look at it... The SD card should not be removed the way a CD is in a computer. You should get the biggest card you can get and leave it alone.

On Nov 17, 2008 9:56 AM, "Tom Gibara" <m...@tomgibara.com> wrote:

This is clearly a more complete solution than my ad-hoc suggestion.

Does (2) extend to the SD card being mounted on a host computer via USB?

What isn't clear to me is how (or even whether) users will be involved in deciding where an application is installed. A user will probably regard some applications as critical, (eg. a replacement home application or replacement dialer) and will want these to operate with or without an SD card. But involving the user in these decisions might be troublesome (should users be expected to understand the technical implications of installing an application to the SD card?). Conversely, it doesn't seem natural for an application developer to control where the application may be deployed.

I suppose the advice to users could be as straightforward as "If you want the app available at all times, install it to the phone, otherwise you can install it to the SD card)." However, users could still be surprised that an application installed on the phone fails on removal of the SD card, because they are unaware that it depends on an application that resides on the card.

I can't immediately see a practical solution to this other than leaving it to the application developer to educate the user about how their app (and its dependencies) need to be installed.

Tom.

2008/11/17 hackbod <hac...@gmail.com>

> > > I should say, we definitely -do- want to allow users to store > applications on an SD card, b...

[Andrew Gorcester]

Certainly, many users will not see it that way. Also, currently the
SD card is unmounted when the device is hooked up to a computer via
the USB cable and the card is mounted on the computer. That would
have the same effect on running applications as removing the card.

[Dave Johnson]

Well put.

On Nov 17, 2008 1:19 PM, "Andrew Gorcester" <andrew.g...@gmail.com> wrote:


Certainly, many users will not see it that way.  Also, currently the
SD card is unmounted when the device is hooked up to a computer via
the USB cable and the card is mounted on the computer.  That would
have the same effect on running applications as removing the card.

On Mon, Nov 17, 2008 at 1:36 PM, Dave Johnson <davefilms.us@gmail.com> wrote: > The way I look at i...

[Jay Freeman (saurik)]

One possible suggestion: using a second partition on the SD card. This
poses other issues and has some negative tradeoffs as well
(repartitioning user's existing cards is a serious one, as well as
being unable to share cards between devices as easily due to your
encryption requirements), but I think it is an interesting option to
consider as it has one major upside: you can mount the device over USB
to the primary partition and still have the second partition mounted.

In practice this doesn't quite work currently as /sdcard never comes
back... ;P it somehow notices that the device is in use and doesn't
take it back when you unplug the USB cable (or during bootup, if you
convince init to mount the partition at that time). This problem,
however, is likely somewhat simply solvable, and I think this entire
model is much simpler from a purely selfish Android point of view than
the loopback mounted file. ;P

-J

[Jondice]

I like your thinking Jay! This definitely seems desirable.

[Jean-Baptiste Queru]

That's another approach, with pros and cons.

Comparing the idea of filesystem-in-a-partition and
filesystem-in-a-file, I can think of the following differences:

-filesystem in a partition makes the process of preparing an SD card
appropriately a bit more awkward (e.g. what if the SD card didn't have
a partition table to start with?), makes it a bit harder to
dynamically resize the android-specific filesystem, and makes it quite
hard to restore the SD card to a "normal" state by using only a
computer. It might also cause various operating systems to complain
that the SD card isn't properly initialize and cause users to
mistakenly re-initialize the second partition with another filesystem
and lose their android apps.

-on the other hand, filesystem in a partition allows to trivially
UMS-mount the "regular" partition on a computer while leaving the
"android" partition mounted on the phone.

JBQ

[Peli]

From the user's perspective, probably it would be enough to extend the
Activity lifecycle a bit, to give the illusion of "pause" even if the
application is technically killed:

Allow for regular saveInstanceStates (every few seconds if there is
activity) to the SD card. If the application is suddenly killed
because the card has been pulled, it can start next time with the
state as it was right before the kill, so the user can have the
impression that the application was simply paused in the mean-time.

This could be supported by an activity manager that lists these
activities as "paused" even though they are killed and their last
state has been stored on the SD card.

Peli

On Nov 17, 5:21 pm, Jean-Baptiste Queru <j...@google.com> wrote:
> Besides the kernel issues related to the way linux isn't happy when a
> device disappears while there are files opened on it, I'd be worried
> about all the IPCs that happen between apps. Any situation where
> paused app exposes a content provider, a service, or even an activity
> could cause many other apps to get transitively paused.
>
> Let me take an example: suppose that the download manager downloads
> data into another app's content provider (that's currently the
> long-term design to make the download manager download files into
> other apps' private storage locations). Suppose that such a target app
> gets paused. The download manager now gets paused. This can in turn
> cause the browser, market, gmail, and even system updates to get
> paused (if they happen to start a download). Things might ripple much
> further. I'd prefer that the download manager get a clear message that
> the app is gone and handle that situation in the most appropriate way
> (probably mark the download as paused for now, waiting for the app to
> resume it when it returns, and flushing it away after a while if it
> doesn't hear back from the app).
>
> JBQ
>
> On Mon, Nov 17, 2008 at 8:10 AM, Andrew Gorcester
>

> <andrew.gorces...@gmail.com> wrote:
>
> > Are there technical reasons why the application must be killed and not
> > paused on removal of the SD card, or is that a design decision?
>

[Peli]

I like and support this concept. :-)

Will the single file have a random unique name so that one SD card can
be used to host applications of several devices? (although each device
can only access "their" file on the SD card). Or it is actually enough
if they are numbered sequentially: android1.fs, android2.fs, etc. and
each device looks for the file they can read.

Peli

[Peli]

I guess the easiest solution would be to treat applications living in
internal memory and on the SD card just like two applications that
live in different packages. (which easily extends to living on several
SD cards).

(technically, this could even be a prefix to package names: com.myapp -
> internal.com.myapp or sdcard1.com.myapp)

If a user calls an intent that is available internally and on the SD
card, they will be presented an Activity Chooser to choose which of
the two versions to use. Each of them comes with their own content
providers and private data, so conflicts are resolved and handled just
as they are now for different packages.

There could be a possibility to move an application from one location
to another, together with all data. This could simply be an additional
context menu entry in the list of all applications.

The user could choose at download time where to initially store an
application, through an additional drop-down list, or through a menu
setting.

Peli

On Nov 17, 5:12 pm, Jean-Baptiste Queru <j...@google.com> wrote:
> Now we're talking :)
>
> Within this design, I have a few questions around the notion of
> "unavailable" apps:
>
> -what if an app is installed on the SD card, the SD card is removed
> and lost, and the user wants to install the same card on internal
> storage?
>
> -will it be possible to migrate apps from internal storage to the SD
> card (and vice versa) (this would be another way to achieve Tom's
> goals, if I've read correctly).
>
> -what will happen if a user wants to install an app on multiple SD cards?
>
> -how will this scale if Android evolves to systems that support
> multiple external storage devices?
>
> Thanks,
> JBQ
>

[Peli]

> -on the other hand, filesystem in a partition allows to trivially
> UMS-mount the "regular" partition on a computer while leaving the
> "android" partition mounted on the phone.

Silly question from a non-expert in this field:
Is it very difficult to keep the Android filesystem (stored as a file
on the SD card) locked and grant access to the rest of the SD card to
a computer through USB? (assuming that the file size does not change
while the USB cable is plugged in - which would be also the case if
one had 2 partitions).

Then one could combine both advantages. Anyway, the computer could not
make any use of accessing the Android specific file, and locking it
would add another (small) layer of security. (Of course, the user
could still delete the file in the SD card slot on their computer).

Peli

[Jean-Baptiste Queru]

It's not that easy, because you'd have to prevent the computer from
modifying anything that's related to the image (both its content and
its FAT entries, along with its parent directory, recursively until
the root directory).

Even if you did that, that wouldn't really solve the problem, because
the computer could be using a cache and could have started to write a
new state from its cache when it hits a "restricted" block, and then
leave the filesystem in a dirty state.

To make things worse, you couldn't protect the FATs themselves on a
block-by-block basis, but you'd have to do it on an entry-by-entry
basis.

Once you dig a bit deeper in that direction, the complexity of the
task becomes so high that other options become more practical.

JBQ

[Dianne Hackbod]

On Wed, Nov 19, 2008 at 8:06 AM, Peli <peli...@googlemail.com> wrote:

Will the single file have a random unique name so that one SD card can
be used to host applications of several devices? (although each device
can only access "their" file on the SD card). Or it is actually enough
if they are numbered sequentially: android1.fs, android2.fs, etc. and
each device looks for the file they can read.

There really isn't much more thinking on this than what I said.  This is an area we are thinking about, but right now there are higher priorities so there is not much active work going on.  At any rate, the apps in the SD card would need to be associated with a particular device, because that is the only one they were actually installed/accepted on.  I don't think you would want the act of putting in an SD card to automatically install all of the apps that happen to be on it.  Also the key for the encrypted filesystem needs to be owned and private to the device it is intended for, or else you defeat the whole purpose of having an encrypted filesystem.

--
Dianne Hackborn
hac...@android.com

[Dianne Hackbod]

I have no expectation of doing anything so complicated any time soon.  Whenever an implementation for this appears, I expect at the application level the removal of the SD card will result in all applications that are on there disappearing, their processes being killed, and current state in the activity manager removed.  Trying to do otherwise just opens a big can of worms of what to do with any of these things if they are somehow needed: someone tries to access a content provider on them, return to or start one of their activities, send a broadcast, bind to a service, etc.  Best to just pretend they aren't there.

Also you won't be able to have the same application installed on the internal flash or external SD card.  It's one or the other.

--
Dianne Hackborn
hac...@android.com

[Peli]

Ok. If an application really needs to, it could auto-save frequently
to protect the user from data loss, but I see that it is best and
easier for the framework to just kill the applications.

> Also you won't be able to have the same application installed on the
> internal flash or external SD card. It's one or the other.

But this would immediately bring Jean-Baptiste's question up again:
What if the user lost their SD card - can they install the application
again on internal memory?
If yes, what happens if they found their SD card again? (Hm, probably
in that case simply the SD card version would be locked and cannot be
accessed anymore...?)

Peli

> hack...@android.com

[Dianne Hackbod]

On Wed, Nov 19, 2008 at 12:34 PM, Peli <peli...@googlemail.com> wrote:

But this would immediately bring Jean-Baptiste's question up again:
What if the user lost their SD card - can they install the application
again on internal memory?
If yes, what happens if they found their SD card again? (Hm, probably
in that case simply the SD card version would be locked and cannot be
accessed anymore...?)

Probably whatever is on the internal flash would override anything on the sd card.
 

--
Dianne Hackborn
hac...@android.com

[Todd Mars]

I'm confused, what is the definition of 'app installed on SD card'?
do you mean just the app's code?
the OS should know what apps are installed, if the code for the app is
on the SD card,
then there is some pointer to the code. If the pointer is invalid,
the app cannot run and must
be uninstalled. Apps also have data, that data is in the mysql
database right? Which is a totally seperate
issue, that of retaining app data if the app is uninstalled.

So what other data is involved in the definition of 'app installed on
SD card' ?
Thanks
T.

[Peli]

I guess the whole point of having an encrypted file on the SD card is
that it can hold the apk file as well as the private data files
(databases, files, shared preferences), so that one can make most use
of the space on the SD card. (there is no point in outsourcing only
the apk files, if the databases fill up all internal memory...)

Peli

[Dianne Hackborn]

On Wed, Nov 19, 2008 at 1:51 PM, Peli <peli...@googlemail.com> wrote:

I guess the whole point of having an encrypted file on the SD card is
that it can hold the apk file as well as the private data files
(databases, files, shared preferences), so that one can make most use
of the space on the SD card. (there is no point in outsourcing only
the apk files, if the databases fill up all internal memory...)

No, I doubt we would be storing normal private data there.  This would be problematic due to the issue of managing the file size -- creating files may require growing the file system, which is a very expensive operation, so you can end up just freezing the app at arbitrary points for arbitrary amounts of time.

--
Dianne Hackborn
hac...@android.com

[Andrew Gorcester]

How set in stone is that decision? I'm not certain the alternative,
where the tiny available portion of internal storage on the G1 (which
will get tinier as features are added to Android) becomes completely
filled with data and applications cease to function, is preferable.

There is a limited amount of space in internal storage, and there is a
limited amount of space on any given encrypted filesystem on the SD
card. But one limitation is potentially controllable to some degree
and one isn't. Applications will have to deal with running out of
space either way, but if their options include "allocate more space
from the beginning as instructed by the developer" and "take some time
to recreate the filesystem" it should be easier to handle.

Other options include making a single filesystem for all apps and data
stored on a card, with the size determined by the user. This
filesystem could be a file or it could be a partition as previously
suggested, although I think the partition idea (while clever!) is
likely to cause a lot of trouble for other devices.

The impossibly cramped internal storage on the G1 in particular makes
the issue more serious than it would be otherwise. I actually wonder
if Android itself won't grow too large to fit on the G1 in the future.

[Dianne Hackborn]

On Wed, Nov 19, 2008 at 4:23 PM, Andrew Gorcester <andrew.g...@gmail.com> wrote:

How set in stone is that decision?  I'm not certain the alternative,
where the tiny available portion of internal storage on the G1 (which
will get tinier as features are added to Android) becomes completely
filled with data and applications cease to function, is preferable.

Applications are free to put their non-secure data in the SD card.  This isn't really a decision, it's a basic characteristic of creating a secure filesystem in a file for secure storage, you just can't treat it as a dynamic resource like apps would expect when writing on a file.  Of course you could instead partition the SD card, but I think that would have a worse set of problems.
 

There is a limited amount of space in internal storage, and there is a
limited amount of space on any given encrypted filesystem on the SD
card.  But one limitation is potentially controllable to some degree
and one isn't.  Applications will have to deal with running out of
space either way, but if their options include "allocate more space
from the beginning as instructed by the developer" and "take some time
to recreate the filesystem" it should be easier to handle.

One could imagine an API for an application to reserve X bytes of storage, but this is a far different model than the current internal storage, and thus would require them to explicitly use it.
 

Other options include making a single filesystem for all apps and data
stored on a card, with the size determined by the user.  This
filesystem could be a file or it could be a partition as previously
suggested, although I think the partition idea (while clever!) is
likely to cause a lot of trouble for other devices.

I think there are many reasons the partition approach would be inferior.  And if you addressed some of them -- such as dynamically resizing the partition based on the usage of the filesystems, you then introduce the whole set of same problems for both the encrypted side of things as well as the existing normal SD card.

--
Dianne Hackborn
hac...@android.com

[Jay Freeman (saurik)]

For the record, I figured out what was causing this on the android-
porting list. I now have my G1 setup to have /mnt mounted from /dev/
block/mmcblk0p2 (the second partition of the SD card) at boot without
losing /sdcard or USB mounting support.

http://groups.google.com/group/android-porting/browse_thread/thread/a67cbe36603d429a
http://review.source.android.com/4044

My goal is to have / be mounted from that partition (which I'm still
having a problem with, but will hopefully have solved soon), but a
very simple trick would be to just make it some random sub-folder of /
system.

http://www.telesphoreo.org/pipermail/g1-hackers/2008-November/000032.html

Translation: users who want to try this today should be able to do so
(assuming they have root access on their G1 thanks to the RC<30 bug).
This means someone can start working on whatever UI or backend
improvements are needed to handle the otherwise undefined behavior
when the card is removed and have a viable platform to test it on.

Mounting it over loopback to a file on the SD card would also be
doable at this point, although has the aforementioned problems with
USB mounting. I have submitted a patch to init to support mounting
over /dev/block/loop*.

http://review.source.android.com/4045

If anyone wanted to try going the "second partition" route, I also
have a static build of parted (compiled to ARM EABI to use on the G1)
for use with repartitioning existing cards. I tested this by buying an
8G microSD card, putting it in my G1, and repartitioning it to be a
4GB/4GB split: vfat/ext2. I hope to have a front-end for helping with
this at some point (if anyone wants to help out on that front it would
be awesome ;P).

http://cache.saurik.com/android/armel/parted

Sincerely,
Jay Freeman (saurik)
sau...@saurik.com
http://www.saurik.com/

On Nov 17, 5:47 pm, "Jay Freeman (saurik)" <sau...@saurik.com> wrote:
...

> In practice this doesn't quite work currently as /sdcard never comes
> back... ;P it somehow notices that the device is in use and doesn't
> take it back when you unplug the USB cable (or during bootup, if you
> convince init to mount the partition at that time).
>

> -J

[Peli]

But then I don't really understand the point of having an encrypted
file system on the SD card anymore...

apk files are signed anyway, so even if they are stored on an insecure
file system, their integrity could be checked anytime.

If the question is just about storing the apk files on the SD card -
why can't they be put there direclty? I really thought this whole
discussion was about storing internal data securely together with the
apk files...

Peli

On 20 Nov., 00:20, "Dianne Hackborn" <hack...@android.com> wrote:

> hack...@android.com

[Jean-Baptiste Queru]

The fact that they're signed doesn't prevent anyone from re-signing
them with their own signature, or from copying them to e.g. a web
server from which anyone could install them.

JBQ

[Peli]

Regarding the first point, the system would just have to store the
signature of the apk in an internal database to notice a change.

Regarding the second point, nobody prevents me even now to use adb in
developer mode to copy apk files from /data/app to my computer and
install them on some other phone or publish them on the web..

Peli

On 20 Nov., 06:22, Jean-Baptiste Queru <j...@google.com> wrote:
> The fact that they're signed doesn't prevent anyone from re-signing
> them with their own signature, or from copying them to e.g. a web
> server from which anyone could install them.
>
> JBQ
>

[Dianne Hackborn]

On Wed, Nov 19, 2008 at 9:42 PM, Peli <peli...@googlemail.com> wrote:

Regarding the first point, the system would just have to store the
signature of the apk in an internal database to notice a change.

Validating the signature of an .apk/.jar file is quite expensive -- easily 10s or more of seconds -- to do.  And if the file is stored directly on the SD card, any part of it could be modified at any time, so you couldn't even trust it was okay once you were done verifying it.
 

Regarding the second point, nobody prevents me even now to use adb in
developer mode to copy apk files from /data/app to my computer and
install them on some other phone or publish them on the web..

This is a completely different issue.  What we are addressing is having a security sandbox at -all- to prevent any other apps on the phone from corrupting, infecting, or otherwise using the contents of the SD card to do Bad Things.  This has nothing to do with being able to copy the contents of an .apk.  It is a matter of the system being able to guarantee that what is and the user think they have installed is indeed that, and never changes from under them.

--
Dianne Hackborn
hac...@android.com

[Peli]

Ok, thanks for the clarifications.

If validating a signature can take so long, are apk file signatures
actually validated when they are downloaded?
(I ask because "adb install" complains quite quickly if signatures
differ - or maybe my apk files are not big enough :-) )

Peli

On Nov 20, 8:57 am, "Dianne Hackborn" <hack...@android.com> wrote:

> hack...@android.com

[al74]

You should take a look at online forums and understand that this
problem is widespread and should be prioritized accordingly. The App
Market is only a month old and we have to uninstall apps to add new
ones.

[Andrew Gorcester]

I think I understand some of the technical reasons why data can't be
stored in the same filesystem as the application, but I still think
there needs to be some kind of solution for this issue.

A year from now, when there are plenty of other devices running
Android on the market and all of those devices have more internal
flash than the G1, developers may feel little obligation to put
relatively small (1-10 megabyte) amounts of user data explicitly on
the SD card. If there is no general solution for moving that user
data off of internal memory, the G1 could become very difficult to
use.

What about giving users the ability to create another filesystem
exclusively for user data, shared across applications installed on
that SD card? Users would have to choose how much space to allocate,
which is not an optimal solution. But given that only a few tens of
megabytes are available on the G1 - and that amount will shrink as
features are added to Android - I think it would be much better for
users than the alternative.

[Peli]

It is also a financial problem... if people don't download and pay for
applications just because their internal memory is full, this has
immediate impact on sales by developers and the carriers.

Probably you should talk to T-mobile about this point and ask them for
opinion ;-)

Peli
PS: If you only store apk files on the SD card and no data, what about
adding one encrypted file system file for each apk? Then there would
not be any issue anymore with growing or shrinking it, just creating
or deleting. Just a thought :-)

PPS: If Google doesn't have the resources right now, is there a way to
split the problem into things Google has to do (the most critical
internal things), and things maybe "the community" can supply? (like
changes to the UI of the application manager...). Is there a way this
can be approached in small steps? (i.e. maybe first a version that is
not very user friendly, but that works?)

[Dianne Hackborn]

On Thu, Nov 20, 2008 at 1:39 AM, Peli <peli...@googlemail.com> wrote:

If validating a signature can take so long, are apk file signatures
actually validated when they are downloaded?
(I ask because "adb install" complains quite quickly if signatures
differ - or maybe my apk files are not big enough :-) )

Reading the signature the app purports to be signed with is easy; after that, you need to read through every file in the entire .apk to verify it is actually signed with the signature.

--
Dianne Hackborn
hac...@android.com

[Dianne Hackborn]

On Thu, Nov 20, 2008 at 7:22 AM, Andrew Gorcester <andrew.g...@gmail.com> wrote:

I think I understand some of the technical reasons why data can't be
stored in the same filesystem as the application, but I still think
there needs to be some kind of solution for this issue.

I have posted a fairly detailed description of the approach I think we will take for storing apps on the SD card.  Nobody has said it shouldn't be done.  There is no need to try to convince anyone that it should be done.

--
Dianne Hackborn
hac...@android.com

[Dianne Hackborn]

On Thu, Nov 20, 2008 at 8:08 AM, Peli <peli...@googlemail.com> wrote:

PPS: If Google doesn't have the resources right now, is there a way to
split the problem into things Google has to do (the most critical
internal things), and things maybe "the community" can supply? (like
changes to the UI of the application manager...). Is there a way this
can be approached in small steps? (i.e. maybe first a version that is
not very user friendly, but that works?)

There is -nothing- that Google has to do.  This is an open source project.  You have all of the source available to you.  Nothing is preventing anyone from adding these kinds of features...  except that, well, they are kind-of hard to do in a way that will be production quality and thus acceptable for the overall platform.

I do expect the core Android platform developers to implement this feature at some point, but it is not currently on our roadmap (but that also isn't saying much, since our roadmap currently only extends to Q1 next year :p).

--
Dianne Hackborn
hac...@android.com

[al74]

Nothing that Google has to do? Open source project?

I am probably in the wrong place then because as far as I am aware,
nobody other than Google (and perhaps HTC or T-mobile) can actualy
change the code on the only existing device currently implementing
this platform. In fact, I was surprise to discover that even what we
thought to be a simple task, such as adding Hebrew fonts so that the
G1 will be able to read Hebrew is not possible without root access.

The fact that it is not on your roadmap for Q1 2009 is frustrating, as
I don't see how users like me can actualy enjoy the benefit of the App
Market with the limited internal storage capcity most of us already
almost exhusted. Without the app market, Android is no better than a
regular cell phone with a nice touch screen.

On Nov 20, 4:34 pm, "Dianne Hackborn" <hack...@android.com> wrote:

> hack...@android.com

[Dianne Hackborn]

On Thu, Nov 20, 2008 at 1:52 PM, al74 <aviad...@gmail.com> wrote:

Nothing that Google has to do? Open source project?
I am probably in the wrong place then because as far as I am aware,
nobody other than Google (and perhaps HTC or T-mobile) can actualy
change the code on the only existing device currently implementing
this platform. In fact, I was surprise to discover that even what we
thought to be a simple task, such as adding Hebrew fonts so that the
G1 will be able to read Hebrew is not possible without root access.

Any patches that are contributed and accepted into the Android open source project will become part of the standard platform, and thus part of any future updates that are delivered to phones.  (Though of course ultimately it is up to the vendor to decide the exact software they deliver as updates, I think it would be an extremely unusual case for someone to take a platform update except some parts, especially if that part is something like an apps on SD cards feature.)

And this really has little to do with Google.  If I were to go out and buy a G1 from the store, I couldn't replace the software on it just like you can't.  Those certificates are owned by the phone manufacturer and/or carrier, not Google.
 

The fact that it is not on your roadmap for Q1 2009 is frustrating, as
I don't see how users like me can actualy enjoy the benefit of the App
Market with the limited internal storage capcity most of us already
almost exhusted. Without the app market, Android is no better than a
regular cell phone with a nice touch screen.

I totally understand about the desire to be able to have apps on the SD card, and the pain of not being able to do so.  I hope I have indicated that we have certainly put some thought into it and have I think a pretty good plan of how to accomplish it, but unfortunately I can't at this point give you a time when that will happen.

As far as the roadmap goes, you may not realize that shipping significant new versions of the platform (which this would be) in Q1 of 2009 essentially means being feature complete about this time right now.  So even if we all decided that yes, this is the highest priority thing to work on, and we're going to start coding right now...  well, it isn't going to be in the Q1 2009 roadmap.  We don't yet have a roadmap beyond that, though, so after Q1 it's a fair amount more open about what will be delivered. :)

--
Dianne Hackborn
Android framework engineer
hac...@android.com

Note: please don't send private questions to me, as I don't have time to provide private support.  All such questions should be posted on public forums, where I and others can see and answer them.

[Aaron]

Thanks Dianne for all the inputs. It's unfortunate that we are not
able to distribute our own android builds onto the phone, that was
something I was really looking forward for the G1, but I understand
the circumstances presented.

Secondly when you mentioned Q1 of 2009, that would mean like the march/
april time frame correct?

On Nov 20, 3:51 pm, "Dianne Hackborn" <hack...@android.com> wrote:

> hack...@android.com

[Dianne Hackborn]

On Thu, Nov 20, 2008 at 5:09 PM, Aaron <arr...@gmail.com> wrote:

Secondly when you mentioned Q1 of 2009, that would mean like the march/
april time frame correct?

Some time in Q1 - Jan, Feb, Mar, Apr.  Yeah the later part of those is usually the safe assumption. :)

--
Dianne Hackborn
Android framework engineer

hac...@android.com

[Jay Freeman (saurik)]

You may still be able to obtain an RC<30 (or as I like to say: RC-
hearty) build from the existing stock at a T-Mobile store and get root
access to it. I am currently doing active development on the Android
platform using my rooted G1 as a test bed (and therefore can mount my
SD card anywhere and however I want).

Sincerely,
Jay Freeman (saurik)
sau...@saurik.com
http://www.saurik.com/

[Lauren]

Greetings. Since I've blogged and written in various venues rather
enthusiastically about various aspects of Android and the G1, I've
become something of an unofficial magnet for related comments and
complaints. For what it's worth, the three top issues people are
asking me about (most frequent first):

1) The need for larger font choices, sticky font choices, and some
sort of magnifying zoom that can be applied to any displayed material
regardless of application. Small (tiny!) fonts and such are driving
folks without 20/20 vision absolutely batty.

2) Apps on SD card. The lack of this ability *right now* has become
widely enough known that it is scaring people away from the platform.
Even if in reality a given user wouldn't actually be constrained by
this at the moment, they don't want to buy in when they're worried
that they will be constrained down the line. Many of these are
current WM5/6 users who are used to installing apps on SD cards and
are incredulous that it can't be done now under Android, at least for
free apps.

3) SDK prohibition on turn-by-turn navigation programs. At least for
this one I'm able to point out that the iPhone SDK shares the same
prohibition and that licensing restrictions are likely the current
problem (though I'm still waiting for official confirmation on this).

Anyway, the SD card situation is definitely a big deal.

--Lauren--
Lauren Weinstein
lau...@vortex.com or lau...@pfir.org
Tel: +1 (818) 225-2800
http://www.pfir.org/lauren
Co-Founder, PFIR
- People For Internet Responsibility - http://www.pfir.org
Co-Founder, NNSquad
- Network Neutrality Squad - http://www.nnsquad.org
Founder, PRIVACY Forum - http://www.vortex.com
Member, ACM Committee on Computers and Public Policy
Lauren's Blog: http://lauren.vortex.com

[Phil HUXLEY]

You may want to introduce a system, where not all files are verified
against the signature to speed up the process. For example in a Game
containing a lot of texure maps and modelling data, the code in the package
can be small (< 10%) of the package size. The default behaviour would be
to check everything, but information in the manifest could be used say to
skip checking files of a certain type - e.g. *.jpg. The theory being that
the validating the exe is more important than validating the data (the
developer can decide).

BR,
Phil.

"Dianne Hackborn"
<hackbod@android.
com> To
Sent by: android-...@googlegroups.com
android-platform@ cc
googlegroups.com
Subject
Re: Apps on SD
20/11/2008 21:30


Please respond to
android-platform@
googlegroups.com

--
Dianne Hackborn
hac...@android.com

ForwardSourceID:NT0000356E

[Eric F]

It seems to me that keeping the filesystem FAT, is a necessary
constraint.

If that is the case, then certainly the installed apps has to exist as
a file on the FAT fileystem. I would propose a name that makes it
obvious to users what it is. AndroidApplications-<uniqueid> or
something like that. Since ejecting the SD card and putting it in a
reader is going to be a very likely scenario. Going to free up space
by deleting this unusual 2GB file is going to cross a lot of people's
minds if it isn't completely clear what it is.

The bulk of application data for many apps is not going to be in
the .apk file. The solution must allow apps that install onto an
external memory location to also grow their data outside of that
location. I think doing it any other way would defeat the purpose of
the solution. How about a game that has downloadable modules for
different levels or scenarios. Oh the 200KB base game installs on the
SD card but everything else goes onto the internal memory? No thank
you.

If dynamically resizing this file is going to be a performance killer
than I think a different solution is in order. If you are storing your
apps on an external card, it is a safe bet you have at least some
interest in technology. I think a reasonable solution could be to
involve the user. (Even though users can't be trusted to be incredibly
smart, I think interfaces can lead users to do the right thing if
designed correctly).

User buys G1.

User downloads app from marketplace. User begins installation of App1,
User downloads a second app from marketplace. "Your device only has
18MB remaining on internal storage. Many applications can store their
data or even be entirely stored on external memory cards".... after
installation user only has 2MB remaining on internal storage card.
User purchases and inserts 8GB SD card and inserts into Android phone.
User downloads a 3rd app from marketplace. A game that requires 20MB
to install, and up to 100MB of downloadable content. User is given two
options: Install on internal memory <Not enough space 2MB out of
20MB>, install on sd card.
User selects install on SD card.
"You have not yet prepared this SD card for use as an Android
Application store. You must do so now in order to continue with the
installation. Continue / Go Back"
User selects continue
"Your SD card has 7567MB of free space, how much space would you like
to allocate for android application storage? The SD card will be
unavailable to the rest of your device during this process."
User selects 2000MB.
User installs game to SD card.

User downloads App #4 from marketplace, a replacement homescreen with
several graphics (4MB in size)
"Please select where you want to install this application:
1-internal storage (not enough space) _free_up_space
2-external card (not applicable for this application)
"

The user cannot install the app at the moment. Selects free up space.
Launches a free up space on internal memory intent that launches up an
app allowing them to delete temp files, AND/OR move applications
currently installed to the internal memory to the SD card.
Applications are terminated when moving and temporarily unavailable.

User then finishes moving apps to the SD card, and installs the
homescreen app on internal memory.

Later the user is done installing apps and wants to recover space on
SD card for more music.

Launches the installed apps manager and selects resize space reserved
for applications on SD card. Can increase reservation up to 100% of SD
card's space and can decrease usage to 100% of the currently used
virtual file system on the FAT file. SD card is unavailable during the
transaction, as are all the SD card apps.

When an app queries for how much space it has for writing data files,
it is not told the whole size of the SD card, only the current
reservation. If it tries to use more, it gets a out of space error. So
no on the fly resizing must be done. I think this is the best possible
solution and I would like to know what everyone else thinks. I think
the most important part of having an idea is to recognize that if the
problem is interesting to solve, any given solution is going to have
some drawbacks and to recognize them even though you believe in the
idea.

I think the drawbacks here are:
A) It is not hidden from the users. Users must understand what is
going on behind the scenes to get more applications to install than
they normally could.
B) Users would be required to resize the SD card file frequently if
they are conservative in the amount of space they choose to reserve.

I personally think these drawbacks are superior to not having apps be
able to write their datafiles onto the SD card for fear of dynamic
allocation. A), it's a good thing that users understand what is going
on, because no matter how behind the scenes everything is taken care
of by Android, a user who doesn't understand what is going on is going
to eject the SD card and wonder where their phone user experience
went.

B), users can be given hints and help through the process telling them
the average size of an application, that if they allocate more, they
won't have to bother doing it again ever, but might not have space for
media content etc so they can make a good decision that will negate
most of the bad side effects.

OK, what other downsides are there I am not thinking of. What does
everyone think?

-E

On Nov 19, 3:20 pm, "Dianne Hackborn" <hack...@android.com> wrote:

> hack...@android.com

[Peli]

Looks like a good plan to me, but I'm not really in a position to
judge this.

I just wanted to add that I also think that it is important to keep
the apk as well as internal data *both* on the SD card.

As an example, I could state a 3rd party email application that stores
emails in a proprietary content provider.
If that content provider is restricted to stay in internal memory
(even if the apk is located on the SD card), then the end user will
not be very happy when internal memory is full due to large email
attachements, while the SD card is half empty.

Given the options to either not being able to store content on the SD
card, or having to adjust the size of the reserved space on the SD
card manually, I guess those who care would prefer the second
solution, and those who don't care - well - they don't care anyway.

Peli